I know that presently order by is not there in BeamSQL is there any work around for it
This is present
PCollection<BeamRecord> rec = rec_out.apply(BeamSql.query(
"SELECT bnk_name,state_name,val from PCOLLECTION order by val desc limit 2"));
But I need
PCollection<BeamRecord> rec = rec_out.apply(BeamSql.query(
"SELECT bnk_name,state_name,val from PCOLLECTION order by val desc "));
apache
google-cloud-platform google-cloud-dataflow apache-beam apache-beam-io edited Nov 28 '18 at 15:27
Kenn Knowles
2,940814
asked Nov 23 '18 at 2:20
TEJASWAKUMAR SIRIGIRI
12
add a comment |
This is present
PCollection<BeamRecord> rec = rec_out.apply(BeamSql.query(
"SELECT bnk_name,state_name,val from PCOLLECTION order by val desc limit 2"));
But I need
PCollection<BeamRecord> rec = rec_out.apply(BeamSql.query(
"SELECT bnk_name,state_name,val from PCOLLECTION order by val desc "));
apache
google-cloud-platform google-cloud-dataflow apache-beam apache-beam-io edited Nov 28 '18 at 15:27
Kenn Knowles
2,940814
asked Nov 23 '18 at 2:20
TEJASWAKUMAR SIRIGIRI
12
What will you do with the data after it is output? What is the purpose of the order?
– Kenn Knowles
Nov 28 '18 at 15:30
add a comment |
This is present
PCollection<BeamRecord> rec = rec_out.apply(BeamSql.query(
"SELECT bnk_name,state_name,val from PCOLLECTION order by val desc limit 2"));
But I need
PCollection<BeamRecord> rec = rec_out.apply(BeamSql.query(
"SELECT bnk_name,state_name,val from PCOLLECTION order by val desc "));
apache
google-cloud-platform google-cloud-dataflow apache-beam apache-beam-io edited Nov 28 '18 at 15:27
Kenn Knowles
2,940814
asked Nov 23 '18 at 2:20
TEJASWAKUMAR SIRIGIRI
12
This is present
PCollection<BeamRecord> rec = rec_out.apply(BeamSql.query(
"SELECT bnk_name,state_name,val from PCOLLECTION order by val desc limit 2"));
But I need
PCollection<BeamRecord> rec = rec_out.apply(BeamSql.query(
"SELECT bnk_name,state_name,val from PCOLLECTION order by val desc "));
apache
google-cloud-platform google-cloud-dataflow apache-beam apache-beam-io apache
google-cloud-platform google-cloud-dataflow apache-beam apache-beam-io edited Nov 28 '18 at 15:27
Kenn Knowles
2,940814
asked Nov 23 '18 at 2:20
TEJASWAKUMAR SIRIGIRI
12
edited Nov 28 '18 at 15:27
Kenn Knowles
2,940814
asked Nov 23 '18 at 2:20
TEJASWAKUMAR SIRIGIRI
12
edited Nov 28 '18 at 15:27
Kenn Knowles
2,940814
edited Nov 28 '18 at 15:27
Kenn Knowles
2,940814
edited Nov 28 '18 at 15:27
Kenn Knowles
2,940814
2,940814
asked Nov 23 '18 at 2:20
TEJASWAKUMAR SIRIGIRI
12
asked Nov 23 '18 at 2:20
TEJASWAKUMAR SIRIGIRI
12
asked Nov 23 '18 at 2:20
TEJASWAKUMAR SIRIGIRI
12
12
What will you do with the data after it is output? What is the purpose of the order?
– Kenn Knowles
Nov 28 '18 at 15:30
add a comment |
What will you do with the data after it is output? What is the purpose of the order?
– Kenn Knowles
Nov 28 '18 at 15:30
What will you do with the data after it is output? What is the purpose of the order?
– Kenn Knowles
Nov 28 '18 at 15:30
What will you do with the data after it is output? What is the purpose of the order?
– Kenn Knowles
Nov 28 '18 at 15:30
add a comment |
1 Answer
1
active
oldest
votes
This actually wouldn't help. PCollections are unordered in Beam, so you cannot apply ordering to the whole output.
Beam does not have ordering because it is usually not necessary, often not possible. Most problems where ordering is used have another solution that does not require global sorting.
If you describe more about what you will do with the output of the SQL query, I will update my answer with advice.
If you cannot share more, then you have the option of using Beam's sorter extension outside of the SQL query, but again it is not really for global sorting, but per-key order after a shuffle.
answered Nov 28 '18 at 15:29
Kenn Knowles
2,940814
I will store my output Pcollection to a csv file or any database .
– TEJASWAKUMAR SIRIGIRI
Dec 1 '18 at 13:19
That makes sense - if your data is small enough to store in a database then it is a great choice.
– Kenn Knowles
Dec 1 '18 at 17:22
Ok thanks for the information and suggestions.
– TEJASWAKUMAR SIRIGIRI
Dec 2 '18 at 15:58
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53439933%2fi-know-that-presently-order-by-is-not-there-in-beamsql-is-there-any-work-around%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
This actually wouldn't help. PCollections are unordered in Beam, so you cannot apply ordering to the whole output.
Beam does not have ordering because it is usually not necessary, often not possible. Most problems where ordering is used have another solution that does not require global sorting.
If you describe more about what you will do with the output of the SQL query, I will update my answer with advice.
If you cannot share more, then you have the option of using Beam's sorter extension outside of the SQL query, but again it is not really for global sorting, but per-key order after a shuffle.
answered Nov 28 '18 at 15:29
Kenn Knowles
2,940814
I will store my output Pcollection to a csv file or any database .
– TEJASWAKUMAR SIRIGIRI
Dec 1 '18 at 13:19
That makes sense - if your data is small enough to store in a database then it is a great choice.
– Kenn Knowles
Dec 1 '18 at 17:22
Ok thanks for the information and suggestions.
– TEJASWAKUMAR SIRIGIRI
Dec 2 '18 at 15:58
add a comment |
This actually wouldn't help. PCollections are unordered in Beam, so you cannot apply ordering to the whole output.
Beam does not have ordering because it is usually not necessary, often not possible. Most problems where ordering is used have another solution that does not require global sorting.
If you describe more about what you will do with the output of the SQL query, I will update my answer with advice.
If you cannot share more, then you have the option of using Beam's sorter extension outside of the SQL query, but again it is not really for global sorting, but per-key order after a shuffle.
answered Nov 28 '18 at 15:29
Kenn Knowles
2,940814
I will store my output Pcollection to a csv file or any database .
– TEJASWAKUMAR SIRIGIRI
Dec 1 '18 at 13:19
That makes sense - if your data is small enough to store in a database then it is a great choice.
– Kenn Knowles
Dec 1 '18 at 17:22
Ok thanks for the information and suggestions.
– TEJASWAKUMAR SIRIGIRI
Dec 2 '18 at 15:58
add a comment |
This actually wouldn't help. PCollections are unordered in Beam, so you cannot apply ordering to the whole output.
Beam does not have ordering because it is usually not necessary, often not possible. Most problems where ordering is used have another solution that does not require global sorting.
If you describe more about what you will do with the output of the SQL query, I will update my answer with advice.
If you cannot share more, then you have the option of using Beam's sorter extension outside of the SQL query, but again it is not really for global sorting, but per-key order after a shuffle.
answered Nov 28 '18 at 15:29
Kenn Knowles
2,940814
This actually wouldn't help. PCollections are unordered in Beam, so you cannot apply ordering to the whole output.
Beam does not have ordering because it is usually not necessary, often not possible. Most problems where ordering is used have another solution that does not require global sorting.
If you describe more about what you will do with the output of the SQL query, I will update my answer with advice.
If you cannot share more, then you have the option of using Beam's sorter extension outside of the SQL query, but again it is not really for global sorting, but per-key order after a shuffle.
answered Nov 28 '18 at 15:29
Kenn Knowles
2,940814
answered Nov 28 '18 at 15:29
Kenn Knowles
2,940814
answered Nov 28 '18 at 15:29
Kenn Knowles
2,940814
answered Nov 28 '18 at 15:29
Kenn Knowles
2,940814
2,940814
I will store my output Pcollection to a csv file or any database .
– TEJASWAKUMAR SIRIGIRI
Dec 1 '18 at 13:19
That makes sense - if your data is small enough to store in a database then it is a great choice.
– Kenn Knowles
Dec 1 '18 at 17:22
Ok thanks for the information and suggestions.
– TEJASWAKUMAR SIRIGIRI
Dec 2 '18 at 15:58
add a comment |
I will store my output Pcollection to a csv file or any database .
– TEJASWAKUMAR SIRIGIRI
Dec 1 '18 at 13:19
That makes sense - if your data is small enough to store in a database then it is a great choice.
– Kenn Knowles
Dec 1 '18 at 17:22
Ok thanks for the information and suggestions.
– TEJASWAKUMAR SIRIGIRI
Dec 2 '18 at 15:58
I will store my output Pcollection to a csv file or any database .
– TEJASWAKUMAR SIRIGIRI
Dec 1 '18 at 13:19
I will store my output Pcollection to a csv file or any database .
– TEJASWAKUMAR SIRIGIRI
Dec 1 '18 at 13:19
That makes sense - if your data is small enough to store in a database then it is a great choice.
– Kenn Knowles
Dec 1 '18 at 17:22
That makes sense - if your data is small enough to store in a database then it is a great choice.
– Kenn Knowles
Dec 1 '18 at 17:22
Ok thanks for the information and suggestions.
– TEJASWAKUMAR SIRIGIRI
Dec 2 '18 at 15:58
Ok thanks for the information and suggestions.
– TEJASWAKUMAR SIRIGIRI
Dec 2 '18 at 15:58
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53439933%2fi-know-that-presently-order-by-is-not-there-in-beamsql-is-there-any-work-around%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
What will you do with the data after it is output? What is the purpose of the order?
– Kenn Knowles
Nov 28 '18 at 15:30