Why only one core is taking all the load , how to make other 29 cores to take load?
1
I am trying to push my Spark processed data to 3 node cluster of C*. I am pushing 200 million records to Cassandra it is failing with below err.
Below it the my spark cluster configuration
Nodes : 12
vCores Total : 112
Total memory : 1.5 TB.
Below are my spark-sumbit parameters:
$SPARK_HOME/bin/spark-submit
--master yarn
--deploy-mode cluster
--name app
--class Driver
--executor-cores 3
--executor-memory 8g
--num-executors 10
--driver-cores 2
--driver-memory 10g
--conf spark.shuffle.service.enabled=true
--conf spark.dynamicAllocation.enabled=false
--conf spark.task.maxFailures=8
--conf spark.ui.killEnabled=false
oracle2c.jar
I have made the Spark data frame partitions 30 as below
+--------------------+-------+
|SPARK_PARTITION_ID()| count|
+--------------------+-------+
| 22|6687176|
| 29|6687175|
| 8|6687176|
| 21|6687176|
| 23|6687176|
| 5|6687176|
| 9|6687176|
| 17|6687176|
| 26|6687175|
| 27|6687175|
| 4|6687176|
| 10|6687176|
| 18|6687176|
| 25|6687175|
| 1|6687176|
| 20|6687176|
| 12|6687176|
| 28|6687175|
| 24|6687176|
| 15|6687176|
| 14|6687176|
| 6|6687176|
| 19|6687176|
| 0|6687175|
| 3|6687176|
| 11|6687176|
| 2|6687176|
| 7|6687176|
| 13|6687176|
| 16|6687176|
+--------------------+-------+
But when I run the job I see only one core in executor is running , how to make all the cores take part of load distribution?
What more configuration parameters I need to add to make all the cores take the load ?
apache-spark apache-spark-sql databricks
edited Nov 23 '18 at 13:39
user10465355
1,3581413
asked Nov 23 '18 at 7:14
user3252097
3061416
|
show 3 more comments
1
I am trying to push my Spark processed data to 3 node cluster of C*. I am pushing 200 million records to Cassandra it is failing with below err.
Below it the my spark cluster configuration
Nodes : 12
vCores Total : 112
Total memory : 1.5 TB.
Below are my spark-sumbit parameters:
$SPARK_HOME/bin/spark-submit
--master yarn
--deploy-mode cluster
--name app
--class Driver
--executor-cores 3
--executor-memory 8g
--num-executors 10
--driver-cores 2
--driver-memory 10g
--conf spark.shuffle.service.enabled=true
--conf spark.dynamicAllocation.enabled=false
--conf spark.task.maxFailures=8
--conf spark.ui.killEnabled=false
oracle2c.jar
I have made the Spark data frame partitions 30 as below
+--------------------+-------+
|SPARK_PARTITION_ID()| count|
+--------------------+-------+
| 22|6687176|
| 29|6687175|
| 8|6687176|
| 21|6687176|
| 23|6687176|
| 5|6687176|
| 9|6687176|
| 17|6687176|
| 26|6687175|
| 27|6687175|
| 4|6687176|
| 10|6687176|
| 18|6687176|
| 25|6687175|
| 1|6687176|
| 20|6687176|
| 12|6687176|
| 28|6687175|
| 24|6687176|
| 15|6687176|
| 14|6687176|
| 6|6687176|
| 19|6687176|
| 0|6687175|
| 3|6687176|
| 11|6687176|
| 2|6687176|
| 7|6687176|
| 13|6687176|
| 16|6687176|
+--------------------+-------+
But when I run the job I see only one core in executor is running , how to make all the cores take part of load distribution?
What more configuration parameters I need to add to make all the cores take the load ?
apache-spark apache-spark-sql databricks
edited Nov 23 '18 at 13:39
user10465355
1,3581413
asked Nov 23 '18 at 7:14
user3252097
3061416
do you have an spark executor on every C* node? Like explained here opencredo.com/deploy-spark-apache-cassandra on Design the deployment
– Josemy
Nov 23 '18 at 11:55
@Joss No , we have 3 node C* cluster on a different nodes all together and 12 node spark cluster on different nodes.... not able to understand what exactly you are trying to convey here.
– user3252097
Nov 26 '18 at 7:14
@AlexOtt , sir can you please advice on this issue ? my spark job , load is taken by only one core , even thought number of partitions & 29 other cores available.
– user3252097
Nov 26 '18 at 7:16
How are you loading the data to Cassandra? Can you make use offoreachPartition? So that each partition will run parallelly based on the available cores. @user3252097
– Bhima Rao Gogineni
Nov 28 '18 at 8:55
@BhimaRaoGogineni """ val numOfPartitions = 20 /// read from property file val new_df= org_df.repartition(numOfPartitions) new_df.write.format("org.apache.spark.sql.cassandra") .options(Map( "table" -> cFName, "keyspace" -> keyspace )) .mode(SaveMode.Append) .save() """"
– user3252097
Nov 28 '18 at 9:57
|
show 3 more comments
1
1
1
1
I am trying to push my Spark processed data to 3 node cluster of C*. I am pushing 200 million records to Cassandra it is failing with below err.
Below it the my spark cluster configuration
Nodes : 12
vCores Total : 112
Total memory : 1.5 TB.
Below are my spark-sumbit parameters:
$SPARK_HOME/bin/spark-submit
--master yarn
--deploy-mode cluster
--name app
--class Driver
--executor-cores 3
--executor-memory 8g
--num-executors 10
--driver-cores 2
--driver-memory 10g
--conf spark.shuffle.service.enabled=true
--conf spark.dynamicAllocation.enabled=false
--conf spark.task.maxFailures=8
--conf spark.ui.killEnabled=false
oracle2c.jar
I have made the Spark data frame partitions 30 as below
+--------------------+-------+
|SPARK_PARTITION_ID()| count|
+--------------------+-------+
| 22|6687176|
| 29|6687175|
| 8|6687176|
| 21|6687176|
| 23|6687176|
| 5|6687176|
| 9|6687176|
| 17|6687176|
| 26|6687175|
| 27|6687175|
| 4|6687176|
| 10|6687176|
| 18|6687176|
| 25|6687175|
| 1|6687176|
| 20|6687176|
| 12|6687176|
| 28|6687175|
| 24|6687176|
| 15|6687176|
| 14|6687176|
| 6|6687176|
| 19|6687176|
| 0|6687175|
| 3|6687176|
| 11|6687176|
| 2|6687176|
| 7|6687176|
| 13|6687176|
| 16|6687176|
+--------------------+-------+
But when I run the job I see only one core in executor is running , how to make all the cores take part of load distribution?
What more configuration parameters I need to add to make all the cores take the load ?
apache-spark apache-spark-sql databricks
edited Nov 23 '18 at 13:39
user10465355
1,3581413
asked Nov 23 '18 at 7:14
user3252097
3061416
I am trying to push my Spark processed data to 3 node cluster of C*. I am pushing 200 million records to Cassandra it is failing with below err.
Below it the my spark cluster configuration
Nodes : 12
vCores Total : 112
Total memory : 1.5 TB.
Below are my spark-sumbit parameters:
$SPARK_HOME/bin/spark-submit
--master yarn
--deploy-mode cluster
--name app
--class Driver
--executor-cores 3
--executor-memory 8g
--num-executors 10
--driver-cores 2
--driver-memory 10g
--conf spark.shuffle.service.enabled=true
--conf spark.dynamicAllocation.enabled=false
--conf spark.task.maxFailures=8
--conf spark.ui.killEnabled=false
oracle2c.jar
I have made the Spark data frame partitions 30 as below
+--------------------+-------+
|SPARK_PARTITION_ID()| count|
+--------------------+-------+
| 22|6687176|
| 29|6687175|
| 8|6687176|
| 21|6687176|
| 23|6687176|
| 5|6687176|
| 9|6687176|
| 17|6687176|
| 26|6687175|
| 27|6687175|
| 4|6687176|
| 10|6687176|
| 18|6687176|
| 25|6687175|
| 1|6687176|
| 20|6687176|
| 12|6687176|
| 28|6687175|
| 24|6687176|
| 15|6687176|
| 14|6687176|
| 6|6687176|
| 19|6687176|
| 0|6687175|
| 3|6687176|
| 11|6687176|
| 2|6687176|
| 7|6687176|
| 13|6687176|
| 16|6687176|
+--------------------+-------+
But when I run the job I see only one core in executor is running , how to make all the cores take part of load distribution?
What more configuration parameters I need to add to make all the cores take the load ?
apache-spark apache-spark-sql databricks
apache-spark apache-spark-sql databricks
edited Nov 23 '18 at 13:39
user10465355
1,3581413
asked Nov 23 '18 at 7:14
user3252097
3061416
edited Nov 23 '18 at 13:39
user10465355
1,3581413
asked Nov 23 '18 at 7:14
user3252097
3061416
edited Nov 23 '18 at 13:39
user10465355
1,3581413
edited Nov 23 '18 at 13:39
user10465355
1,3581413
edited Nov 23 '18 at 13:39
user10465355
1,3581413
1,3581413
asked Nov 23 '18 at 7:14
user3252097
3061416
asked Nov 23 '18 at 7:14
user3252097
3061416
asked Nov 23 '18 at 7:14
user3252097
3061416
3061416
do you have an spark executor on every C* node? Like explained here opencredo.com/deploy-spark-apache-cassandra on Design the deployment
– Josemy
Nov 23 '18 at 11:55
@Joss No , we have 3 node C* cluster on a different nodes all together and 12 node spark cluster on different nodes.... not able to understand what exactly you are trying to convey here.
– user3252097
Nov 26 '18 at 7:14
@AlexOtt , sir can you please advice on this issue ? my spark job , load is taken by only one core , even thought number of partitions & 29 other cores available.
– user3252097
Nov 26 '18 at 7:16
How are you loading the data to Cassandra? Can you make use offoreachPartition? So that each partition will run parallelly based on the available cores. @user3252097
– Bhima Rao Gogineni
Nov 28 '18 at 8:55
@BhimaRaoGogineni """ val numOfPartitions = 20 /// read from property file val new_df= org_df.repartition(numOfPartitions) new_df.write.format("org.apache.spark.sql.cassandra") .options(Map( "table" -> cFName, "keyspace" -> keyspace )) .mode(SaveMode.Append) .save() """"
– user3252097
Nov 28 '18 at 9:57
|
show 3 more comments
do you have an spark executor on every C* node? Like explained here opencredo.com/deploy-spark-apache-cassandra on Design the deployment
– Josemy
Nov 23 '18 at 11:55
@Joss No , we have 3 node C* cluster on a different nodes all together and 12 node spark cluster on different nodes.... not able to understand what exactly you are trying to convey here.
– user3252097
Nov 26 '18 at 7:14
@AlexOtt , sir can you please advice on this issue ? my spark job , load is taken by only one core , even thought number of partitions & 29 other cores available.
– user3252097
Nov 26 '18 at 7:16
How are you loading the data to Cassandra? Can you make use offoreachPartition? So that each partition will run parallelly based on the available cores. @user3252097
– Bhima Rao Gogineni
Nov 28 '18 at 8:55
@BhimaRaoGogineni """ val numOfPartitions = 20 /// read from property file val new_df= org_df.repartition(numOfPartitions) new_df.write.format("org.apache.spark.sql.cassandra") .options(Map( "table" -> cFName, "keyspace" -> keyspace )) .mode(SaveMode.Append) .save() """"
– user3252097
Nov 28 '18 at 9:57
do you have an spark executor on every C* node? Like explained here opencredo.com/deploy-spark-apache-cassandra on Design the deployment
– Josemy
Nov 23 '18 at 11:55
do you have an spark executor on every C* node? Like explained here opencredo.com/deploy-spark-apache-cassandra on Design the deployment
– Josemy
Nov 23 '18 at 11:55
@Joss No , we have 3 node C* cluster on a different nodes all together and 12 node spark cluster on different nodes.... not able to understand what exactly you are trying to convey here.
– user3252097
Nov 26 '18 at 7:14
@Joss No , we have 3 node C* cluster on a different nodes all together and 12 node spark cluster on different nodes.... not able to understand what exactly you are trying to convey here.
– user3252097
Nov 26 '18 at 7:14
@AlexOtt , sir can you please advice on this issue ? my spark job , load is taken by only one core , even thought number of partitions & 29 other cores available.
– user3252097
Nov 26 '18 at 7:16
@AlexOtt , sir can you please advice on this issue ? my spark job , load is taken by only one core , even thought number of partitions & 29 other cores available.
– user3252097
Nov 26 '18 at 7:16
How are you loading the data to Cassandra? Can you make use of
foreachPartition? So that each partition will run parallelly based on the available cores. @user3252097– Bhima Rao Gogineni
Nov 28 '18 at 8:55
How are you loading the data to Cassandra? Can you make use of
foreachPartition? So that each partition will run parallelly based on the available cores. @user3252097– Bhima Rao Gogineni
Nov 28 '18 at 8:55
@BhimaRaoGogineni """ val numOfPartitions = 20 /// read from property file val new_df= org_df.repartition(numOfPartitions) new_df.write.format("org.apache.spark.sql.cassandra") .options(Map( "table" -> cFName, "keyspace" -> keyspace )) .mode(SaveMode.Append) .save() """"
– user3252097
Nov 28 '18 at 9:57
@BhimaRaoGogineni """ val numOfPartitions = 20 /// read from property file val new_df= org_df.repartition(numOfPartitions) new_df.write.format("org.apache.spark.sql.cassandra") .options(Map( "table" -> cFName, "keyspace" -> keyspace )) .mode(SaveMode.Append) .save() """"
– user3252097
Nov 28 '18 at 9:57
|
show 3 more comments
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53442130%2fwhy-only-one-core-is-taking-all-the-load-how-to-make-other-29-cores-to-take-lo%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53442130%2fwhy-only-one-core-is-taking-all-the-load-how-to-make-other-29-cores-to-take-lo%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
do you have an spark executor on every C* node? Like explained here opencredo.com/deploy-spark-apache-cassandra on Design the deployment
– Josemy
Nov 23 '18 at 11:55
@Joss No , we have 3 node C* cluster on a different nodes all together and 12 node spark cluster on different nodes.... not able to understand what exactly you are trying to convey here.
– user3252097
Nov 26 '18 at 7:14
@AlexOtt , sir can you please advice on this issue ? my spark job , load is taken by only one core , even thought number of partitions & 29 other cores available.
– user3252097
Nov 26 '18 at 7:16
How are you loading the data to Cassandra? Can you make use of
foreachPartition? So that each partition will run parallelly based on the available cores. @user3252097– Bhima Rao Gogineni
Nov 28 '18 at 8:55
@BhimaRaoGogineni """ val numOfPartitions = 20 /// read from property file val new_df= org_df.repartition(numOfPartitions) new_df.write.format("org.apache.spark.sql.cassandra") .options(Map( "table" -> cFName, "keyspace" -> keyspace )) .mode(SaveMode.Append) .save() """"
– user3252097
Nov 28 '18 at 9:57