Kafka compaction on a topic stopped
up vote
0
down vote
favorite
I have multiple Kafka topics on which compaction is enabled. On all the topics compaction is happening fine but on one topic I am facing below issue.
The compaction happened until a point of time, after that it never happened. When I went through log cleaner logs there weren't any exceptions or logs telling me the potential reason why compaction stopped.
I am wondering what could be the reason for this behavior. How do i force my compaction to happen?
Any sort of advice would be helpful.
apache-kafka
edited Nov 21 at 21:23
Mickael Maison
6,56732428
asked Nov 21 at 21:07
user304611
368
add a comment |
up vote
0
down vote
favorite
I have multiple Kafka topics on which compaction is enabled. On all the topics compaction is happening fine but on one topic I am facing below issue.
The compaction happened until a point of time, after that it never happened. When I went through log cleaner logs there weren't any exceptions or logs telling me the potential reason why compaction stopped.
I am wondering what could be the reason for this behavior. How do i force my compaction to happen?
Any sort of advice would be helpful.
apache-kafka
edited Nov 21 at 21:23
Mickael Maison
6,56732428
asked Nov 21 at 21:07
user304611
368
How do you know compaction has stopped?
– Mickael Maison
Nov 21 at 21:10
My disk size is growing more than it should, and from the logs I could see compaction is not happening.
– user304611
Nov 21 at 21:13
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
I have multiple Kafka topics on which compaction is enabled. On all the topics compaction is happening fine but on one topic I am facing below issue.
The compaction happened until a point of time, after that it never happened. When I went through log cleaner logs there weren't any exceptions or logs telling me the potential reason why compaction stopped.
I am wondering what could be the reason for this behavior. How do i force my compaction to happen?
Any sort of advice would be helpful.
apache-kafka
edited Nov 21 at 21:23
Mickael Maison
6,56732428
asked Nov 21 at 21:07
user304611
368
I have multiple Kafka topics on which compaction is enabled. On all the topics compaction is happening fine but on one topic I am facing below issue.
The compaction happened until a point of time, after that it never happened. When I went through log cleaner logs there weren't any exceptions or logs telling me the potential reason why compaction stopped.
I am wondering what could be the reason for this behavior. How do i force my compaction to happen?
Any sort of advice would be helpful.
apache-kafka
apache-kafka
edited Nov 21 at 21:23
Mickael Maison
6,56732428
asked Nov 21 at 21:07
user304611
368
edited Nov 21 at 21:23
Mickael Maison
6,56732428
asked Nov 21 at 21:07
user304611
368
edited Nov 21 at 21:23
Mickael Maison
6,56732428
edited Nov 21 at 21:23
Mickael Maison
6,56732428
edited Nov 21 at 21:23
Mickael Maison
6,56732428
6,56732428
asked Nov 21 at 21:07
user304611
368
asked Nov 21 at 21:07
user304611
368
asked Nov 21 at 21:07
user304611
368
368
How do you know compaction has stopped?
– Mickael Maison
Nov 21 at 21:10
My disk size is growing more than it should, and from the logs I could see compaction is not happening.
– user304611
Nov 21 at 21:13
add a comment |
How do you know compaction has stopped?
– Mickael Maison
Nov 21 at 21:10
My disk size is growing more than it should, and from the logs I could see compaction is not happening.
– user304611
Nov 21 at 21:13
How do you know compaction has stopped?
– Mickael Maison
Nov 21 at 21:10
How do you know compaction has stopped?
– Mickael Maison
Nov 21 at 21:10
My disk size is growing more than it should, and from the logs I could see compaction is not happening.
– user304611
Nov 21 at 21:13
My disk size is growing more than it should, and from the logs I could see compaction is not happening.
– user304611
Nov 21 at 21:13
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
This usually happen because the compacting thread (log cleaner) crashed. Next step in troubleshooting is to find the reason it crashed and fix it. I'd search the main server logs for exceptions.
Log cleaner became a lot more stable over time, so if you are running versions older than 2.0, I recommend upgrading.
answered Nov 21 at 21:21
Gwen Shapira
2,3741318
I am running version above 2, i have gone through all the server logs, nothing regarding any exceptions is found. Are the log cleaner threads separate for every topic? Could i force compact somehow?
– user304611
Nov 22 at 3:33
All the new index, timestamp index files are getting created for new segments, but the process after that compaction is not happening.
– user304611
Nov 22 at 4:12
@user306log-cleaner.logis separate fromserver.log, yes... But what do you mean "version above 2"? That's the current, and highest version of Kafka
– cricket_007
Nov 22 at 20:23
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
This usually happen because the compacting thread (log cleaner) crashed. Next step in troubleshooting is to find the reason it crashed and fix it. I'd search the main server logs for exceptions.
Log cleaner became a lot more stable over time, so if you are running versions older than 2.0, I recommend upgrading.
answered Nov 21 at 21:21
Gwen Shapira
2,3741318
I am running version above 2, i have gone through all the server logs, nothing regarding any exceptions is found. Are the log cleaner threads separate for every topic? Could i force compact somehow?
– user304611
Nov 22 at 3:33
All the new index, timestamp index files are getting created for new segments, but the process after that compaction is not happening.
– user304611
Nov 22 at 4:12
@user306log-cleaner.logis separate fromserver.log, yes... But what do you mean "version above 2"? That's the current, and highest version of Kafka
– cricket_007
Nov 22 at 20:23
add a comment |
up vote
0
down vote
This usually happen because the compacting thread (log cleaner) crashed. Next step in troubleshooting is to find the reason it crashed and fix it. I'd search the main server logs for exceptions.
Log cleaner became a lot more stable over time, so if you are running versions older than 2.0, I recommend upgrading.
answered Nov 21 at 21:21
Gwen Shapira
2,3741318
I am running version above 2, i have gone through all the server logs, nothing regarding any exceptions is found. Are the log cleaner threads separate for every topic? Could i force compact somehow?
– user304611
Nov 22 at 3:33
All the new index, timestamp index files are getting created for new segments, but the process after that compaction is not happening.
– user304611
Nov 22 at 4:12
@user306log-cleaner.logis separate fromserver.log, yes... But what do you mean "version above 2"? That's the current, and highest version of Kafka
– cricket_007
Nov 22 at 20:23
add a comment |
up vote
0
down vote
up vote
0
down vote
This usually happen because the compacting thread (log cleaner) crashed. Next step in troubleshooting is to find the reason it crashed and fix it. I'd search the main server logs for exceptions.
Log cleaner became a lot more stable over time, so if you are running versions older than 2.0, I recommend upgrading.
answered Nov 21 at 21:21
Gwen Shapira
2,3741318
This usually happen because the compacting thread (log cleaner) crashed. Next step in troubleshooting is to find the reason it crashed and fix it. I'd search the main server logs for exceptions.
Log cleaner became a lot more stable over time, so if you are running versions older than 2.0, I recommend upgrading.
answered Nov 21 at 21:21
Gwen Shapira
2,3741318
answered Nov 21 at 21:21
Gwen Shapira
2,3741318
answered Nov 21 at 21:21
Gwen Shapira
2,3741318
answered Nov 21 at 21:21
Gwen Shapira
2,3741318
2,3741318
I am running version above 2, i have gone through all the server logs, nothing regarding any exceptions is found. Are the log cleaner threads separate for every topic? Could i force compact somehow?
– user304611
Nov 22 at 3:33
All the new index, timestamp index files are getting created for new segments, but the process after that compaction is not happening.
– user304611
Nov 22 at 4:12
@user306log-cleaner.logis separate fromserver.log, yes... But what do you mean "version above 2"? That's the current, and highest version of Kafka
– cricket_007
Nov 22 at 20:23
add a comment |
I am running version above 2, i have gone through all the server logs, nothing regarding any exceptions is found. Are the log cleaner threads separate for every topic? Could i force compact somehow?
– user304611
Nov 22 at 3:33
All the new index, timestamp index files are getting created for new segments, but the process after that compaction is not happening.
– user304611
Nov 22 at 4:12
@user306log-cleaner.logis separate fromserver.log, yes... But what do you mean "version above 2"? That's the current, and highest version of Kafka
– cricket_007
Nov 22 at 20:23
I am running version above 2, i have gone through all the server logs, nothing regarding any exceptions is found. Are the log cleaner threads separate for every topic? Could i force compact somehow?
– user304611
Nov 22 at 3:33
I am running version above 2, i have gone through all the server logs, nothing regarding any exceptions is found. Are the log cleaner threads separate for every topic? Could i force compact somehow?
– user304611
Nov 22 at 3:33
All the new index, timestamp index files are getting created for new segments, but the process after that compaction is not happening.
– user304611
Nov 22 at 4:12
All the new index, timestamp index files are getting created for new segments, but the process after that compaction is not happening.
– user304611
Nov 22 at 4:12
@user306
log-cleaner.log is separate from server.log, yes... But what do you mean "version above 2"? That's the current, and highest version of Kafka– cricket_007
Nov 22 at 20:23
@user306
log-cleaner.log is separate from server.log, yes... But what do you mean "version above 2"? That's the current, and highest version of Kafka– cricket_007
Nov 22 at 20:23
add a comment |
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53420484%2fkafka-compaction-on-a-topic-stopped%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
How do you know compaction has stopped?
– Mickael Maison
Nov 21 at 21:10
My disk size is growing more than it should, and from the logs I could see compaction is not happening.
– user304611
Nov 21 at 21:13