NiFi blocked/hanging
up vote
2
down vote
favorite
We're running a NiFi 1.7.1 server that picks up files via 3 GetFile processors and uploads them to the cloud.
When the server starts, it runs fine and chugs through any files that it finds. However, after running for a few days it seems to grind to a halt:
- The GetFile processors all show 1 thread running, but they don't seem to be doing anything, even though there are files present in their source directories.
- Nothing is waiting in any queues.
- No messages appear in the logs.
- The "top" command shows java using about 3% CPU and 21% mem. This is a 4-CPU server with 8GB of memory.
- If I try to stop any processors via the web interface, it will become unresponsive. Upon reloading the page, I get the login screen, but after login it hangs on the loading animation, without showing the flow schema.
- If I restart the NiFi service, it suddenly runs fine: it will pick up all the waiting files and not leave any threads hanging (according to the web interface). This will last another few days...
What is going on here? How can I resolve this?
Edit: The three GetFile processors each read from a different folder, but they all send their files to the same place. They are configured to pick up all files (filepattern .*), poll every 10 seconds, minimum file age 1 minute, don't keep the source file. I didn't touch the scheduling tab so it's just defaults.
apache-nifi
asked Nov 22 at 16:23
Kricket
2,04642334
add a comment |
up vote
2
down vote
favorite
We're running a NiFi 1.7.1 server that picks up files via 3 GetFile processors and uploads them to the cloud.
When the server starts, it runs fine and chugs through any files that it finds. However, after running for a few days it seems to grind to a halt:
- The GetFile processors all show 1 thread running, but they don't seem to be doing anything, even though there are files present in their source directories.
- Nothing is waiting in any queues.
- No messages appear in the logs.
- The "top" command shows java using about 3% CPU and 21% mem. This is a 4-CPU server with 8GB of memory.
- If I try to stop any processors via the web interface, it will become unresponsive. Upon reloading the page, I get the login screen, but after login it hangs on the loading animation, without showing the flow schema.
- If I restart the NiFi service, it suddenly runs fine: it will pick up all the waiting files and not leave any threads hanging (according to the web interface). This will last another few days...
What is going on here? How can I resolve this?
Edit: The three GetFile processors each read from a different folder, but they all send their files to the same place. They are configured to pick up all files (filepattern .*), poll every 10 seconds, minimum file age 1 minute, don't keep the source file. I didn't touch the scheduling tab so it's just defaults.
apache-nifi
asked Nov 22 at 16:23
Kricket
2,04642334
3 GetFile processors are reading from one folder? could you share the parameters of all three? could you activate jmx and do thread dump?
– daggett
Nov 22 at 17:28
1
When this happens please run "./bin/nifi.sh dump" and get the thread dump from nifi-bootstrap.log and find the thread related to the GetFile processor, we will need to see the stack for that thread to understand what it is doing
– Bryan Bende
Nov 27 at 14:14
add a comment |
up vote
2
down vote
favorite
up vote
2
down vote
favorite
We're running a NiFi 1.7.1 server that picks up files via 3 GetFile processors and uploads them to the cloud.
When the server starts, it runs fine and chugs through any files that it finds. However, after running for a few days it seems to grind to a halt:
- The GetFile processors all show 1 thread running, but they don't seem to be doing anything, even though there are files present in their source directories.
- Nothing is waiting in any queues.
- No messages appear in the logs.
- The "top" command shows java using about 3% CPU and 21% mem. This is a 4-CPU server with 8GB of memory.
- If I try to stop any processors via the web interface, it will become unresponsive. Upon reloading the page, I get the login screen, but after login it hangs on the loading animation, without showing the flow schema.
- If I restart the NiFi service, it suddenly runs fine: it will pick up all the waiting files and not leave any threads hanging (according to the web interface). This will last another few days...
What is going on here? How can I resolve this?
Edit: The three GetFile processors each read from a different folder, but they all send their files to the same place. They are configured to pick up all files (filepattern .*), poll every 10 seconds, minimum file age 1 minute, don't keep the source file. I didn't touch the scheduling tab so it's just defaults.
apache-nifi
asked Nov 22 at 16:23
Kricket
2,04642334
We're running a NiFi 1.7.1 server that picks up files via 3 GetFile processors and uploads them to the cloud.
When the server starts, it runs fine and chugs through any files that it finds. However, after running for a few days it seems to grind to a halt:
- The GetFile processors all show 1 thread running, but they don't seem to be doing anything, even though there are files present in their source directories.
- Nothing is waiting in any queues.
- No messages appear in the logs.
- The "top" command shows java using about 3% CPU and 21% mem. This is a 4-CPU server with 8GB of memory.
- If I try to stop any processors via the web interface, it will become unresponsive. Upon reloading the page, I get the login screen, but after login it hangs on the loading animation, without showing the flow schema.
- If I restart the NiFi service, it suddenly runs fine: it will pick up all the waiting files and not leave any threads hanging (according to the web interface). This will last another few days...
What is going on here? How can I resolve this?
Edit: The three GetFile processors each read from a different folder, but they all send their files to the same place. They are configured to pick up all files (filepattern .*), poll every 10 seconds, minimum file age 1 minute, don't keep the source file. I didn't touch the scheduling tab so it's just defaults.
apache-nifi
apache-nifi
asked Nov 22 at 16:23
Kricket
2,04642334
asked Nov 22 at 16:23
Kricket
2,04642334
edited Nov 23 at 7:41
asked Nov 22 at 16:23
Kricket
2,04642334
asked Nov 22 at 16:23
Kricket
2,04642334
asked Nov 22 at 16:23
Kricket
2,04642334
2,04642334
3 GetFile processors are reading from one folder? could you share the parameters of all three? could you activate jmx and do thread dump?
– daggett
Nov 22 at 17:28
1
When this happens please run "./bin/nifi.sh dump" and get the thread dump from nifi-bootstrap.log and find the thread related to the GetFile processor, we will need to see the stack for that thread to understand what it is doing
– Bryan Bende
Nov 27 at 14:14
add a comment |
3 GetFile processors are reading from one folder? could you share the parameters of all three? could you activate jmx and do thread dump?
– daggett
Nov 22 at 17:28
1
When this happens please run "./bin/nifi.sh dump" and get the thread dump from nifi-bootstrap.log and find the thread related to the GetFile processor, we will need to see the stack for that thread to understand what it is doing
– Bryan Bende
Nov 27 at 14:14
3 GetFile processors are reading from one folder? could you share the parameters of all three? could you activate jmx and do thread dump?
– daggett
Nov 22 at 17:28
3 GetFile processors are reading from one folder? could you share the parameters of all three? could you activate jmx and do thread dump?
– daggett
Nov 22 at 17:28
1
1
When this happens please run "./bin/nifi.sh dump" and get the thread dump from nifi-bootstrap.log and find the thread related to the GetFile processor, we will need to see the stack for that thread to understand what it is doing
– Bryan Bende
Nov 27 at 14:14
When this happens please run "./bin/nifi.sh dump" and get the thread dump from nifi-bootstrap.log and find the thread related to the GetFile processor, we will need to see the stack for that thread to understand what it is doing
– Bryan Bende
Nov 27 at 14:14
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53434957%2fnifi-blocked-hanging%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
3 GetFile processors are reading from one folder? could you share the parameters of all three? could you activate jmx and do thread dump?
– daggett
Nov 22 at 17:28
1
When this happens please run "./bin/nifi.sh dump" and get the thread dump from nifi-bootstrap.log and find the thread related to the GetFile processor, we will need to see the stack for that thread to understand what it is doing
– Bryan Bende
Nov 27 at 14:14