Servers fail to bind to addresses at boot











up vote
3
down vote

favorite












I'm dealing with a known issue in RHEL 7 whereby services that specify an address to bind to will not start correctly. I've found a number of similar reports, many say they have been resolved with updates to systemd but I still face this problem. This affects all the services on my box (sshd, sshd, vsftpd, nginx) that don't just bind to 0.0.0.0.



I've found all sorts of supposed workarounds but none of them work for me consistently. Taking sshd as an example, config looks like this:



Port 22
ListenAddress 192.168.242.225
...


Here's what I've tried, alone and in combinations:



From https://bugzilla.redhat.com/show_bug.cgi?id=1352214#c4 (I've also tried sys-subsystem-net-devices-eth1.device in place of network-online.target but I suspect this doesn't wait for addressing to happen.)



mkdir /etc/systemd/system/sshd.service.d
tee /etc/systemd/system/sshd.service.d/wait.conf << 'EOF'
[Unit]
After=network-online.target
EOF


From https://bugzilla.redhat.com/show_bug.cgi?id=1352214#c11



mkdir /etc/systemd/system/sshd.service.d
tee /etc/systemd/system/sshd.service.d/wait.conf << 'EOF'
[Unit]
Wants=network-online.target
After=network-online.target
EOF


From https://bugzilla.redhat.com/show_bug.cgi?id=1438749#c0



systemctl add-wants multi-user.target network.target


From somewhere



mkdir /etc/systemd/system/sshd.service.requires
ln -s /usr/lib/systemd/system/network-online.target /etc/systemd/system/sshd.service.requires/


No matter what I try, I usually end up with "error: bind to port 22 on 192.168.242.125 failed: Cannot assign requested address". Sometimes, everything starts up perfectly, which I am guessing is down to a timing issue.



Running Scientific Linux (RHEL) 7.5 and network manager is enabled, all IP addressing is static. If there are any other details that might help, please let me know. Here is the output of journalctl after a failed startup, with After=network-online.target in the sshd unit file. Relevant stuff starts down around line 1700. Hoping someone has come across this issue and solved it successfully!










share|improve this question




























    up vote
    3
    down vote

    favorite












    I'm dealing with a known issue in RHEL 7 whereby services that specify an address to bind to will not start correctly. I've found a number of similar reports, many say they have been resolved with updates to systemd but I still face this problem. This affects all the services on my box (sshd, sshd, vsftpd, nginx) that don't just bind to 0.0.0.0.



    I've found all sorts of supposed workarounds but none of them work for me consistently. Taking sshd as an example, config looks like this:



    Port 22
    ListenAddress 192.168.242.225
    ...


    Here's what I've tried, alone and in combinations:



    From https://bugzilla.redhat.com/show_bug.cgi?id=1352214#c4 (I've also tried sys-subsystem-net-devices-eth1.device in place of network-online.target but I suspect this doesn't wait for addressing to happen.)



    mkdir /etc/systemd/system/sshd.service.d
    tee /etc/systemd/system/sshd.service.d/wait.conf << 'EOF'
    [Unit]
    After=network-online.target
    EOF


    From https://bugzilla.redhat.com/show_bug.cgi?id=1352214#c11



    mkdir /etc/systemd/system/sshd.service.d
    tee /etc/systemd/system/sshd.service.d/wait.conf << 'EOF'
    [Unit]
    Wants=network-online.target
    After=network-online.target
    EOF


    From https://bugzilla.redhat.com/show_bug.cgi?id=1438749#c0



    systemctl add-wants multi-user.target network.target


    From somewhere



    mkdir /etc/systemd/system/sshd.service.requires
    ln -s /usr/lib/systemd/system/network-online.target /etc/systemd/system/sshd.service.requires/


    No matter what I try, I usually end up with "error: bind to port 22 on 192.168.242.125 failed: Cannot assign requested address". Sometimes, everything starts up perfectly, which I am guessing is down to a timing issue.



    Running Scientific Linux (RHEL) 7.5 and network manager is enabled, all IP addressing is static. If there are any other details that might help, please let me know. Here is the output of journalctl after a failed startup, with After=network-online.target in the sshd unit file. Relevant stuff starts down around line 1700. Hoping someone has come across this issue and solved it successfully!










    share|improve this question


























      up vote
      3
      down vote

      favorite









      up vote
      3
      down vote

      favorite











      I'm dealing with a known issue in RHEL 7 whereby services that specify an address to bind to will not start correctly. I've found a number of similar reports, many say they have been resolved with updates to systemd but I still face this problem. This affects all the services on my box (sshd, sshd, vsftpd, nginx) that don't just bind to 0.0.0.0.



      I've found all sorts of supposed workarounds but none of them work for me consistently. Taking sshd as an example, config looks like this:



      Port 22
      ListenAddress 192.168.242.225
      ...


      Here's what I've tried, alone and in combinations:



      From https://bugzilla.redhat.com/show_bug.cgi?id=1352214#c4 (I've also tried sys-subsystem-net-devices-eth1.device in place of network-online.target but I suspect this doesn't wait for addressing to happen.)



      mkdir /etc/systemd/system/sshd.service.d
      tee /etc/systemd/system/sshd.service.d/wait.conf << 'EOF'
      [Unit]
      After=network-online.target
      EOF


      From https://bugzilla.redhat.com/show_bug.cgi?id=1352214#c11



      mkdir /etc/systemd/system/sshd.service.d
      tee /etc/systemd/system/sshd.service.d/wait.conf << 'EOF'
      [Unit]
      Wants=network-online.target
      After=network-online.target
      EOF


      From https://bugzilla.redhat.com/show_bug.cgi?id=1438749#c0



      systemctl add-wants multi-user.target network.target


      From somewhere



      mkdir /etc/systemd/system/sshd.service.requires
      ln -s /usr/lib/systemd/system/network-online.target /etc/systemd/system/sshd.service.requires/


      No matter what I try, I usually end up with "error: bind to port 22 on 192.168.242.125 failed: Cannot assign requested address". Sometimes, everything starts up perfectly, which I am guessing is down to a timing issue.



      Running Scientific Linux (RHEL) 7.5 and network manager is enabled, all IP addressing is static. If there are any other details that might help, please let me know. Here is the output of journalctl after a failed startup, with After=network-online.target in the sshd unit file. Relevant stuff starts down around line 1700. Hoping someone has come across this issue and solved it successfully!










      share|improve this question















      I'm dealing with a known issue in RHEL 7 whereby services that specify an address to bind to will not start correctly. I've found a number of similar reports, many say they have been resolved with updates to systemd but I still face this problem. This affects all the services on my box (sshd, sshd, vsftpd, nginx) that don't just bind to 0.0.0.0.



      I've found all sorts of supposed workarounds but none of them work for me consistently. Taking sshd as an example, config looks like this:



      Port 22
      ListenAddress 192.168.242.225
      ...


      Here's what I've tried, alone and in combinations:



      From https://bugzilla.redhat.com/show_bug.cgi?id=1352214#c4 (I've also tried sys-subsystem-net-devices-eth1.device in place of network-online.target but I suspect this doesn't wait for addressing to happen.)



      mkdir /etc/systemd/system/sshd.service.d
      tee /etc/systemd/system/sshd.service.d/wait.conf << 'EOF'
      [Unit]
      After=network-online.target
      EOF


      From https://bugzilla.redhat.com/show_bug.cgi?id=1352214#c11



      mkdir /etc/systemd/system/sshd.service.d
      tee /etc/systemd/system/sshd.service.d/wait.conf << 'EOF'
      [Unit]
      Wants=network-online.target
      After=network-online.target
      EOF


      From https://bugzilla.redhat.com/show_bug.cgi?id=1438749#c0



      systemctl add-wants multi-user.target network.target


      From somewhere



      mkdir /etc/systemd/system/sshd.service.requires
      ln -s /usr/lib/systemd/system/network-online.target /etc/systemd/system/sshd.service.requires/


      No matter what I try, I usually end up with "error: bind to port 22 on 192.168.242.125 failed: Cannot assign requested address". Sometimes, everything starts up perfectly, which I am guessing is down to a timing issue.



      Running Scientific Linux (RHEL) 7.5 and network manager is enabled, all IP addressing is static. If there are any other details that might help, please let me know. Here is the output of journalctl after a failed startup, with After=network-online.target in the sshd unit file. Relevant stuff starts down around line 1700. Hoping someone has come across this issue and solved it successfully!







      systemd rhel7






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited 1 hour ago

























      asked 3 hours ago









      miken32

      437621




      437621






















          1 Answer
          1






          active

          oldest

          votes

















          up vote
          3
          down vote













          It may be better to not configure system services to listen on specific IP addresses, and to control access to them via the host firewall if necessary.



          If you really need to be able to bind to specific IP addresses before they are configured on a network interface, you can work around the timing issue by setting the sysctl net.ipv4.ip_nonlocal_bind for IPv4 and the sysctl net.ipv6.ip_nonlocal_bind for IPv6. Services can then bind to IP addresses not configured on any network interface, but they will not be accessible until those IP addresses are configured on an interface.






          share|improve this answer





















          • For most of the services, listening on 0.0.0.0 would be fine, but nginx needs to run different services on different IP addresses. Are there any negative implications to allowing the services to attempt to bind to arbitrary IP addresses? (Assuming, of course, that the system administrator is halfway competent and doesn't configure services with bogus addresses!)
            – miken32
            1 hour ago












          • It might be confusing if an admin isn't expecting it. It's not normally done to configure services this way. Most of the time it isn't necessary as a host usually isn't on two or more networks at once. In the case of a web server, configuring a specific IP address to listen on means that a virtual host is inaccessible from other IP addresses (including the localhost addresses), which can make host agent based monitoring tricky. Overall it's just something to be avoided unless you really, really have a need for it.
            – Michael Hampton
            1 hour ago










          • This is a FreePBX box which includes a web-based administration tool and a user-facing portal. The "recommended" setup runs both these through the same virtual host, just sticks the admin stuff under an /admin directory. But I do not want a web app running as the asterisk user on a public IP address! So I need a completely separate vhost, php-fpm instance, etc. Admin runs on the private IP, which we access via VPN, user portal runs on the public IP.
            – miken32
            1 hour ago








          • 1




            Oh, well in that case... I'll buy you a beer, you're going to need it!
            – Michael Hampton
            1 hour ago






          • 1




            Hmm. I think if I were you I would just make Apache listen on localhost, and stick a copy of nginx in front of it, make that public facing, and filter by URL and IP address/network in the nginx configuration. Then I wouldn't have to disturb FreePBX's Apache customizations much or at all.
            – Michael Hampton
            1 hour ago













          Your Answer








          StackExchange.ready(function() {
          var channelOptions = {
          tags: "".split(" "),
          id: "2"
          };
          initTagRenderer("".split(" "), "".split(" "), channelOptions);

          StackExchange.using("externalEditor", function() {
          // Have to fire editor after snippets, if snippets enabled
          if (StackExchange.settings.snippets.snippetsEnabled) {
          StackExchange.using("snippets", function() {
          createEditor();
          });
          }
          else {
          createEditor();
          }
          });

          function createEditor() {
          StackExchange.prepareEditor({
          heartbeatType: 'answer',
          convertImagesToLinks: true,
          noModals: true,
          showLowRepImageUploadWarning: true,
          reputationToPostImages: 10,
          bindNavPrevention: true,
          postfix: "",
          imageUploader: {
          brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
          contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
          allowUrls: true
          },
          onDemand: true,
          discardSelector: ".discard-answer"
          ,immediatelyShowMarkdownHelp:true
          });


          }
          });














           

          draft saved


          draft discarded


















          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f941421%2fservers-fail-to-bind-to-addresses-at-boot%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown

























          1 Answer
          1






          active

          oldest

          votes








          1 Answer
          1






          active

          oldest

          votes









          active

          oldest

          votes






          active

          oldest

          votes








          up vote
          3
          down vote













          It may be better to not configure system services to listen on specific IP addresses, and to control access to them via the host firewall if necessary.



          If you really need to be able to bind to specific IP addresses before they are configured on a network interface, you can work around the timing issue by setting the sysctl net.ipv4.ip_nonlocal_bind for IPv4 and the sysctl net.ipv6.ip_nonlocal_bind for IPv6. Services can then bind to IP addresses not configured on any network interface, but they will not be accessible until those IP addresses are configured on an interface.






          share|improve this answer





















          • For most of the services, listening on 0.0.0.0 would be fine, but nginx needs to run different services on different IP addresses. Are there any negative implications to allowing the services to attempt to bind to arbitrary IP addresses? (Assuming, of course, that the system administrator is halfway competent and doesn't configure services with bogus addresses!)
            – miken32
            1 hour ago












          • It might be confusing if an admin isn't expecting it. It's not normally done to configure services this way. Most of the time it isn't necessary as a host usually isn't on two or more networks at once. In the case of a web server, configuring a specific IP address to listen on means that a virtual host is inaccessible from other IP addresses (including the localhost addresses), which can make host agent based monitoring tricky. Overall it's just something to be avoided unless you really, really have a need for it.
            – Michael Hampton
            1 hour ago










          • This is a FreePBX box which includes a web-based administration tool and a user-facing portal. The "recommended" setup runs both these through the same virtual host, just sticks the admin stuff under an /admin directory. But I do not want a web app running as the asterisk user on a public IP address! So I need a completely separate vhost, php-fpm instance, etc. Admin runs on the private IP, which we access via VPN, user portal runs on the public IP.
            – miken32
            1 hour ago








          • 1




            Oh, well in that case... I'll buy you a beer, you're going to need it!
            – Michael Hampton
            1 hour ago






          • 1




            Hmm. I think if I were you I would just make Apache listen on localhost, and stick a copy of nginx in front of it, make that public facing, and filter by URL and IP address/network in the nginx configuration. Then I wouldn't have to disturb FreePBX's Apache customizations much or at all.
            – Michael Hampton
            1 hour ago

















          up vote
          3
          down vote













          It may be better to not configure system services to listen on specific IP addresses, and to control access to them via the host firewall if necessary.



          If you really need to be able to bind to specific IP addresses before they are configured on a network interface, you can work around the timing issue by setting the sysctl net.ipv4.ip_nonlocal_bind for IPv4 and the sysctl net.ipv6.ip_nonlocal_bind for IPv6. Services can then bind to IP addresses not configured on any network interface, but they will not be accessible until those IP addresses are configured on an interface.






          share|improve this answer





















          • For most of the services, listening on 0.0.0.0 would be fine, but nginx needs to run different services on different IP addresses. Are there any negative implications to allowing the services to attempt to bind to arbitrary IP addresses? (Assuming, of course, that the system administrator is halfway competent and doesn't configure services with bogus addresses!)
            – miken32
            1 hour ago












          • It might be confusing if an admin isn't expecting it. It's not normally done to configure services this way. Most of the time it isn't necessary as a host usually isn't on two or more networks at once. In the case of a web server, configuring a specific IP address to listen on means that a virtual host is inaccessible from other IP addresses (including the localhost addresses), which can make host agent based monitoring tricky. Overall it's just something to be avoided unless you really, really have a need for it.
            – Michael Hampton
            1 hour ago










          • This is a FreePBX box which includes a web-based administration tool and a user-facing portal. The "recommended" setup runs both these through the same virtual host, just sticks the admin stuff under an /admin directory. But I do not want a web app running as the asterisk user on a public IP address! So I need a completely separate vhost, php-fpm instance, etc. Admin runs on the private IP, which we access via VPN, user portal runs on the public IP.
            – miken32
            1 hour ago








          • 1




            Oh, well in that case... I'll buy you a beer, you're going to need it!
            – Michael Hampton
            1 hour ago






          • 1




            Hmm. I think if I were you I would just make Apache listen on localhost, and stick a copy of nginx in front of it, make that public facing, and filter by URL and IP address/network in the nginx configuration. Then I wouldn't have to disturb FreePBX's Apache customizations much or at all.
            – Michael Hampton
            1 hour ago















          up vote
          3
          down vote










          up vote
          3
          down vote









          It may be better to not configure system services to listen on specific IP addresses, and to control access to them via the host firewall if necessary.



          If you really need to be able to bind to specific IP addresses before they are configured on a network interface, you can work around the timing issue by setting the sysctl net.ipv4.ip_nonlocal_bind for IPv4 and the sysctl net.ipv6.ip_nonlocal_bind for IPv6. Services can then bind to IP addresses not configured on any network interface, but they will not be accessible until those IP addresses are configured on an interface.






          share|improve this answer












          It may be better to not configure system services to listen on specific IP addresses, and to control access to them via the host firewall if necessary.



          If you really need to be able to bind to specific IP addresses before they are configured on a network interface, you can work around the timing issue by setting the sysctl net.ipv4.ip_nonlocal_bind for IPv4 and the sysctl net.ipv6.ip_nonlocal_bind for IPv6. Services can then bind to IP addresses not configured on any network interface, but they will not be accessible until those IP addresses are configured on an interface.







          share|improve this answer












          share|improve this answer



          share|improve this answer










          answered 1 hour ago









          Michael Hampton

          162k26299612




          162k26299612












          • For most of the services, listening on 0.0.0.0 would be fine, but nginx needs to run different services on different IP addresses. Are there any negative implications to allowing the services to attempt to bind to arbitrary IP addresses? (Assuming, of course, that the system administrator is halfway competent and doesn't configure services with bogus addresses!)
            – miken32
            1 hour ago












          • It might be confusing if an admin isn't expecting it. It's not normally done to configure services this way. Most of the time it isn't necessary as a host usually isn't on two or more networks at once. In the case of a web server, configuring a specific IP address to listen on means that a virtual host is inaccessible from other IP addresses (including the localhost addresses), which can make host agent based monitoring tricky. Overall it's just something to be avoided unless you really, really have a need for it.
            – Michael Hampton
            1 hour ago










          • This is a FreePBX box which includes a web-based administration tool and a user-facing portal. The "recommended" setup runs both these through the same virtual host, just sticks the admin stuff under an /admin directory. But I do not want a web app running as the asterisk user on a public IP address! So I need a completely separate vhost, php-fpm instance, etc. Admin runs on the private IP, which we access via VPN, user portal runs on the public IP.
            – miken32
            1 hour ago








          • 1




            Oh, well in that case... I'll buy you a beer, you're going to need it!
            – Michael Hampton
            1 hour ago






          • 1




            Hmm. I think if I were you I would just make Apache listen on localhost, and stick a copy of nginx in front of it, make that public facing, and filter by URL and IP address/network in the nginx configuration. Then I wouldn't have to disturb FreePBX's Apache customizations much or at all.
            – Michael Hampton
            1 hour ago




















          • For most of the services, listening on 0.0.0.0 would be fine, but nginx needs to run different services on different IP addresses. Are there any negative implications to allowing the services to attempt to bind to arbitrary IP addresses? (Assuming, of course, that the system administrator is halfway competent and doesn't configure services with bogus addresses!)
            – miken32
            1 hour ago












          • It might be confusing if an admin isn't expecting it. It's not normally done to configure services this way. Most of the time it isn't necessary as a host usually isn't on two or more networks at once. In the case of a web server, configuring a specific IP address to listen on means that a virtual host is inaccessible from other IP addresses (including the localhost addresses), which can make host agent based monitoring tricky. Overall it's just something to be avoided unless you really, really have a need for it.
            – Michael Hampton
            1 hour ago










          • This is a FreePBX box which includes a web-based administration tool and a user-facing portal. The "recommended" setup runs both these through the same virtual host, just sticks the admin stuff under an /admin directory. But I do not want a web app running as the asterisk user on a public IP address! So I need a completely separate vhost, php-fpm instance, etc. Admin runs on the private IP, which we access via VPN, user portal runs on the public IP.
            – miken32
            1 hour ago








          • 1




            Oh, well in that case... I'll buy you a beer, you're going to need it!
            – Michael Hampton
            1 hour ago






          • 1




            Hmm. I think if I were you I would just make Apache listen on localhost, and stick a copy of nginx in front of it, make that public facing, and filter by URL and IP address/network in the nginx configuration. Then I wouldn't have to disturb FreePBX's Apache customizations much or at all.
            – Michael Hampton
            1 hour ago


















          For most of the services, listening on 0.0.0.0 would be fine, but nginx needs to run different services on different IP addresses. Are there any negative implications to allowing the services to attempt to bind to arbitrary IP addresses? (Assuming, of course, that the system administrator is halfway competent and doesn't configure services with bogus addresses!)
          – miken32
          1 hour ago






          For most of the services, listening on 0.0.0.0 would be fine, but nginx needs to run different services on different IP addresses. Are there any negative implications to allowing the services to attempt to bind to arbitrary IP addresses? (Assuming, of course, that the system administrator is halfway competent and doesn't configure services with bogus addresses!)
          – miken32
          1 hour ago














          It might be confusing if an admin isn't expecting it. It's not normally done to configure services this way. Most of the time it isn't necessary as a host usually isn't on two or more networks at once. In the case of a web server, configuring a specific IP address to listen on means that a virtual host is inaccessible from other IP addresses (including the localhost addresses), which can make host agent based monitoring tricky. Overall it's just something to be avoided unless you really, really have a need for it.
          – Michael Hampton
          1 hour ago




          It might be confusing if an admin isn't expecting it. It's not normally done to configure services this way. Most of the time it isn't necessary as a host usually isn't on two or more networks at once. In the case of a web server, configuring a specific IP address to listen on means that a virtual host is inaccessible from other IP addresses (including the localhost addresses), which can make host agent based monitoring tricky. Overall it's just something to be avoided unless you really, really have a need for it.
          – Michael Hampton
          1 hour ago












          This is a FreePBX box which includes a web-based administration tool and a user-facing portal. The "recommended" setup runs both these through the same virtual host, just sticks the admin stuff under an /admin directory. But I do not want a web app running as the asterisk user on a public IP address! So I need a completely separate vhost, php-fpm instance, etc. Admin runs on the private IP, which we access via VPN, user portal runs on the public IP.
          – miken32
          1 hour ago






          This is a FreePBX box which includes a web-based administration tool and a user-facing portal. The "recommended" setup runs both these through the same virtual host, just sticks the admin stuff under an /admin directory. But I do not want a web app running as the asterisk user on a public IP address! So I need a completely separate vhost, php-fpm instance, etc. Admin runs on the private IP, which we access via VPN, user portal runs on the public IP.
          – miken32
          1 hour ago






          1




          1




          Oh, well in that case... I'll buy you a beer, you're going to need it!
          – Michael Hampton
          1 hour ago




          Oh, well in that case... I'll buy you a beer, you're going to need it!
          – Michael Hampton
          1 hour ago




          1




          1




          Hmm. I think if I were you I would just make Apache listen on localhost, and stick a copy of nginx in front of it, make that public facing, and filter by URL and IP address/network in the nginx configuration. Then I wouldn't have to disturb FreePBX's Apache customizations much or at all.
          – Michael Hampton
          1 hour ago






          Hmm. I think if I were you I would just make Apache listen on localhost, and stick a copy of nginx in front of it, make that public facing, and filter by URL and IP address/network in the nginx configuration. Then I wouldn't have to disturb FreePBX's Apache customizations much or at all.
          – Michael Hampton
          1 hour ago




















           

          draft saved


          draft discarded



















































           


          draft saved


          draft discarded














          StackExchange.ready(
          function () {
          StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fserverfault.com%2fquestions%2f941421%2fservers-fail-to-bind-to-addresses-at-boot%23new-answer', 'question_page');
          }
          );

          Post as a guest















          Required, but never shown





















































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown

































          Required, but never shown














          Required, but never shown












          Required, but never shown







          Required, but never shown







          Popular posts from this blog

          What visual should I use to simply compare current year value vs last year in Power BI desktop

          Alexandru Averescu

          Trompette piccolo