Why would using Selenium webdriver to execute JS fine locally on my Mac, but not in a Docker container?
up vote
1
down vote
favorite
Docker environment: Docker Container, ubuntu:18.10, geckodriver-v0.23.0-linux64, selenium-3.14.1
Local environment: Mac OSX, geckodriver-v0.23.0-linux64, selenium-3.14.1
Why would executing JS work fine locally on my Mac, but not in a Docker container?
Python function that is triggering error on nfl.com:
def scroll_down(driver, value):
    """ Scroll down some """
    driver.execute_script("window.scrollBy(0,"+str(value)+")")
Traceback from nfl.com
Traceback (most recent call last):
  File "/app/foo/foo.py", line 159, in <module>
    main()
  File "/app/foo/foo.py", line 150, in main
    detect_full_html_loaded(driver)
  File "/app/foo/foo.py", line 55, in detect_full_html_loaded
    scroll_down(driver, 1000)
  File "/app/foo/foo.py", line 45, in scroll_down
    driver.execute_script("window.scrollBy(0,"+str(value)+")")
  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 636, in execute_script
    'args': converted_args})['value']
  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Failed to decode response from marionette
Python function that is triggering error on nytimes.com and sony.com:
def capture_screenshot(assets_dir, url, sleep, driver):
    """ Take simple screenshot of above-the-fold """
    driver.get(url)
    time.sleep(sleep)
    driver.set_window_size(1400, 700)
    Image.open(BytesIO(driver.get_screenshot_as_png())).save(os.path.join(assets_dir,'screenshot.png'))
    print('Created: "' + assets_dir  + 'screenshot.png' + '"')
Traceback on nytimes.com:
Traceback (most recent call last):
      File "/app/foo/foo.py", line 159, in <module>
        main()
      File "/app/foo/foo.py", line 152, in main
        capture_element_pic(input_file, assets_dir, data['finalUrl'], sel, sleep, driver)
      File "/app/foo/foo.py", line 89, in capture_element_pic
        driver.set_window_size(1400, driver.execute_script("return document.body.parentNode.scrollHeight"))
      File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1090, in set_window_size
        self.set_window_rect(width=int(width), height=int(height))
      File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1182, in set_window_rect
        "height": height})['value']
      File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
        self.error_handler.check_response(response)
      File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
        raise exception_class(message, screen, stacktrace)
    selenium.common.exceptions.WebDriverException: Message: [Exception... "Failure"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: chrome://marionette/content/browser.js :: get rect :: line 254"  data: no]
Traceback from sony.com
Traceback (most recent call last):
  File "/app/foo/foo.py", line 159, in <module>
    main()
  File "/app/foo/foo.py", line 149, in main
    capture_screenshot(assets_dir, data['finalUrl'], sleep, driver)
  File "/app/foo/foo.py", line 82, in capture_screenshot
    Image.open(BytesIO(driver.get_screenshot_as_png())).save(os.path.join(assets_dir,'screenshot.png'))
  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1064, in get_screenshot_as_png
    return base64.b64decode(self.get_screenshot_as_base64().encode('ascii'))
AttributeError: 'NoneType' object has no attribute 'encode'
Again, this works fine locally.
python selenium docker selenium-webdriver
add a comment |
up vote
1
down vote
favorite
Docker environment: Docker Container, ubuntu:18.10, geckodriver-v0.23.0-linux64, selenium-3.14.1
Local environment: Mac OSX, geckodriver-v0.23.0-linux64, selenium-3.14.1
Why would executing JS work fine locally on my Mac, but not in a Docker container?
Python function that is triggering error on nfl.com:
def scroll_down(driver, value):
    """ Scroll down some """
    driver.execute_script("window.scrollBy(0,"+str(value)+")")
Traceback from nfl.com
Traceback (most recent call last):
  File "/app/foo/foo.py", line 159, in <module>
    main()
  File "/app/foo/foo.py", line 150, in main
    detect_full_html_loaded(driver)
  File "/app/foo/foo.py", line 55, in detect_full_html_loaded
    scroll_down(driver, 1000)
  File "/app/foo/foo.py", line 45, in scroll_down
    driver.execute_script("window.scrollBy(0,"+str(value)+")")
  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 636, in execute_script
    'args': converted_args})['value']
  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Failed to decode response from marionette
Python function that is triggering error on nytimes.com and sony.com:
def capture_screenshot(assets_dir, url, sleep, driver):
    """ Take simple screenshot of above-the-fold """
    driver.get(url)
    time.sleep(sleep)
    driver.set_window_size(1400, 700)
    Image.open(BytesIO(driver.get_screenshot_as_png())).save(os.path.join(assets_dir,'screenshot.png'))
    print('Created: "' + assets_dir  + 'screenshot.png' + '"')
Traceback on nytimes.com:
Traceback (most recent call last):
      File "/app/foo/foo.py", line 159, in <module>
        main()
      File "/app/foo/foo.py", line 152, in main
        capture_element_pic(input_file, assets_dir, data['finalUrl'], sel, sleep, driver)
      File "/app/foo/foo.py", line 89, in capture_element_pic
        driver.set_window_size(1400, driver.execute_script("return document.body.parentNode.scrollHeight"))
      File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1090, in set_window_size
        self.set_window_rect(width=int(width), height=int(height))
      File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1182, in set_window_rect
        "height": height})['value']
      File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
        self.error_handler.check_response(response)
      File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
        raise exception_class(message, screen, stacktrace)
    selenium.common.exceptions.WebDriverException: Message: [Exception... "Failure"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: chrome://marionette/content/browser.js :: get rect :: line 254"  data: no]
Traceback from sony.com
Traceback (most recent call last):
  File "/app/foo/foo.py", line 159, in <module>
    main()
  File "/app/foo/foo.py", line 149, in main
    capture_screenshot(assets_dir, data['finalUrl'], sleep, driver)
  File "/app/foo/foo.py", line 82, in capture_screenshot
    Image.open(BytesIO(driver.get_screenshot_as_png())).save(os.path.join(assets_dir,'screenshot.png'))
  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1064, in get_screenshot_as_png
    return base64.b64decode(self.get_screenshot_as_base64().encode('ascii'))
AttributeError: 'NoneType' object has no attribute 'encode'
Again, this works fine locally.
python selenium docker selenium-webdriver
 
 
 
 
 
 
 Considering exception logs it doesn't seem like website is rejecting your stuff because it is Selenium
 – Andersson
 11 hours ago
 
 
 
 
 
 
 
 
 
 Forgot to mention that I am running this in a Docker container. And its seems to work when I run it natively on my Mac. So maybe its actually a question for the Docker people.
 – heart.cooks.mind
 11 hours ago
 
 
 
 
 
 1
 
 
 
 
 This might be configuration/performance issue...
 – Andersson
 11 hours ago
 
 
 
 
 
 
 
 
 
 Is it possible for a site to detect that you're using selenium? They can detect which browser you're using which is the geckodriver but I can't see how they would know if you're controlling the driver via selenium or not.
 – Marcel Wilson
 9 hours ago
 
 
 
 
 
 
 
 
 
 I think @Anderson is correct I think this ended up being a Docker issue related to configuration/performance, since I could make it work locally but not in the container. So I am going to pursue it in that direction and close this issue.
 – heart.cooks.mind
 9 hours ago
 
 
 
add a comment |
up vote
1
down vote
favorite
up vote
1
down vote
favorite
Docker environment: Docker Container, ubuntu:18.10, geckodriver-v0.23.0-linux64, selenium-3.14.1
Local environment: Mac OSX, geckodriver-v0.23.0-linux64, selenium-3.14.1
Why would executing JS work fine locally on my Mac, but not in a Docker container?
Python function that is triggering error on nfl.com:
def scroll_down(driver, value):
    """ Scroll down some """
    driver.execute_script("window.scrollBy(0,"+str(value)+")")
Traceback from nfl.com
Traceback (most recent call last):
  File "/app/foo/foo.py", line 159, in <module>
    main()
  File "/app/foo/foo.py", line 150, in main
    detect_full_html_loaded(driver)
  File "/app/foo/foo.py", line 55, in detect_full_html_loaded
    scroll_down(driver, 1000)
  File "/app/foo/foo.py", line 45, in scroll_down
    driver.execute_script("window.scrollBy(0,"+str(value)+")")
  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 636, in execute_script
    'args': converted_args})['value']
  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Failed to decode response from marionette
Python function that is triggering error on nytimes.com and sony.com:
def capture_screenshot(assets_dir, url, sleep, driver):
    """ Take simple screenshot of above-the-fold """
    driver.get(url)
    time.sleep(sleep)
    driver.set_window_size(1400, 700)
    Image.open(BytesIO(driver.get_screenshot_as_png())).save(os.path.join(assets_dir,'screenshot.png'))
    print('Created: "' + assets_dir  + 'screenshot.png' + '"')
Traceback on nytimes.com:
Traceback (most recent call last):
      File "/app/foo/foo.py", line 159, in <module>
        main()
      File "/app/foo/foo.py", line 152, in main
        capture_element_pic(input_file, assets_dir, data['finalUrl'], sel, sleep, driver)
      File "/app/foo/foo.py", line 89, in capture_element_pic
        driver.set_window_size(1400, driver.execute_script("return document.body.parentNode.scrollHeight"))
      File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1090, in set_window_size
        self.set_window_rect(width=int(width), height=int(height))
      File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1182, in set_window_rect
        "height": height})['value']
      File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
        self.error_handler.check_response(response)
      File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
        raise exception_class(message, screen, stacktrace)
    selenium.common.exceptions.WebDriverException: Message: [Exception... "Failure"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: chrome://marionette/content/browser.js :: get rect :: line 254"  data: no]
Traceback from sony.com
Traceback (most recent call last):
  File "/app/foo/foo.py", line 159, in <module>
    main()
  File "/app/foo/foo.py", line 149, in main
    capture_screenshot(assets_dir, data['finalUrl'], sleep, driver)
  File "/app/foo/foo.py", line 82, in capture_screenshot
    Image.open(BytesIO(driver.get_screenshot_as_png())).save(os.path.join(assets_dir,'screenshot.png'))
  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1064, in get_screenshot_as_png
    return base64.b64decode(self.get_screenshot_as_base64().encode('ascii'))
AttributeError: 'NoneType' object has no attribute 'encode'
Again, this works fine locally.
python selenium docker selenium-webdriver
Docker environment: Docker Container, ubuntu:18.10, geckodriver-v0.23.0-linux64, selenium-3.14.1
Local environment: Mac OSX, geckodriver-v0.23.0-linux64, selenium-3.14.1
Why would executing JS work fine locally on my Mac, but not in a Docker container?
Python function that is triggering error on nfl.com:
def scroll_down(driver, value):
    """ Scroll down some """
    driver.execute_script("window.scrollBy(0,"+str(value)+")")
Traceback from nfl.com
Traceback (most recent call last):
  File "/app/foo/foo.py", line 159, in <module>
    main()
  File "/app/foo/foo.py", line 150, in main
    detect_full_html_loaded(driver)
  File "/app/foo/foo.py", line 55, in detect_full_html_loaded
    scroll_down(driver, 1000)
  File "/app/foo/foo.py", line 45, in scroll_down
    driver.execute_script("window.scrollBy(0,"+str(value)+")")
  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 636, in execute_script
    'args': converted_args})['value']
  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Failed to decode response from marionette
Python function that is triggering error on nytimes.com and sony.com:
def capture_screenshot(assets_dir, url, sleep, driver):
    """ Take simple screenshot of above-the-fold """
    driver.get(url)
    time.sleep(sleep)
    driver.set_window_size(1400, 700)
    Image.open(BytesIO(driver.get_screenshot_as_png())).save(os.path.join(assets_dir,'screenshot.png'))
    print('Created: "' + assets_dir  + 'screenshot.png' + '"')
Traceback on nytimes.com:
Traceback (most recent call last):
      File "/app/foo/foo.py", line 159, in <module>
        main()
      File "/app/foo/foo.py", line 152, in main
        capture_element_pic(input_file, assets_dir, data['finalUrl'], sel, sleep, driver)
      File "/app/foo/foo.py", line 89, in capture_element_pic
        driver.set_window_size(1400, driver.execute_script("return document.body.parentNode.scrollHeight"))
      File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1090, in set_window_size
        self.set_window_rect(width=int(width), height=int(height))
      File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1182, in set_window_rect
        "height": height})['value']
      File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
        self.error_handler.check_response(response)
      File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
        raise exception_class(message, screen, stacktrace)
    selenium.common.exceptions.WebDriverException: Message: [Exception... "Failure"  nsresult: "0x80004005 (NS_ERROR_FAILURE)"  location: "JS frame :: chrome://marionette/content/browser.js :: get rect :: line 254"  data: no]
Traceback from sony.com
Traceback (most recent call last):
  File "/app/foo/foo.py", line 159, in <module>
    main()
  File "/app/foo/foo.py", line 149, in main
    capture_screenshot(assets_dir, data['finalUrl'], sleep, driver)
  File "/app/foo/foo.py", line 82, in capture_screenshot
    Image.open(BytesIO(driver.get_screenshot_as_png())).save(os.path.join(assets_dir,'screenshot.png'))
  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1064, in get_screenshot_as_png
    return base64.b64decode(self.get_screenshot_as_base64().encode('ascii'))
AttributeError: 'NoneType' object has no attribute 'encode'
Again, this works fine locally.
python selenium docker selenium-webdriver
python selenium docker selenium-webdriver
edited 9 hours ago
asked 11 hours ago
heart.cooks.mind
442214
442214
 
 
 
 
 
 
 Considering exception logs it doesn't seem like website is rejecting your stuff because it is Selenium
 – Andersson
 11 hours ago
 
 
 
 
 
 
 
 
 
 Forgot to mention that I am running this in a Docker container. And its seems to work when I run it natively on my Mac. So maybe its actually a question for the Docker people.
 – heart.cooks.mind
 11 hours ago
 
 
 
 
 
 1
 
 
 
 
 This might be configuration/performance issue...
 – Andersson
 11 hours ago
 
 
 
 
 
 
 
 
 
 Is it possible for a site to detect that you're using selenium? They can detect which browser you're using which is the geckodriver but I can't see how they would know if you're controlling the driver via selenium or not.
 – Marcel Wilson
 9 hours ago
 
 
 
 
 
 
 
 
 
 I think @Anderson is correct I think this ended up being a Docker issue related to configuration/performance, since I could make it work locally but not in the container. So I am going to pursue it in that direction and close this issue.
 – heart.cooks.mind
 9 hours ago
 
 
 
add a comment |
 
 
 
 
 
 
 Considering exception logs it doesn't seem like website is rejecting your stuff because it is Selenium
 – Andersson
 11 hours ago
 
 
 
 
 
 
 
 
 
 Forgot to mention that I am running this in a Docker container. And its seems to work when I run it natively on my Mac. So maybe its actually a question for the Docker people.
 – heart.cooks.mind
 11 hours ago
 
 
 
 
 
 1
 
 
 
 
 This might be configuration/performance issue...
 – Andersson
 11 hours ago
 
 
 
 
 
 
 
 
 
 Is it possible for a site to detect that you're using selenium? They can detect which browser you're using which is the geckodriver but I can't see how they would know if you're controlling the driver via selenium or not.
 – Marcel Wilson
 9 hours ago
 
 
 
 
 
 
 
 
 
 I think @Anderson is correct I think this ended up being a Docker issue related to configuration/performance, since I could make it work locally but not in the container. So I am going to pursue it in that direction and close this issue.
 – heart.cooks.mind
 9 hours ago
 
 
 
Considering exception logs it doesn't seem like website is rejecting your stuff because it is Selenium
– Andersson
11 hours ago
Considering exception logs it doesn't seem like website is rejecting your stuff because it is Selenium
– Andersson
11 hours ago
Forgot to mention that I am running this in a Docker container. And its seems to work when I run it natively on my Mac. So maybe its actually a question for the Docker people.
– heart.cooks.mind
11 hours ago
Forgot to mention that I am running this in a Docker container. And its seems to work when I run it natively on my Mac. So maybe its actually a question for the Docker people.
– heart.cooks.mind
11 hours ago
1
1
This might be configuration/performance issue...
– Andersson
11 hours ago
This might be configuration/performance issue...
– Andersson
11 hours ago
Is it possible for a site to detect that you're using selenium? They can detect which browser you're using which is the geckodriver but I can't see how they would know if you're controlling the driver via selenium or not.
– Marcel Wilson
9 hours ago
Is it possible for a site to detect that you're using selenium? They can detect which browser you're using which is the geckodriver but I can't see how they would know if you're controlling the driver via selenium or not.
– Marcel Wilson
9 hours ago
I think @Anderson is correct I think this ended up being a Docker issue related to configuration/performance, since I could make it work locally but not in the container. So I am going to pursue it in that direction and close this issue.
– heart.cooks.mind
9 hours ago
I think @Anderson is correct I think this ended up being a Docker issue related to configuration/performance, since I could make it work locally but not in the container. So I am going to pursue it in that direction and close this issue.
– heart.cooks.mind
9 hours ago
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53416143%2fwhy-would-using-selenium-webdriver-to-execute-js-fine-locally-on-my-mac-but-not%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Considering exception logs it doesn't seem like website is rejecting your stuff because it is Selenium
– Andersson
11 hours ago
Forgot to mention that I am running this in a Docker container. And its seems to work when I run it natively on my Mac. So maybe its actually a question for the Docker people.
– heart.cooks.mind
11 hours ago
1
This might be configuration/performance issue...
– Andersson
11 hours ago
Is it possible for a site to detect that you're using selenium? They can detect which browser you're using which is the geckodriver but I can't see how they would know if you're controlling the driver via selenium or not.
– Marcel Wilson
9 hours ago
I think @Anderson is correct I think this ended up being a Docker issue related to configuration/performance, since I could make it work locally but not in the container. So I am going to pursue it in that direction and close this issue.
– heart.cooks.mind
9 hours ago