Why would using Selenium webdriver to execute JS fine locally on my Mac, but not in a Docker container?
up vote
1
down vote
favorite
Docker environment: Docker Container, ubuntu:18.10, geckodriver-v0.23.0-linux64, selenium-3.14.1
Local environment: Mac OSX, geckodriver-v0.23.0-linux64, selenium-3.14.1
Why would executing JS work fine locally on my Mac, but not in a Docker container?
Python function that is triggering error on nfl.com:
def scroll_down(driver, value):
""" Scroll down some """
driver.execute_script("window.scrollBy(0,"+str(value)+")")
Traceback from nfl.com
Traceback (most recent call last):
File "/app/foo/foo.py", line 159, in <module>
main()
File "/app/foo/foo.py", line 150, in main
detect_full_html_loaded(driver)
File "/app/foo/foo.py", line 55, in detect_full_html_loaded
scroll_down(driver, 1000)
File "/app/foo/foo.py", line 45, in scroll_down
driver.execute_script("window.scrollBy(0,"+str(value)+")")
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 636, in execute_script
'args': converted_args})['value']
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Failed to decode response from marionette
Python function that is triggering error on nytimes.com and sony.com:
def capture_screenshot(assets_dir, url, sleep, driver):
""" Take simple screenshot of above-the-fold """
driver.get(url)
time.sleep(sleep)
driver.set_window_size(1400, 700)
Image.open(BytesIO(driver.get_screenshot_as_png())).save(os.path.join(assets_dir,'screenshot.png'))
print('Created: "' + assets_dir + 'screenshot.png' + '"')
Traceback on nytimes.com:
Traceback (most recent call last):
File "/app/foo/foo.py", line 159, in <module>
main()
File "/app/foo/foo.py", line 152, in main
capture_element_pic(input_file, assets_dir, data['finalUrl'], sel, sleep, driver)
File "/app/foo/foo.py", line 89, in capture_element_pic
driver.set_window_size(1400, driver.execute_script("return document.body.parentNode.scrollHeight"))
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1090, in set_window_size
self.set_window_rect(width=int(width), height=int(height))
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1182, in set_window_rect
"height": height})['value']
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: [Exception... "Failure" nsresult: "0x80004005 (NS_ERROR_FAILURE)" location: "JS frame :: chrome://marionette/content/browser.js :: get rect :: line 254" data: no]
Traceback from sony.com
Traceback (most recent call last):
File "/app/foo/foo.py", line 159, in <module>
main()
File "/app/foo/foo.py", line 149, in main
capture_screenshot(assets_dir, data['finalUrl'], sleep, driver)
File "/app/foo/foo.py", line 82, in capture_screenshot
Image.open(BytesIO(driver.get_screenshot_as_png())).save(os.path.join(assets_dir,'screenshot.png'))
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1064, in get_screenshot_as_png
return base64.b64decode(self.get_screenshot_as_base64().encode('ascii'))
AttributeError: 'NoneType' object has no attribute 'encode'
Again, this works fine locally.
python selenium docker selenium-webdriver
add a comment |
up vote
1
down vote
favorite
Docker environment: Docker Container, ubuntu:18.10, geckodriver-v0.23.0-linux64, selenium-3.14.1
Local environment: Mac OSX, geckodriver-v0.23.0-linux64, selenium-3.14.1
Why would executing JS work fine locally on my Mac, but not in a Docker container?
Python function that is triggering error on nfl.com:
def scroll_down(driver, value):
""" Scroll down some """
driver.execute_script("window.scrollBy(0,"+str(value)+")")
Traceback from nfl.com
Traceback (most recent call last):
File "/app/foo/foo.py", line 159, in <module>
main()
File "/app/foo/foo.py", line 150, in main
detect_full_html_loaded(driver)
File "/app/foo/foo.py", line 55, in detect_full_html_loaded
scroll_down(driver, 1000)
File "/app/foo/foo.py", line 45, in scroll_down
driver.execute_script("window.scrollBy(0,"+str(value)+")")
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 636, in execute_script
'args': converted_args})['value']
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Failed to decode response from marionette
Python function that is triggering error on nytimes.com and sony.com:
def capture_screenshot(assets_dir, url, sleep, driver):
""" Take simple screenshot of above-the-fold """
driver.get(url)
time.sleep(sleep)
driver.set_window_size(1400, 700)
Image.open(BytesIO(driver.get_screenshot_as_png())).save(os.path.join(assets_dir,'screenshot.png'))
print('Created: "' + assets_dir + 'screenshot.png' + '"')
Traceback on nytimes.com:
Traceback (most recent call last):
File "/app/foo/foo.py", line 159, in <module>
main()
File "/app/foo/foo.py", line 152, in main
capture_element_pic(input_file, assets_dir, data['finalUrl'], sel, sleep, driver)
File "/app/foo/foo.py", line 89, in capture_element_pic
driver.set_window_size(1400, driver.execute_script("return document.body.parentNode.scrollHeight"))
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1090, in set_window_size
self.set_window_rect(width=int(width), height=int(height))
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1182, in set_window_rect
"height": height})['value']
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: [Exception... "Failure" nsresult: "0x80004005 (NS_ERROR_FAILURE)" location: "JS frame :: chrome://marionette/content/browser.js :: get rect :: line 254" data: no]
Traceback from sony.com
Traceback (most recent call last):
File "/app/foo/foo.py", line 159, in <module>
main()
File "/app/foo/foo.py", line 149, in main
capture_screenshot(assets_dir, data['finalUrl'], sleep, driver)
File "/app/foo/foo.py", line 82, in capture_screenshot
Image.open(BytesIO(driver.get_screenshot_as_png())).save(os.path.join(assets_dir,'screenshot.png'))
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1064, in get_screenshot_as_png
return base64.b64decode(self.get_screenshot_as_base64().encode('ascii'))
AttributeError: 'NoneType' object has no attribute 'encode'
Again, this works fine locally.
python selenium docker selenium-webdriver
Considering exception logs it doesn't seem like website is rejecting your stuff because it is Selenium
– Andersson
11 hours ago
Forgot to mention that I am running this in a Docker container. And its seems to work when I run it natively on my Mac. So maybe its actually a question for the Docker people.
– heart.cooks.mind
11 hours ago
1
This might be configuration/performance issue...
– Andersson
11 hours ago
Is it possible for a site to detect that you're using selenium? They can detect which browser you're using which is the geckodriver but I can't see how they would know if you're controlling the driver via selenium or not.
– Marcel Wilson
9 hours ago
I think @Anderson is correct I think this ended up being a Docker issue related to configuration/performance, since I could make it work locally but not in the container. So I am going to pursue it in that direction and close this issue.
– heart.cooks.mind
9 hours ago
add a comment |
up vote
1
down vote
favorite
up vote
1
down vote
favorite
Docker environment: Docker Container, ubuntu:18.10, geckodriver-v0.23.0-linux64, selenium-3.14.1
Local environment: Mac OSX, geckodriver-v0.23.0-linux64, selenium-3.14.1
Why would executing JS work fine locally on my Mac, but not in a Docker container?
Python function that is triggering error on nfl.com:
def scroll_down(driver, value):
""" Scroll down some """
driver.execute_script("window.scrollBy(0,"+str(value)+")")
Traceback from nfl.com
Traceback (most recent call last):
File "/app/foo/foo.py", line 159, in <module>
main()
File "/app/foo/foo.py", line 150, in main
detect_full_html_loaded(driver)
File "/app/foo/foo.py", line 55, in detect_full_html_loaded
scroll_down(driver, 1000)
File "/app/foo/foo.py", line 45, in scroll_down
driver.execute_script("window.scrollBy(0,"+str(value)+")")
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 636, in execute_script
'args': converted_args})['value']
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Failed to decode response from marionette
Python function that is triggering error on nytimes.com and sony.com:
def capture_screenshot(assets_dir, url, sleep, driver):
""" Take simple screenshot of above-the-fold """
driver.get(url)
time.sleep(sleep)
driver.set_window_size(1400, 700)
Image.open(BytesIO(driver.get_screenshot_as_png())).save(os.path.join(assets_dir,'screenshot.png'))
print('Created: "' + assets_dir + 'screenshot.png' + '"')
Traceback on nytimes.com:
Traceback (most recent call last):
File "/app/foo/foo.py", line 159, in <module>
main()
File "/app/foo/foo.py", line 152, in main
capture_element_pic(input_file, assets_dir, data['finalUrl'], sel, sleep, driver)
File "/app/foo/foo.py", line 89, in capture_element_pic
driver.set_window_size(1400, driver.execute_script("return document.body.parentNode.scrollHeight"))
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1090, in set_window_size
self.set_window_rect(width=int(width), height=int(height))
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1182, in set_window_rect
"height": height})['value']
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: [Exception... "Failure" nsresult: "0x80004005 (NS_ERROR_FAILURE)" location: "JS frame :: chrome://marionette/content/browser.js :: get rect :: line 254" data: no]
Traceback from sony.com
Traceback (most recent call last):
File "/app/foo/foo.py", line 159, in <module>
main()
File "/app/foo/foo.py", line 149, in main
capture_screenshot(assets_dir, data['finalUrl'], sleep, driver)
File "/app/foo/foo.py", line 82, in capture_screenshot
Image.open(BytesIO(driver.get_screenshot_as_png())).save(os.path.join(assets_dir,'screenshot.png'))
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1064, in get_screenshot_as_png
return base64.b64decode(self.get_screenshot_as_base64().encode('ascii'))
AttributeError: 'NoneType' object has no attribute 'encode'
Again, this works fine locally.
python selenium docker selenium-webdriver
Docker environment: Docker Container, ubuntu:18.10, geckodriver-v0.23.0-linux64, selenium-3.14.1
Local environment: Mac OSX, geckodriver-v0.23.0-linux64, selenium-3.14.1
Why would executing JS work fine locally on my Mac, but not in a Docker container?
Python function that is triggering error on nfl.com:
def scroll_down(driver, value):
""" Scroll down some """
driver.execute_script("window.scrollBy(0,"+str(value)+")")
Traceback from nfl.com
Traceback (most recent call last):
File "/app/foo/foo.py", line 159, in <module>
main()
File "/app/foo/foo.py", line 150, in main
detect_full_html_loaded(driver)
File "/app/foo/foo.py", line 55, in detect_full_html_loaded
scroll_down(driver, 1000)
File "/app/foo/foo.py", line 45, in scroll_down
driver.execute_script("window.scrollBy(0,"+str(value)+")")
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 636, in execute_script
'args': converted_args})['value']
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Failed to decode response from marionette
Python function that is triggering error on nytimes.com and sony.com:
def capture_screenshot(assets_dir, url, sleep, driver):
""" Take simple screenshot of above-the-fold """
driver.get(url)
time.sleep(sleep)
driver.set_window_size(1400, 700)
Image.open(BytesIO(driver.get_screenshot_as_png())).save(os.path.join(assets_dir,'screenshot.png'))
print('Created: "' + assets_dir + 'screenshot.png' + '"')
Traceback on nytimes.com:
Traceback (most recent call last):
File "/app/foo/foo.py", line 159, in <module>
main()
File "/app/foo/foo.py", line 152, in main
capture_element_pic(input_file, assets_dir, data['finalUrl'], sel, sleep, driver)
File "/app/foo/foo.py", line 89, in capture_element_pic
driver.set_window_size(1400, driver.execute_script("return document.body.parentNode.scrollHeight"))
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1090, in set_window_size
self.set_window_rect(width=int(width), height=int(height))
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1182, in set_window_rect
"height": height})['value']
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: [Exception... "Failure" nsresult: "0x80004005 (NS_ERROR_FAILURE)" location: "JS frame :: chrome://marionette/content/browser.js :: get rect :: line 254" data: no]
Traceback from sony.com
Traceback (most recent call last):
File "/app/foo/foo.py", line 159, in <module>
main()
File "/app/foo/foo.py", line 149, in main
capture_screenshot(assets_dir, data['finalUrl'], sleep, driver)
File "/app/foo/foo.py", line 82, in capture_screenshot
Image.open(BytesIO(driver.get_screenshot_as_png())).save(os.path.join(assets_dir,'screenshot.png'))
File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 1064, in get_screenshot_as_png
return base64.b64decode(self.get_screenshot_as_base64().encode('ascii'))
AttributeError: 'NoneType' object has no attribute 'encode'
Again, this works fine locally.
python selenium docker selenium-webdriver
python selenium docker selenium-webdriver
edited 9 hours ago
asked 11 hours ago
heart.cooks.mind
442214
442214
Considering exception logs it doesn't seem like website is rejecting your stuff because it is Selenium
– Andersson
11 hours ago
Forgot to mention that I am running this in a Docker container. And its seems to work when I run it natively on my Mac. So maybe its actually a question for the Docker people.
– heart.cooks.mind
11 hours ago
1
This might be configuration/performance issue...
– Andersson
11 hours ago
Is it possible for a site to detect that you're using selenium? They can detect which browser you're using which is the geckodriver but I can't see how they would know if you're controlling the driver via selenium or not.
– Marcel Wilson
9 hours ago
I think @Anderson is correct I think this ended up being a Docker issue related to configuration/performance, since I could make it work locally but not in the container. So I am going to pursue it in that direction and close this issue.
– heart.cooks.mind
9 hours ago
add a comment |
Considering exception logs it doesn't seem like website is rejecting your stuff because it is Selenium
– Andersson
11 hours ago
Forgot to mention that I am running this in a Docker container. And its seems to work when I run it natively on my Mac. So maybe its actually a question for the Docker people.
– heart.cooks.mind
11 hours ago
1
This might be configuration/performance issue...
– Andersson
11 hours ago
Is it possible for a site to detect that you're using selenium? They can detect which browser you're using which is the geckodriver but I can't see how they would know if you're controlling the driver via selenium or not.
– Marcel Wilson
9 hours ago
I think @Anderson is correct I think this ended up being a Docker issue related to configuration/performance, since I could make it work locally but not in the container. So I am going to pursue it in that direction and close this issue.
– heart.cooks.mind
9 hours ago
Considering exception logs it doesn't seem like website is rejecting your stuff because it is Selenium
– Andersson
11 hours ago
Considering exception logs it doesn't seem like website is rejecting your stuff because it is Selenium
– Andersson
11 hours ago
Forgot to mention that I am running this in a Docker container. And its seems to work when I run it natively on my Mac. So maybe its actually a question for the Docker people.
– heart.cooks.mind
11 hours ago
Forgot to mention that I am running this in a Docker container. And its seems to work when I run it natively on my Mac. So maybe its actually a question for the Docker people.
– heart.cooks.mind
11 hours ago
1
1
This might be configuration/performance issue...
– Andersson
11 hours ago
This might be configuration/performance issue...
– Andersson
11 hours ago
Is it possible for a site to detect that you're using selenium? They can detect which browser you're using which is the geckodriver but I can't see how they would know if you're controlling the driver via selenium or not.
– Marcel Wilson
9 hours ago
Is it possible for a site to detect that you're using selenium? They can detect which browser you're using which is the geckodriver but I can't see how they would know if you're controlling the driver via selenium or not.
– Marcel Wilson
9 hours ago
I think @Anderson is correct I think this ended up being a Docker issue related to configuration/performance, since I could make it work locally but not in the container. So I am going to pursue it in that direction and close this issue.
– heart.cooks.mind
9 hours ago
I think @Anderson is correct I think this ended up being a Docker issue related to configuration/performance, since I could make it work locally but not in the container. So I am going to pursue it in that direction and close this issue.
– heart.cooks.mind
9 hours ago
add a comment |
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53416143%2fwhy-would-using-selenium-webdriver-to-execute-js-fine-locally-on-my-mac-but-not%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Considering exception logs it doesn't seem like website is rejecting your stuff because it is Selenium
– Andersson
11 hours ago
Forgot to mention that I am running this in a Docker container. And its seems to work when I run it natively on my Mac. So maybe its actually a question for the Docker people.
– heart.cooks.mind
11 hours ago
1
This might be configuration/performance issue...
– Andersson
11 hours ago
Is it possible for a site to detect that you're using selenium? They can detect which browser you're using which is the geckodriver but I can't see how they would know if you're controlling the driver via selenium or not.
– Marcel Wilson
9 hours ago
I think @Anderson is correct I think this ended up being a Docker issue related to configuration/performance, since I could make it work locally but not in the container. So I am going to pursue it in that direction and close this issue.
– heart.cooks.mind
9 hours ago