cProfiling a route in Python Flask

up vote
-2
down vote

favorite

So I ran a cProfile on my analysis route which I know takes time and looks like I found the bottleneck but don't know how to fix it. I am using requests 7 times so I can see why but its showing it did 18 calls to requests and 12 to socket.

I did research and it seems like it has something to do with the requests module using PyopenSSl see here: https://bugzilla.redhat.com/show_bug.cgi?id=1567862#c11

But I dont see a way to fix, any assistance is greatly appreciated.

Ordered by: internal time, call count

   List reduced from 1586 to 30 due to restriction <30>



   ncalls  tottime  percall  cumtime  percall filename:lineno(function)

       18    3.677    0.204    3.677    0.204 {built-in method SSL_read}

       12    1.492    0.124    1.493    0.124 {method 'connect' of '_socket.socket' objects}

      946    1.041    0.001    1.041    0.001 {built-in method __new__ of type object at 0x0000000055F8FD50}

       10    0.843    0.084    0.844    0.084 {built-in method getaddrinfo}

        6    0.777    0.129    0.777    0.129 {method 'recv_into' of '_socket.socket' objects}

        5    0.772    0.154    0.773    0.155 {built-in method SSL_do_handshake}

       10    0.504    0.050    0.504    0.050 {method 'read' of '_ssl._SSLSocket' objects}

        4    0.268    0.067    0.268    0.067 {method 'recv' of '_socket.socket' objects}

        2    0.231    0.115    0.231    0.115 {method 'do_handshake' of '_ssl._SSLSocket' objects}

        5    0.087    0.017    0.087    0.017 {built-in method SSL_CTX_load_verify_locations}

        3    0.070    0.023    0.070    0.023 <frozen importlib._bootstrap>:1598(get_data)

        1    0.048    0.048    0.122    0.122 C:UsersdevanditPycharmProjectswhovenvscrapslibsite-packagespyphen__init__.py:117(__init__)

        3    0.028    0.009    0.028    0.009 {built-in method open}

    11152    0.025    0.000    0.025    0.000 {method 'findall' of '_sre.SRE_Pattern' objects}

        1    0.024    0.024    0.071    0.071 C:UsersdevanditAppDataRoamingPythonPython34site-packagestextblob_text.py:732(load)

    11129    0.023    0.000    0.023    0.000 C:UsersdevanditPycharmProjectswhovenvscrapslibsite-packagespyphen__init__.py:149(<listcomp>)

        1    0.015    0.015    0.015    0.015 {method '_parse_whole' of 'xml.etree.ElementTree.XMLParser' objects}

        6    0.008    0.001    0.011    0.002 C:UsersdevanditAppDataRoamingPythonPython34site-packagesurllib3connectionpool.py:407(close)

    11295    0.008    0.000    0.008    0.000 {method 'sub' of '_sre.SRE_Pattern' objects}

    16296    0.007    0.000    0.010    0.000 C:UsersdevanditAppDataRoamingPythonPython34site-packagestextblob_text.py:688(avg)

32008/31977    0.006    0.000    0.013    0.000 {built-in method isinstance}

      190    0.005    0.000    0.008    0.000 C:UsersdevanditPycharmProjectswhovenvscrapslibsite-packagespyphen__init__.py:168(positions)

    34615    0.005    0.000    0.005    0.000 {method 'startswith' of 'str' objects}

     7775    0.005    0.000    0.005    0.000 C:UsersdevanditPycharmProjectswhovenvscrapslibsite-packagestextstattextstat.py:41(<genexpr>)

    11336    0.005    0.000    0.007    0.000 {built-in method max}

1785/1160    0.004    0.000    0.010    0.000 C:UsersdevanditAppDataRoamingPythonPython34site-packagesbs4element.py:1792(_matches)

     8067    0.004    0.000    0.007    0.000 C:UsersdevanditPycharmProjectswhovenvscrapslibsite-packageswhois__init__.py:77(<genexpr>)

11834/11830    0.004    0.000    0.004    0.000 {method 'decode' of 'bytes' objects}

       35    0.003    0.000    0.003    0.000 {built-in method stat}

    30138    0.003    0.000    0.003    0.000 {method 'get' of 'dict' objects}

edited Nov 22 at 17:57

davidism

61.5k12156175

asked Nov 22 at 14:55

Jason Martinez

2416

1

Hi, tyre slowdown might be just related to network overhead (read, recv, do_handshake) are you accessing the server locally or remotely? Also, you might want to check if you can use HTTP/2 as it reduces SSL overhead of multiple requests. Posting your relevant code would be appreciated.
– lolopop
Nov 22 at 20:35

I don't think you know the requests package because your saying to slow down and to show you code? Its requests so 7 different times I said requests.get(site) the site being different locations.
– Jason Martinez
Nov 23 at 3:10

So what evidence do you have that the code of requests itself is slowing this down significantly, i.e. that this is not due to the other parties involved which are (I would have thought) much greater unknowns, i.e. the internet and the remote server? And even if it is Requests, why ask here? You have the source code so you could try fix it, or you can raise an issue on the requests github... or you could evn try uninstalling PyOpenSSL to see if requests falls back to something better as implied by the redhat issue you link to.
– barny
Nov 23 at 11:01

add a comment |

up vote
-2
down vote

favorite

I did research and it seems like it has something to do with the requests module using PyopenSSl see here: https://bugzilla.redhat.com/show_bug.cgi?id=1567862#c11

But I dont see a way to fix, any assistance is greatly appreciated.

Ordered by: internal time, call count

   List reduced from 1586 to 30 due to restriction <30>



   ncalls  tottime  percall  cumtime  percall filename:lineno(function)

       18    3.677    0.204    3.677    0.204 {built-in method SSL_read}

       12    1.492    0.124    1.493    0.124 {method 'connect' of '_socket.socket' objects}

      946    1.041    0.001    1.041    0.001 {built-in method __new__ of type object at 0x0000000055F8FD50}

       10    0.843    0.084    0.844    0.084 {built-in method getaddrinfo}

        6    0.777    0.129    0.777    0.129 {method 'recv_into' of '_socket.socket' objects}

        5    0.772    0.154    0.773    0.155 {built-in method SSL_do_handshake}

       10    0.504    0.050    0.504    0.050 {method 'read' of '_ssl._SSLSocket' objects}

        4    0.268    0.067    0.268    0.067 {method 'recv' of '_socket.socket' objects}

        2    0.231    0.115    0.231    0.115 {method 'do_handshake' of '_ssl._SSLSocket' objects}

        5    0.087    0.017    0.087    0.017 {built-in method SSL_CTX_load_verify_locations}

        3    0.070    0.023    0.070    0.023 <frozen importlib._bootstrap>:1598(get_data)

        1    0.048    0.048    0.122    0.122 C:UsersdevanditPycharmProjectswhovenvscrapslibsite-packagespyphen__init__.py:117(__init__)

        3    0.028    0.009    0.028    0.009 {built-in method open}

    11152    0.025    0.000    0.025    0.000 {method 'findall' of '_sre.SRE_Pattern' objects}

        1    0.024    0.024    0.071    0.071 C:UsersdevanditAppDataRoamingPythonPython34site-packagestextblob_text.py:732(load)

    11129    0.023    0.000    0.023    0.000 C:UsersdevanditPycharmProjectswhovenvscrapslibsite-packagespyphen__init__.py:149(<listcomp>)

        1    0.015    0.015    0.015    0.015 {method '_parse_whole' of 'xml.etree.ElementTree.XMLParser' objects}

        6    0.008    0.001    0.011    0.002 C:UsersdevanditAppDataRoamingPythonPython34site-packagesurllib3connectionpool.py:407(close)

    11295    0.008    0.000    0.008    0.000 {method 'sub' of '_sre.SRE_Pattern' objects}

    16296    0.007    0.000    0.010    0.000 C:UsersdevanditAppDataRoamingPythonPython34site-packagestextblob_text.py:688(avg)

32008/31977    0.006    0.000    0.013    0.000 {built-in method isinstance}

      190    0.005    0.000    0.008    0.000 C:UsersdevanditPycharmProjectswhovenvscrapslibsite-packagespyphen__init__.py:168(positions)

    34615    0.005    0.000    0.005    0.000 {method 'startswith' of 'str' objects}

     7775    0.005    0.000    0.005    0.000 C:UsersdevanditPycharmProjectswhovenvscrapslibsite-packagestextstattextstat.py:41(<genexpr>)

    11336    0.005    0.000    0.007    0.000 {built-in method max}

1785/1160    0.004    0.000    0.010    0.000 C:UsersdevanditAppDataRoamingPythonPython34site-packagesbs4element.py:1792(_matches)

     8067    0.004    0.000    0.007    0.000 C:UsersdevanditPycharmProjectswhovenvscrapslibsite-packageswhois__init__.py:77(<genexpr>)

11834/11830    0.004    0.000    0.004    0.000 {method 'decode' of 'bytes' objects}

       35    0.003    0.000    0.003    0.000 {built-in method stat}

    30138    0.003    0.000    0.003    0.000 {method 'get' of 'dict' objects}

edited Nov 22 at 17:57

davidism

61.5k12156175

asked Nov 22 at 14:55

Jason Martinez

2416

1

Hi, tyre slowdown might be just related to network overhead (read, recv, do_handshake) are you accessing the server locally or remotely? Also, you might want to check if you can use HTTP/2 as it reduces SSL overhead of multiple requests. Posting your relevant code would be appreciated.
– lolopop
Nov 22 at 20:35

I don't think you know the requests package because your saying to slow down and to show you code? Its requests so 7 different times I said requests.get(site) the site being different locations.
– Jason Martinez
Nov 23 at 3:10

So what evidence do you have that the code of requests itself is slowing this down significantly, i.e. that this is not due to the other parties involved which are (I would have thought) much greater unknowns, i.e. the internet and the remote server? And even if it is Requests, why ask here? You have the source code so you could try fix it, or you can raise an issue on the requests github... or you could evn try uninstalling PyOpenSSL to see if requests falls back to something better as implied by the redhat issue you link to.
– barny
Nov 23 at 11:01

add a comment |

up vote
-2
down vote

favorite

I did research and it seems like it has something to do with the requests module using PyopenSSl see here: https://bugzilla.redhat.com/show_bug.cgi?id=1567862#c11

But I dont see a way to fix, any assistance is greatly appreciated.

Ordered by: internal time, call count

   List reduced from 1586 to 30 due to restriction <30>



   ncalls  tottime  percall  cumtime  percall filename:lineno(function)

       18    3.677    0.204    3.677    0.204 {built-in method SSL_read}

       12    1.492    0.124    1.493    0.124 {method 'connect' of '_socket.socket' objects}

      946    1.041    0.001    1.041    0.001 {built-in method __new__ of type object at 0x0000000055F8FD50}

       10    0.843    0.084    0.844    0.084 {built-in method getaddrinfo}

        6    0.777    0.129    0.777    0.129 {method 'recv_into' of '_socket.socket' objects}

        5    0.772    0.154    0.773    0.155 {built-in method SSL_do_handshake}

       10    0.504    0.050    0.504    0.050 {method 'read' of '_ssl._SSLSocket' objects}

        4    0.268    0.067    0.268    0.067 {method 'recv' of '_socket.socket' objects}

        2    0.231    0.115    0.231    0.115 {method 'do_handshake' of '_ssl._SSLSocket' objects}

        5    0.087    0.017    0.087    0.017 {built-in method SSL_CTX_load_verify_locations}

        3    0.070    0.023    0.070    0.023 <frozen importlib._bootstrap>:1598(get_data)

        1    0.048    0.048    0.122    0.122 C:UsersdevanditPycharmProjectswhovenvscrapslibsite-packagespyphen__init__.py:117(__init__)

        3    0.028    0.009    0.028    0.009 {built-in method open}

    11152    0.025    0.000    0.025    0.000 {method 'findall' of '_sre.SRE_Pattern' objects}

        1    0.024    0.024    0.071    0.071 C:UsersdevanditAppDataRoamingPythonPython34site-packagestextblob_text.py:732(load)

    11129    0.023    0.000    0.023    0.000 C:UsersdevanditPycharmProjectswhovenvscrapslibsite-packagespyphen__init__.py:149(<listcomp>)

        1    0.015    0.015    0.015    0.015 {method '_parse_whole' of 'xml.etree.ElementTree.XMLParser' objects}

        6    0.008    0.001    0.011    0.002 C:UsersdevanditAppDataRoamingPythonPython34site-packagesurllib3connectionpool.py:407(close)

    11295    0.008    0.000    0.008    0.000 {method 'sub' of '_sre.SRE_Pattern' objects}

    16296    0.007    0.000    0.010    0.000 C:UsersdevanditAppDataRoamingPythonPython34site-packagestextblob_text.py:688(avg)

32008/31977    0.006    0.000    0.013    0.000 {built-in method isinstance}

      190    0.005    0.000    0.008    0.000 C:UsersdevanditPycharmProjectswhovenvscrapslibsite-packagespyphen__init__.py:168(positions)

    34615    0.005    0.000    0.005    0.000 {method 'startswith' of 'str' objects}

     7775    0.005    0.000    0.005    0.000 C:UsersdevanditPycharmProjectswhovenvscrapslibsite-packagestextstattextstat.py:41(<genexpr>)

    11336    0.005    0.000    0.007    0.000 {built-in method max}

1785/1160    0.004    0.000    0.010    0.000 C:UsersdevanditAppDataRoamingPythonPython34site-packagesbs4element.py:1792(_matches)

     8067    0.004    0.000    0.007    0.000 C:UsersdevanditPycharmProjectswhovenvscrapslibsite-packageswhois__init__.py:77(<genexpr>)

11834/11830    0.004    0.000    0.004    0.000 {method 'decode' of 'bytes' objects}

       35    0.003    0.000    0.003    0.000 {built-in method stat}

    30138    0.003    0.000    0.003    0.000 {method 'get' of 'dict' objects}

edited Nov 22 at 17:57

davidism

61.5k12156175

asked Nov 22 at 14:55

Jason Martinez

2416

I did research and it seems like it has something to do with the requests module using PyopenSSl see here: https://bugzilla.redhat.com/show_bug.cgi?id=1567862#c11

But I dont see a way to fix, any assistance is greatly appreciated.

Ordered by: internal time, call count

   List reduced from 1586 to 30 due to restriction <30>



   ncalls  tottime  percall  cumtime  percall filename:lineno(function)

       18    3.677    0.204    3.677    0.204 {built-in method SSL_read}

       12    1.492    0.124    1.493    0.124 {method 'connect' of '_socket.socket' objects}

      946    1.041    0.001    1.041    0.001 {built-in method __new__ of type object at 0x0000000055F8FD50}

       10    0.843    0.084    0.844    0.084 {built-in method getaddrinfo}

        6    0.777    0.129    0.777    0.129 {method 'recv_into' of '_socket.socket' objects}

        5    0.772    0.154    0.773    0.155 {built-in method SSL_do_handshake}

       10    0.504    0.050    0.504    0.050 {method 'read' of '_ssl._SSLSocket' objects}

        4    0.268    0.067    0.268    0.067 {method 'recv' of '_socket.socket' objects}

        2    0.231    0.115    0.231    0.115 {method 'do_handshake' of '_ssl._SSLSocket' objects}

        5    0.087    0.017    0.087    0.017 {built-in method SSL_CTX_load_verify_locations}

        3    0.070    0.023    0.070    0.023 <frozen importlib._bootstrap>:1598(get_data)

        1    0.048    0.048    0.122    0.122 C:UsersdevanditPycharmProjectswhovenvscrapslibsite-packagespyphen__init__.py:117(__init__)

        3    0.028    0.009    0.028    0.009 {built-in method open}

    11152    0.025    0.000    0.025    0.000 {method 'findall' of '_sre.SRE_Pattern' objects}

        1    0.024    0.024    0.071    0.071 C:UsersdevanditAppDataRoamingPythonPython34site-packagestextblob_text.py:732(load)

    11129    0.023    0.000    0.023    0.000 C:UsersdevanditPycharmProjectswhovenvscrapslibsite-packagespyphen__init__.py:149(<listcomp>)

        1    0.015    0.015    0.015    0.015 {method '_parse_whole' of 'xml.etree.ElementTree.XMLParser' objects}

        6    0.008    0.001    0.011    0.002 C:UsersdevanditAppDataRoamingPythonPython34site-packagesurllib3connectionpool.py:407(close)

    11295    0.008    0.000    0.008    0.000 {method 'sub' of '_sre.SRE_Pattern' objects}

    16296    0.007    0.000    0.010    0.000 C:UsersdevanditAppDataRoamingPythonPython34site-packagestextblob_text.py:688(avg)

32008/31977    0.006    0.000    0.013    0.000 {built-in method isinstance}

      190    0.005    0.000    0.008    0.000 C:UsersdevanditPycharmProjectswhovenvscrapslibsite-packagespyphen__init__.py:168(positions)

    34615    0.005    0.000    0.005    0.000 {method 'startswith' of 'str' objects}

     7775    0.005    0.000    0.005    0.000 C:UsersdevanditPycharmProjectswhovenvscrapslibsite-packagestextstattextstat.py:41(<genexpr>)

    11336    0.005    0.000    0.007    0.000 {built-in method max}

1785/1160    0.004    0.000    0.010    0.000 C:UsersdevanditAppDataRoamingPythonPython34site-packagesbs4element.py:1792(_matches)

     8067    0.004    0.000    0.007    0.000 C:UsersdevanditPycharmProjectswhovenvscrapslibsite-packageswhois__init__.py:77(<genexpr>)

11834/11830    0.004    0.000    0.004    0.000 {method 'decode' of 'bytes' objects}

       35    0.003    0.000    0.003    0.000 {built-in method stat}

    30138    0.003    0.000    0.003    0.000 {method 'get' of 'dict' objects}

python python-requests pyopenssl cprofile

edited Nov 22 at 17:57

davidism

61.5k12156175

asked Nov 22 at 14:55

Jason Martinez

2416

edited Nov 22 at 17:57

davidism

61.5k12156175

asked Nov 22 at 14:55

Jason Martinez

2416

edited Nov 22 at 17:57

davidism

61.5k12156175

edited Nov 22 at 17:57

davidism

61.5k12156175

edited Nov 22 at 17:57

davidism

61.5k12156175

asked Nov 22 at 14:55

Jason Martinez

2416

asked Nov 22 at 14:55

Jason Martinez

2416

asked Nov 22 at 14:55

Jason Martinez

2416

1

Hi, tyre slowdown might be just related to network overhead (read, recv, do_handshake) are you accessing the server locally or remotely? Also, you might want to check if you can use HTTP/2 as it reduces SSL overhead of multiple requests. Posting your relevant code would be appreciated.
– lolopop
Nov 22 at 20:35

I don't think you know the requests package because your saying to slow down and to show you code? Its requests so 7 different times I said requests.get(site) the site being different locations.
– Jason Martinez
Nov 23 at 3:10

So what evidence do you have that the code of requests itself is slowing this down significantly, i.e. that this is not due to the other parties involved which are (I would have thought) much greater unknowns, i.e. the internet and the remote server? And even if it is Requests, why ask here? You have the source code so you could try fix it, or you can raise an issue on the requests github... or you could evn try uninstalling PyOpenSSL to see if requests falls back to something better as implied by the redhat issue you link to.
– barny
Nov 23 at 11:01

add a comment |

1

Hi, tyre slowdown might be just related to network overhead (read, recv, do_handshake) are you accessing the server locally or remotely? Also, you might want to check if you can use HTTP/2 as it reduces SSL overhead of multiple requests. Posting your relevant code would be appreciated.
– lolopop
Nov 22 at 20:35

I don't think you know the requests package because your saying to slow down and to show you code? Its requests so 7 different times I said requests.get(site) the site being different locations.
– Jason Martinez
Nov 23 at 3:10

So what evidence do you have that the code of requests itself is slowing this down significantly, i.e. that this is not due to the other parties involved which are (I would have thought) much greater unknowns, i.e. the internet and the remote server? And even if it is Requests, why ask here? You have the source code so you could try fix it, or you can raise an issue on the requests github... or you could evn try uninstalling PyOpenSSL to see if requests falls back to something better as implied by the redhat issue you link to.
– barny
Nov 23 at 11:01

Hi, tyre slowdown might be just related to network overhead (read, recv, do_handshake) are you accessing the server locally or remotely? Also, you might want to check if you can use HTTP/2 as it reduces SSL overhead of multiple requests. Posting your relevant code would be appreciated.
– lolopop
Nov 22 at 20:35

I don't think you know the requests package because your saying to slow down and to show you code? Its requests so 7 different times I said requests.get(site) the site being different locations.
– Jason Martinez
Nov 23 at 3:10

So what evidence do you have that the code of requests itself is slowing this down significantly, i.e. that this is not due to the other parties involved which are (I would have thought) much greater unknowns, i.e. the internet and the remote server? And even if it is Requests, why ask here? You have the source code so you could try fix it, or you can raise an issue on the requests github... or you could evn try uninstalling PyOpenSSL to see if requests falls back to something better as implied by the redhat issue you link to.
– barny
Nov 23 at 11:01

add a comment |

active

oldest

votes

Your Answer

StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});

}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53433574%2fcprofiling-a-route-in-python-flask%23new-answer', 'question_page');
}
);

Post as a guest

Name

Required, but never shown

active

oldest

votes

draft saved

draft discarded

Thanks for contributing an answer to Stack Overflow!

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

Some of your past answers have not been well-received, and you're in danger of being blocked from answering.

Please pay close attention to the following guidance:

Please be sure to answer the question. Provide details and share your research!

But avoid …

Asking for help, clarification, or responding to other answers.

Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Post as a guest

Name

Required, but never shown

Name

Required, but never shown

Name

Required, but never shown

This page is only for reference, If you need detailed information, please check here

搜尋此網誌

Qfyilyi