How does the asyncio module work, why is my updated sample running synchronously?
up vote
2
down vote
favorite
I have tried the following code in Python 3.6 for asyncio:
Example 1:
import asyncio
import time
async def hello():
print('hello')
await asyncio.sleep(1)
print('hello again')
tasks=[hello(),hello()]
loop=asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
Output is as expected:
hello
hello
hello again
hello again
Then I want to change the asyncio.sleep into another def:
async def sleep():
time.sleep(1)
async def hello():
print('hello')
await sleep()
print('hello again')
tasks=[hello(),hello()]
loop=asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
Output:
hello
hello again
hello
hello again
It seems it is not running in an asynchronous mode, but a normal sync mode.
The question is: Why is it not running in an asynchronous mode and how can I change the old sync module into an 'async' one?
python async-await python-asyncio
add a comment |
up vote
2
down vote
favorite
I have tried the following code in Python 3.6 for asyncio:
Example 1:
import asyncio
import time
async def hello():
print('hello')
await asyncio.sleep(1)
print('hello again')
tasks=[hello(),hello()]
loop=asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
Output is as expected:
hello
hello
hello again
hello again
Then I want to change the asyncio.sleep into another def:
async def sleep():
time.sleep(1)
async def hello():
print('hello')
await sleep()
print('hello again')
tasks=[hello(),hello()]
loop=asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
Output:
hello
hello again
hello
hello again
It seems it is not running in an asynchronous mode, but a normal sync mode.
The question is: Why is it not running in an asynchronous mode and how can I change the old sync module into an 'async' one?
python async-await python-asyncio
add a comment |
up vote
2
down vote
favorite
up vote
2
down vote
favorite
I have tried the following code in Python 3.6 for asyncio:
Example 1:
import asyncio
import time
async def hello():
print('hello')
await asyncio.sleep(1)
print('hello again')
tasks=[hello(),hello()]
loop=asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
Output is as expected:
hello
hello
hello again
hello again
Then I want to change the asyncio.sleep into another def:
async def sleep():
time.sleep(1)
async def hello():
print('hello')
await sleep()
print('hello again')
tasks=[hello(),hello()]
loop=asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
Output:
hello
hello again
hello
hello again
It seems it is not running in an asynchronous mode, but a normal sync mode.
The question is: Why is it not running in an asynchronous mode and how can I change the old sync module into an 'async' one?
python async-await python-asyncio
I have tried the following code in Python 3.6 for asyncio:
Example 1:
import asyncio
import time
async def hello():
print('hello')
await asyncio.sleep(1)
print('hello again')
tasks=[hello(),hello()]
loop=asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
Output is as expected:
hello
hello
hello again
hello again
Then I want to change the asyncio.sleep into another def:
async def sleep():
time.sleep(1)
async def hello():
print('hello')
await sleep()
print('hello again')
tasks=[hello(),hello()]
loop=asyncio.get_event_loop()
loop.run_until_complete(asyncio.wait(tasks))
Output:
hello
hello again
hello
hello again
It seems it is not running in an asynchronous mode, but a normal sync mode.
The question is: Why is it not running in an asynchronous mode and how can I change the old sync module into an 'async' one?
python async-await python-asyncio
python async-await python-asyncio
edited Jul 26 '17 at 7:32
Martijn Pieters♦
694k12924032244
694k12924032244
asked Jul 26 '17 at 7:11
user7936694
183
183
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
up vote
8
down vote
accepted
Asyncio uses an event loop, which selects what task (an independent call chain of coroutines) in the queue to activate next. The event loop can make intelligent decisions as to what task is ready to do actual work. This is why the event loop also is responsible for creating connections and watching file descriptors and other I/O primitives; it gives the event loop insight into when there are I/O operations in progress or when results are available to process.
Whenever you use await
, there is an opportunity to return control to the loop which can then pass control to another task. Which task then is picked for execution depends on the exact implementation; the asyncio
reference implementation offers multiple choices, but there are other implementations, such as the very, very efficient uvloop implementation.
Your sample is still asynchronous. It just so happens that by replacing the await.sleep()
with a synchronous time.sleep()
call, inside a new coroutine function, you introduced 2 coroutines into the task callchain that don't yield, and thus influenced in what order they are executed. That they are executed in what appears to be synchronous order is a coincidence. If you switched event loops, or introduced more coroutines (especially some that use I/O), the order can easily be different again.
Moreover, your new coroutines use time.sleep()
; this makes your coroutines uncooperative. The event loop is not notified that your code is waiting (time.sleep()
will not yield!), so no other coroutine can be executed while time.sleep()
is running. time.sleep()
simply doesn't return or lets any other code run until the requested amount of time has passed. Contrast this with the asyncio.sleep()
implementation, which simply yields to the event loop with a call_later()
hook; the event loop now knows that that task won't need any attention until a later time.
Also see asyncio: why isn't it non-blocking by default for a more in-depth discussion of how tasks and the event loop interact. And if you must run blocking, synchronous code that can't be made to cooperate, then use an executor pool to have the blocking code executed in a separate tread or child process to free up the event loop for other, better behaved tasks.
Hi Martijn, Thanks a lot for the answer. then if I have an old sync def or method, how can I change them into an asynchronous one?
– user7936694
Jul 26 '17 at 13:22
You must either rewrite the function in async manner (there are many ready libs e.g. for pg, redis, http... mainly some io stuff) or use executors (mainly for use with cpu-intensive stuff) likeThreadPoolExecutro
andProcessPoolExecutor
(docs.python.org/3/library/…). More examples can be found at pymotw.com/3/asyncio/executors.html
– kwarunek
Jul 26 '17 at 14:39
@user7936694: exactly what kwarunek said; put code you can't make async in an executor, which then pretends your code is async by executing it under a different thread or in a separate process.
– Martijn Pieters♦
Jul 26 '17 at 14:48
Hi Martijn, thank a lot, seems it will take a while to change my old code into an asynchronous one :)
– user7936694
Jul 26 '17 at 19:01
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
8
down vote
accepted
Asyncio uses an event loop, which selects what task (an independent call chain of coroutines) in the queue to activate next. The event loop can make intelligent decisions as to what task is ready to do actual work. This is why the event loop also is responsible for creating connections and watching file descriptors and other I/O primitives; it gives the event loop insight into when there are I/O operations in progress or when results are available to process.
Whenever you use await
, there is an opportunity to return control to the loop which can then pass control to another task. Which task then is picked for execution depends on the exact implementation; the asyncio
reference implementation offers multiple choices, but there are other implementations, such as the very, very efficient uvloop implementation.
Your sample is still asynchronous. It just so happens that by replacing the await.sleep()
with a synchronous time.sleep()
call, inside a new coroutine function, you introduced 2 coroutines into the task callchain that don't yield, and thus influenced in what order they are executed. That they are executed in what appears to be synchronous order is a coincidence. If you switched event loops, or introduced more coroutines (especially some that use I/O), the order can easily be different again.
Moreover, your new coroutines use time.sleep()
; this makes your coroutines uncooperative. The event loop is not notified that your code is waiting (time.sleep()
will not yield!), so no other coroutine can be executed while time.sleep()
is running. time.sleep()
simply doesn't return or lets any other code run until the requested amount of time has passed. Contrast this with the asyncio.sleep()
implementation, which simply yields to the event loop with a call_later()
hook; the event loop now knows that that task won't need any attention until a later time.
Also see asyncio: why isn't it non-blocking by default for a more in-depth discussion of how tasks and the event loop interact. And if you must run blocking, synchronous code that can't be made to cooperate, then use an executor pool to have the blocking code executed in a separate tread or child process to free up the event loop for other, better behaved tasks.
Hi Martijn, Thanks a lot for the answer. then if I have an old sync def or method, how can I change them into an asynchronous one?
– user7936694
Jul 26 '17 at 13:22
You must either rewrite the function in async manner (there are many ready libs e.g. for pg, redis, http... mainly some io stuff) or use executors (mainly for use with cpu-intensive stuff) likeThreadPoolExecutro
andProcessPoolExecutor
(docs.python.org/3/library/…). More examples can be found at pymotw.com/3/asyncio/executors.html
– kwarunek
Jul 26 '17 at 14:39
@user7936694: exactly what kwarunek said; put code you can't make async in an executor, which then pretends your code is async by executing it under a different thread or in a separate process.
– Martijn Pieters♦
Jul 26 '17 at 14:48
Hi Martijn, thank a lot, seems it will take a while to change my old code into an asynchronous one :)
– user7936694
Jul 26 '17 at 19:01
add a comment |
up vote
8
down vote
accepted
Asyncio uses an event loop, which selects what task (an independent call chain of coroutines) in the queue to activate next. The event loop can make intelligent decisions as to what task is ready to do actual work. This is why the event loop also is responsible for creating connections and watching file descriptors and other I/O primitives; it gives the event loop insight into when there are I/O operations in progress or when results are available to process.
Whenever you use await
, there is an opportunity to return control to the loop which can then pass control to another task. Which task then is picked for execution depends on the exact implementation; the asyncio
reference implementation offers multiple choices, but there are other implementations, such as the very, very efficient uvloop implementation.
Your sample is still asynchronous. It just so happens that by replacing the await.sleep()
with a synchronous time.sleep()
call, inside a new coroutine function, you introduced 2 coroutines into the task callchain that don't yield, and thus influenced in what order they are executed. That they are executed in what appears to be synchronous order is a coincidence. If you switched event loops, or introduced more coroutines (especially some that use I/O), the order can easily be different again.
Moreover, your new coroutines use time.sleep()
; this makes your coroutines uncooperative. The event loop is not notified that your code is waiting (time.sleep()
will not yield!), so no other coroutine can be executed while time.sleep()
is running. time.sleep()
simply doesn't return or lets any other code run until the requested amount of time has passed. Contrast this with the asyncio.sleep()
implementation, which simply yields to the event loop with a call_later()
hook; the event loop now knows that that task won't need any attention until a later time.
Also see asyncio: why isn't it non-blocking by default for a more in-depth discussion of how tasks and the event loop interact. And if you must run blocking, synchronous code that can't be made to cooperate, then use an executor pool to have the blocking code executed in a separate tread or child process to free up the event loop for other, better behaved tasks.
Hi Martijn, Thanks a lot for the answer. then if I have an old sync def or method, how can I change them into an asynchronous one?
– user7936694
Jul 26 '17 at 13:22
You must either rewrite the function in async manner (there are many ready libs e.g. for pg, redis, http... mainly some io stuff) or use executors (mainly for use with cpu-intensive stuff) likeThreadPoolExecutro
andProcessPoolExecutor
(docs.python.org/3/library/…). More examples can be found at pymotw.com/3/asyncio/executors.html
– kwarunek
Jul 26 '17 at 14:39
@user7936694: exactly what kwarunek said; put code you can't make async in an executor, which then pretends your code is async by executing it under a different thread or in a separate process.
– Martijn Pieters♦
Jul 26 '17 at 14:48
Hi Martijn, thank a lot, seems it will take a while to change my old code into an asynchronous one :)
– user7936694
Jul 26 '17 at 19:01
add a comment |
up vote
8
down vote
accepted
up vote
8
down vote
accepted
Asyncio uses an event loop, which selects what task (an independent call chain of coroutines) in the queue to activate next. The event loop can make intelligent decisions as to what task is ready to do actual work. This is why the event loop also is responsible for creating connections and watching file descriptors and other I/O primitives; it gives the event loop insight into when there are I/O operations in progress or when results are available to process.
Whenever you use await
, there is an opportunity to return control to the loop which can then pass control to another task. Which task then is picked for execution depends on the exact implementation; the asyncio
reference implementation offers multiple choices, but there are other implementations, such as the very, very efficient uvloop implementation.
Your sample is still asynchronous. It just so happens that by replacing the await.sleep()
with a synchronous time.sleep()
call, inside a new coroutine function, you introduced 2 coroutines into the task callchain that don't yield, and thus influenced in what order they are executed. That they are executed in what appears to be synchronous order is a coincidence. If you switched event loops, or introduced more coroutines (especially some that use I/O), the order can easily be different again.
Moreover, your new coroutines use time.sleep()
; this makes your coroutines uncooperative. The event loop is not notified that your code is waiting (time.sleep()
will not yield!), so no other coroutine can be executed while time.sleep()
is running. time.sleep()
simply doesn't return or lets any other code run until the requested amount of time has passed. Contrast this with the asyncio.sleep()
implementation, which simply yields to the event loop with a call_later()
hook; the event loop now knows that that task won't need any attention until a later time.
Also see asyncio: why isn't it non-blocking by default for a more in-depth discussion of how tasks and the event loop interact. And if you must run blocking, synchronous code that can't be made to cooperate, then use an executor pool to have the blocking code executed in a separate tread or child process to free up the event loop for other, better behaved tasks.
Asyncio uses an event loop, which selects what task (an independent call chain of coroutines) in the queue to activate next. The event loop can make intelligent decisions as to what task is ready to do actual work. This is why the event loop also is responsible for creating connections and watching file descriptors and other I/O primitives; it gives the event loop insight into when there are I/O operations in progress or when results are available to process.
Whenever you use await
, there is an opportunity to return control to the loop which can then pass control to another task. Which task then is picked for execution depends on the exact implementation; the asyncio
reference implementation offers multiple choices, but there are other implementations, such as the very, very efficient uvloop implementation.
Your sample is still asynchronous. It just so happens that by replacing the await.sleep()
with a synchronous time.sleep()
call, inside a new coroutine function, you introduced 2 coroutines into the task callchain that don't yield, and thus influenced in what order they are executed. That they are executed in what appears to be synchronous order is a coincidence. If you switched event loops, or introduced more coroutines (especially some that use I/O), the order can easily be different again.
Moreover, your new coroutines use time.sleep()
; this makes your coroutines uncooperative. The event loop is not notified that your code is waiting (time.sleep()
will not yield!), so no other coroutine can be executed while time.sleep()
is running. time.sleep()
simply doesn't return or lets any other code run until the requested amount of time has passed. Contrast this with the asyncio.sleep()
implementation, which simply yields to the event loop with a call_later()
hook; the event loop now knows that that task won't need any attention until a later time.
Also see asyncio: why isn't it non-blocking by default for a more in-depth discussion of how tasks and the event loop interact. And if you must run blocking, synchronous code that can't be made to cooperate, then use an executor pool to have the blocking code executed in a separate tread or child process to free up the event loop for other, better behaved tasks.
edited Nov 22 at 12:05
answered Jul 26 '17 at 7:30
Martijn Pieters♦
694k12924032244
694k12924032244
Hi Martijn, Thanks a lot for the answer. then if I have an old sync def or method, how can I change them into an asynchronous one?
– user7936694
Jul 26 '17 at 13:22
You must either rewrite the function in async manner (there are many ready libs e.g. for pg, redis, http... mainly some io stuff) or use executors (mainly for use with cpu-intensive stuff) likeThreadPoolExecutro
andProcessPoolExecutor
(docs.python.org/3/library/…). More examples can be found at pymotw.com/3/asyncio/executors.html
– kwarunek
Jul 26 '17 at 14:39
@user7936694: exactly what kwarunek said; put code you can't make async in an executor, which then pretends your code is async by executing it under a different thread or in a separate process.
– Martijn Pieters♦
Jul 26 '17 at 14:48
Hi Martijn, thank a lot, seems it will take a while to change my old code into an asynchronous one :)
– user7936694
Jul 26 '17 at 19:01
add a comment |
Hi Martijn, Thanks a lot for the answer. then if I have an old sync def or method, how can I change them into an asynchronous one?
– user7936694
Jul 26 '17 at 13:22
You must either rewrite the function in async manner (there are many ready libs e.g. for pg, redis, http... mainly some io stuff) or use executors (mainly for use with cpu-intensive stuff) likeThreadPoolExecutro
andProcessPoolExecutor
(docs.python.org/3/library/…). More examples can be found at pymotw.com/3/asyncio/executors.html
– kwarunek
Jul 26 '17 at 14:39
@user7936694: exactly what kwarunek said; put code you can't make async in an executor, which then pretends your code is async by executing it under a different thread or in a separate process.
– Martijn Pieters♦
Jul 26 '17 at 14:48
Hi Martijn, thank a lot, seems it will take a while to change my old code into an asynchronous one :)
– user7936694
Jul 26 '17 at 19:01
Hi Martijn, Thanks a lot for the answer. then if I have an old sync def or method, how can I change them into an asynchronous one?
– user7936694
Jul 26 '17 at 13:22
Hi Martijn, Thanks a lot for the answer. then if I have an old sync def or method, how can I change them into an asynchronous one?
– user7936694
Jul 26 '17 at 13:22
You must either rewrite the function in async manner (there are many ready libs e.g. for pg, redis, http... mainly some io stuff) or use executors (mainly for use with cpu-intensive stuff) like
ThreadPoolExecutro
and ProcessPoolExecutor
(docs.python.org/3/library/…). More examples can be found at pymotw.com/3/asyncio/executors.html– kwarunek
Jul 26 '17 at 14:39
You must either rewrite the function in async manner (there are many ready libs e.g. for pg, redis, http... mainly some io stuff) or use executors (mainly for use with cpu-intensive stuff) like
ThreadPoolExecutro
and ProcessPoolExecutor
(docs.python.org/3/library/…). More examples can be found at pymotw.com/3/asyncio/executors.html– kwarunek
Jul 26 '17 at 14:39
@user7936694: exactly what kwarunek said; put code you can't make async in an executor, which then pretends your code is async by executing it under a different thread or in a separate process.
– Martijn Pieters♦
Jul 26 '17 at 14:48
@user7936694: exactly what kwarunek said; put code you can't make async in an executor, which then pretends your code is async by executing it under a different thread or in a separate process.
– Martijn Pieters♦
Jul 26 '17 at 14:48
Hi Martijn, thank a lot, seems it will take a while to change my old code into an asynchronous one :)
– user7936694
Jul 26 '17 at 19:01
Hi Martijn, thank a lot, seems it will take a while to change my old code into an asynchronous one :)
– user7936694
Jul 26 '17 at 19:01
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f45319890%2fhow-does-the-asyncio-module-work-why-is-my-updated-sample-running-synchronously%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown