-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
prefect3.0 not running subflows in parallel #15415
Comments
update : if I make use of a wrapper task to call the subflow then the subflows are being run in parallel. import asyncio
from prefect import deploy, flow, task
from datetime import datetime, timedelta, timezone
from prefect.client.schemas.objects import FlowRun
from prefect.client.orchestration import get_client
from prefect.states import Scheduled
from prefect.task_runners import ThreadPoolTaskRunner
@task(
log_prints=True,
)
async def doit_wrapper_task(name):
return await main(name) # TASK > SUB TASK
@task(
log_prints=True
)
async def doit(name):
print('task <',name)
await asyncio.sleep(5)
return f'hello {name} !!'
@flow(
task_runner=ThreadPoolTaskRunner(max_workers=20),
log_prints=True,
)
async def main(name):
print('main <',name)
_r = doit.submit(name)
res = _r.result()
print('main >',res)
return res
@flow(
task_runner=ThreadPoolTaskRunner(max_workers=20),
log_prints=True,
)
async def master(names: list[str]):
_r = [doit_wrapper_task.submit(name) for name in names]
r = [ __r.result() for __r in _r]
print('master > ',r)
return r |
Thanks for the issue @captnced! It looks like you need an import asyncio
from prefect import flow
@flow
async def subflow_1():
print("Subflow 1 started!")
await asyncio.sleep(1)
@flow
async def main_flow():
await asyncio.gather(*[subflow_1() for _ in range(5)])
if __name__ == "__main__":
main_flow_state = asyncio.run(main_flow()) However, even with We can investigate what's blocking the event loop in the flow run engine and post updates here. If you discover anything new, please leave a comment! |
hello community !
I'm using prefect==3.0.2 and there seem to be an issue again with running subflows concurrently
this code triggers 5 subflows that are being run sequentially:
even when using workpool, workers and ad-hoc deployment, I still cannot have my subflows running in parallel:
am I doing anything wrong ?
Originally posted by @captnced in #5853 (comment)
The text was updated successfully, but these errors were encountered: