This can be rewritten with map, which might allow us to use some parallel processing from the refactor, which will help speed things up.
There is a paralleliser plugin for Pytask, which could fit nicely here to parallelise the iterations of the outer loop (i.e. processing the raw batches). This would be equivalent to creating a map and executing in parallel.
Or do you mean something more specific by "the refactor"?
Hmm, I couldn't see from the pytask paralleliser how that could parallelize within a task, just to parallelise all tasks.
I meant to refactor the loop into a map, so that we could e.g. use https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool.map which might make things faster.
The paralleliser would work by declaring each iteration as a task and then parallelising all tasks whenever possible. Under the hood, by default it would use "a more robust implementation of Pool and ProcessPoolExecutor" x. So it would be kinda equivalent to using Pool or ProcessPoolExecutor manually.
The advantages are that the backend is more robust (if we trust that statement) and that orchestrating this in pytask would give us better progress tracking / visibility over the per-batch tasks. It would also give us pytask's dependency tracking mechanism for the batch files (i.e. only rerun if inputs changed), but we might not take advantage of that if we always force run the pipeline.
Another thing to be aware of here is that Polars is multithreaded by default, so we would be piling parallelism on parallelism here. Not necessarily bad, but we would want to limit the number of total threads to roughly the number of cores to avoid "oversubscription" (i.e., having so many threads that the OS can't divvy out one to each core, and instead has to constantly switch between them).
Originally posted by @martonvago in #192
Originally posted by @martonvago in #192