Sorry for reviving this old post. But do you have a solution now? I have a similar task that I have to download a large number of CSV file form MinIO server and process them and then save th e processed results either into same number of CSV files.
When I simply use Threads.@threads in the outer loop, I observe a very peculiar issue that the cpu usage drops along the progress and I suspect 80% of time is used for the last 10% of workloads. I allocated 16 threads and in the middle to final stage, there are only 1-2 threads at working. Does this related to the CSV files IO?