Parallel IO

[catawbasam]

Maybe this post will help–

Using Threads with I/O to processing many files in parallel

I tried this. Seems to work for smaller number of parallelFiles. Although, running it for 1000, I end up with only about 730-750 files. Rerunning a few times and I get the crash. I am wondering if your running into the OS limitation of a maximum number of open files per process. Maybe if the GC is slow to fully close and clear some of the files.