# Using Threads with I/O to processing many files in parallel

**URL:** <https://discourse.julialang.org/t/using-threads-with-i-o-to-processing-many-files-in-parallel/1112>\
**Category:** New to Julia\
**Created:** [December 23, 2016, 9:41am UTC](https://discourse.julialang.org/t/using-threads-with-i-o-to-processing-many-files-in-parallel/1112 "2016-12-23T09:41:21Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jason\_McConochie](https://avatars.discourse-cdn.com/v4/letter/j/b2d939/32.png) [@Jason\_McConochie](https://discourse.julialang.org/u/Jason_McConochie)\
**Post date:** [December 23, 2016, 9:41am UTC](https://discourse.julialang.org/t/using-threads-with-i-o-to-processing-many-files-in-parallel/1112/1 "2016-12-23T09:41:21Z")

</div>

Many cases I want to read thousands of files and do some processing, or conversion, on each file. I’m assuming I can use @threads to do this, however below test case crashes Julia (seg fault below).

```julia
function parallelFile(N::Int64)
    dPath="/tmp/Jason/"
    Threads.@threads for i=1:N
        tFile=*(dPath,"file_",@sprintf("%015d",i))
        f=open(tFile,"w")
        @printf(f,"%s\n",Dates.format(now(),"SS.ssss"))
        close(f)
    end
end

@time parallelFile(10000)

```

Crash output:  
signal (11): Segmentation fault: 11  
while loading no file, in expression starting on line 184  
unknown function (ip: 0x11453452f)  
Allocations: 1467627 (Pool: 1466754; Big: 873); GC: 0

---

<div class="post-metadata">

**Author:** ![lwabeke](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lwabeke/32/4005_2.png) [@lwabeke](https://discourse.julialang.org/u/lwabeke)\
**Post date:** [December 23, 2016, 11:48am UTC](https://discourse.julialang.org/t/using-threads-with-i-o-to-processing-many-files-in-parallel/1112/2 "2016-12-23T11:48:27Z")

</div>

I tried this. Seems to work for smaller number of parallelFiles.

Although, running it for 1000, I end up with only about 730-750 files.  
Rerunning a few times and I get the crash.

I am wondering if your running into the OS limitation of a maximum number of open files per process. Maybe if the GC is slow to fully close and clear some of the files.

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [December 23, 2016, 2:50pm UTC](https://discourse.julialang.org/t/using-threads-with-i-o-to-processing-many-files-in-parallel/1112/3 "2016-12-23T14:50:56Z")

</div>

Threaded IO is not (currently) supported.

---

<div class="post-metadata">

**Author:** ![Jason\_McConochie](https://avatars.discourse-cdn.com/v4/letter/j/b2d939/32.png) [@Jason\_McConochie](https://discourse.julialang.org/u/Jason_McConochie)\
**Post date:** [December 23, 2016, 2:52pm UTC](https://discourse.julialang.org/t/using-threads-with-i-o-to-processing-many-files-in-parallel/1112/4 "2016-12-23T14:52:18Z")

</div>

I’ve now run the same code but using @parallel - code below and it worked fine. Able to generate 100,000 files in 26 seconds.

```julia
function parallelFile(N::Int64)
    dPath="/tmp/Jason/"
    @sync @parallel for i=1:N
        tFile=*(dPath,"file_",@sprintf("%015d",i))
        f=open(tFile,"w")
        @printf(f,"%s\n",Dates.format(now(),"SS.ssss"))
        close(f)
    end
end
addprocs(4)
@time parallelFile(100000)

```