# Parallel IO

**URL:** <https://discourse.julialang.org/t/parallel-io/1390>\
**Category:** General Usage\
**Created:** [January 10, 2017, 1:09pm UTC](https://discourse.julialang.org/t/parallel-io/1390 "2017-01-10T13:09:01Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![barche](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/barche/32/79_2.png) [@barche](https://discourse.julialang.org/u/barche)\
**Post date:** [January 10, 2017, 1:09pm UTC](https://discourse.julialang.org/t/parallel-io/1390/1 "2017-01-10T13:09:01Z")

</div>

Is there something in place to support parallel IO? I’m currently thinking in the context of MPI, where it is nice to have functionality such as only printing on the first rank or prefixing lines with the process rank. Or more challenging, make sure output from different ranks is neatly separated on separate lines.

---

<div class="post-metadata">

**Author:** ![catawbasam](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/catawbasam/32/2936_2.png) [@catawbasam](https://discourse.julialang.org/u/catawbasam)\
**Post date:** [January 10, 2017, 5:24pm UTC](https://discourse.julialang.org/t/parallel-io/1390/2 "2017-01-10T17:24:00Z")

</div>

Maybe this post will help–

> [@Using Threads with I/O to processing many files in parallel](https://discourse.julialang.org/t/using-threads-with-i-o-to-processing-many-files-in-parallel/1112/2):
>
> I tried this. Seems to work for smaller number of parallelFiles. Although, running it for 1000, I end up with only about 730-750 files. Rerunning a few times and I get the crash. I am wondering if your running into the OS limitation of a maximum number of open files per process. Maybe if the GC is slow to fully close and clear some of the files.

---

<div class="post-metadata">

**Author:** ![barche](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/barche/32/79_2.png) [@barche](https://discourse.julialang.org/u/barche)\
**Post date:** [January 11, 2017, 11:07am UTC](https://discourse.julialang.org/t/parallel-io/1390/3 "2017-01-11T11:07:58Z")

</div>

Actually, I’m mostly interested in printing to the console. I found I can do it for `println` as follows:

```julia
immutable RankedIO <: Base.IO
  wrapped_stream::Base.IO
  print_rank::Int
  own_rank::Int
end

function Base.println(io::RankedIO, xs...)
  if io.print_rank == io.own_rank
    Base.println(io.wrapped_stream, xs...)
  end
end

```

This can then be used as follows:

```julia
const io = RankedIO(STDOUT, 0, myrank)
println(io, "this will only print on process rank 0")

```

I have a strong feeling there is a much better way to achieve this, however. I noticed for instance there is an `IOContext`, maybe that is the way to go? Also, is there a way to avoid passing around the `io` object all the time? And finally, this works only for `println`, so maybe there is a more basic method to override?

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [January 11, 2017, 12:59pm UTC](https://discourse.julialang.org/t/parallel-io/1390/4 "2017-01-11T12:59:38Z")

</div>

If you want to suppress printing on processes that are not rank 0, how about using `redirect_stdout` to redirect `STDOUT` to either a file or `/dev/null` on all nodes that are not rank 0.

---

<div class="post-metadata">

**Author:** ![barche](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/barche/32/79_2.png) [@barche](https://discourse.julialang.org/u/barche)\
**Post date:** [January 12, 2017, 9:56pm UTC](https://discourse.julialang.org/t/parallel-io/1390/5 "2017-01-12T21:56:46Z")

</div>

I like the idea, but what do I pass, exactly? This doesn’t work:

```julia
redirect_stdout(DevNull)

```

---

<div class="post-metadata">

**Author:** ![vchuravy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vchuravy/32/8_2.png) [@vchuravy](https://discourse.julialang.org/u/vchuravy)\
**Post date:** [January 13, 2017, 2:30am UTC](https://discourse.julialang.org/t/parallel-io/1390/6 "2017-01-13T02:30:21Z")

</div>

`DevNull` is not quite `/dev/null` but the folowwing should work `redirect_stderr(open("/dev/null", "w"))`

---

<div class="post-metadata">

**Author:** ![barche](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/barche/32/79_2.png) [@barche](https://discourse.julialang.org/u/barche)\
**Post date:** [January 13, 2017, 8:12am UTC](https://discourse.julialang.org/t/parallel-io/1390/7 "2017-01-13T08:12:48Z")

</div>

Thanks, that works. It’s a shame it’s not cross-platform, but that’s not really a concern for me at this time.