# Creating an i/o pipeline which feeds a while loop using multiple cores
**URL:** https://discourse.julialang.org/t/creating-an-i-o-pipeline-which-feeds-a-while-loop-using-multiple-cores/31931
**Category:** General Usage
**Created:** [December 6, 2019, 3:29pm UTC](https://discourse.julialang.org/t/creating-an-i-o-pipeline-which-feeds-a-while-loop-using-multiple-cores/31931 "2019-12-06T15:29:47Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@anon97116444](https://discourse.julialang.org/u/anon97116444)
#### Post date: [December 6, 2019, 3:29pm UTC](https://discourse.julialang.org/t/creating-an-i-o-pipeline-which-feeds-a-while-loop-using-multiple-cores/31931/1 "2019-12-06T15:29:47Z")
no code written YET but getting there. So I am having a little performance anxiety and I think it’s because I’m not seeing the big picture yet. Could someone point me at ANY Julia modules I should look at for the following.
I have prepared a text file of 500mb that has mixed lines in it:
1. all lines are about 300 characters long
2. comment lines
3. comma separated lines
4. space separated lines
5. lines can contain floats and integers
I want to pipe the text file into an I/O stream into a memory buffer.
another way to look at it
using the CLI pipe a text file into a julia process.
the julia process sees the bytes flow in and puts them into a memory structure
the memory structure fills up until it has enough work to facilitate a worker
the worker looks for a place to work and removes the bytes (work) from the memory structure
this process is repeated until all the bytes are processed.
the parsers output to a queue which feeds other processes.
---
### Author:  [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)
#### Post date: [December 6, 2019, 3:44pm UTC](https://discourse.julialang.org/t/creating-an-i-o-pipeline-which-feeds-a-while-loop-using-multiple-cores/31931/2 "2019-12-06T15:44:53Z")
Not exactly what you are looking for, but in the same vein
[https://github.com/baggepinnen/LengthChannels.jl](https://github.com/baggepinnen/LengthChannels.jl)
---
### Author:  [@anon97116444](https://discourse.julialang.org/u/anon97116444)
#### Post date: [December 6, 2019, 4:21pm UTC](https://discourse.julialang.org/t/creating-an-i-o-pipeline-which-feeds-a-while-loop-using-multiple-cores/31931/3 "2019-12-06T16:21:30Z")
@baggepinnen
thanks for the lead, first glance it doesn’t seem a fit BUT it’s a GREAT example. I’ll try to clean up my needs but the concept is:
using the CLI pipe a text file into a julia process.
the julia process sees the bytes flow in and puts them into a memory structure
the memory structure fills up until it has enough work to facilitate a worker
the worker looks for a place to work and removes the bytes (work) from the memory structure
this process is repeated until all the bytes are processed.
does that clear it up? if so I will update the question.
---
### Author:  [@jpsamaroo](https://discourse.julialang.org/u/jpsamaroo)
#### Post date: [December 6, 2019, 4:26pm UTC](https://discourse.julialang.org/t/creating-an-i-o-pipeline-which-feeds-a-while-loop-using-multiple-cores/31931/4 "2019-12-06T16:26:45Z")
I would recommend attempting a first pass at writing the code for this first, and then optimize it after as needed. It’s also hard for others to provide help if the code doesn’t already exist 🙂
---
### Author:  [@anon97116444](https://discourse.julialang.org/u/anon97116444)
#### Post date: [December 6, 2019, 4:39pm UTC](https://discourse.julialang.org/t/creating-an-i-o-pipeline-which-feeds-a-while-loop-using-multiple-cores/31931/5 "2019-12-06T16:39:44Z")
@jpsamaroo Hi there, I understand but I just wanted to set off on the right path. So what I was looking for was guidance as to what packages might be interesting to look at for the tasks outlined. I do see the point you are making but I thought part of Discourse was to offer guidance to people with concepts not just code. This is the third time I have got a question wrong on Discourse 🙂 Ho hum
---
### Author:  [@anon97116444](https://discourse.julialang.org/u/anon97116444)
#### Post date: [December 6, 2019, 5:01pm UTC](https://discourse.julialang.org/t/creating-an-i-o-pipeline-which-feeds-a-while-loop-using-multiple-cores/31931/6 "2019-12-06T17:01:08Z")
I think datastreams.jl is the way ahead for me.
[ ](https://www.youtube.com/watch?v=_XjtR_Jbl6Y)
---
### Author:  [@venuur](https://discourse.julialang.org/u/venuur)
#### Post date: [December 8, 2019, 12:18am UTC](https://discourse.julialang.org/t/creating-an-i-o-pipeline-which-feeds-a-while-loop-using-multiple-cores/31931/7 "2019-12-08T00:18:45Z")
> [@anon97116444](#):
>
> I do see the point you are making but I thought part of Discourse was to offer guidance to people with concepts not just code. This is the third time I have got a question wrong on Discourse 🙂 Ho hum
You seem to feel discouraged by the response. I’m sorry to hear that.
I think the challenge is that without a simple example it’s hard be sure that we understand your requirements. In particular, performance tuning is very situation specific as it depends on both the algorithms and data structures used and the peculiar features of the Julia language.
Good luck, and thanks for sharing the link for your potential solution!