# Lazy read string from iostream?
**URL:** https://discourse.julialang.org/t/lazy-read-string-from-iostream/106782
**Category:** General Usage
**Tags:** question, strings, io
**Created:** [November 27, 2023, 6:06am UTC](https://discourse.julialang.org/t/lazy-read-string-from-iostream/106782 "2023-11-27T06:06:43Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@ryofurue](https://discourse.julialang.org/u/ryofurue)
#### Post date: [November 27, 2023, 6:06am UTC](https://discourse.julialang.org/t/lazy-read-string-from-iostream/106782/1 "2023-11-27T06:06:43Z")
It seems that `read(iostream, String)` reads the entire file into the memory at once. (Please correct me if I’m mistaken.) I’ve just deduced that from this test code:
```julia
f = open("tmp.txt", "r")
s = read(f, String)
typeof(s) #-> just a String.
m = match(someregex, s)
```
Is there an idiom to read a string lazily from an iostream? You sometimes want to read from a pipe and stop reading as soon as you find the substring you wanted. Such a solution would be more general than reading the whole thing at once.
I learned that `read(stream, String)` is what is recommended:
> [@Replacing readstring](https://discourse.julialang.org/t/replacing-readstring/71746):
>
> Trying to run [https://gist.github.com/rafaqz/fede683a3e853f36c9b367471fde2f56](https://gist.github.com/rafaqz/fede683a3e853f36c9b367471fde2f56) with Julia 1.6.1 fails because of the lines filename = string(ARGS[1]) code = readstring(filename) Julia says ERROR: LoadError: UndefVarError: readstring not defined Stacktrace: [1] top-level scope @ c:\julia\fortran-julia.jl:104 in expression starting at c:\julia\fortran-julia.jl:104 I know readstring has been removed and that I should call read, but I don’t know what the alternative code should be.
---
### Author:  [@mkitti](https://discourse.julialang.org/u/mkitti)
#### Post date: [November 27, 2023, 2:27pm UTC](https://discourse.julialang.org/t/lazy-read-string-from-iostream/106782/2 "2023-11-27T14:27:14Z")
Could you be more specific about how you would like to read?
1. Do you want to read bytes? Try just `read(f)`
2. Do you want to read lines? Try `eachline(f)`.
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [November 27, 2023, 2:35pm UTC](https://discourse.julialang.org/t/lazy-read-string-from-iostream/106782/3 "2023-11-27T14:35:20Z")
normal regex engin won’t work, you need non-backtracking ones:
> **[GitHub - intel/hyperscan: High-performance regular expression matching library](https://github.com/intel/hyperscan)**
>
> High-performance regular expression matching library - GitHub - intel/hyperscan: High-performance regular expression matching library
you might be intereted:
[https://docs.julialang.org/en/v1/base/io-network/#Base.readuntil](https://docs.julialang.org/en/v1/base/io-network/#Base.readuntil)
---
### Author:  [@ryofurue](https://discourse.julialang.org/u/ryofurue)
#### Post date: [November 28, 2023, 5:21am UTC](https://discourse.julialang.org/t/lazy-read-string-from-iostream/106782/4 "2023-11-28T05:21:48Z")
> [@mkitti](#):
>
> Could you be more specific about how you would like to read?
>
> 1. Do you want to read bytes? Try just `read(f)`
> 2. Do you want to read lines? Try `eachline(f)`.
My question is more about a potential _general_ idiom you use _by default_.
Here is an analogy. I often store a numeric (eg, Float64) array in a plain binary file. Some people copy the binary data from a file into the main memory at once. What if the file is too big to store in the memory? They would say that they read the data chunk by chunk.
But I don’t do that. I just use mmap by default. In that way, I don’t have to care about whether the file is big or not.
Then, what is your default way for String? . . . Having said that, I realized, after posting my initial message, that I have to think about the danger of infinite read. What if my regular expression doesn’t match any part of the input string, which can be infinitely long if it comes from a pipe?
I would need a way to give up in the middle. This is the difference of reading from a pipe from mmap on a disk file.
That has made me realize that the default way should be either your `eachline()` or @jling 's `readuntil()`.
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [November 28, 2023, 11:16am UTC](https://discourse.julialang.org/t/lazy-read-string-from-iostream/106782/5 "2023-11-28T11:16:08Z")
> [@ryofurue](#):
>
> Here is an analogy. I often store a numeric (eg, Float64) array in a plain binary file. Some people copy the binary data from a file into the main memory at once. What if the file is too big to store in the memory? They would say that they read the data chunk by chunk.
>
> But I don’t do that. I just use mmap by default. In that way, I don’t have to care about whether the file is big or not.
>
> Then, what is your default way for String? . . . Having said that, I realized, after posting my initial message, that I have to think about the danger of infinite read. What if my regular expression doesn’t match any part of the input string, which can be infinitely long if it comes from a pipe?
my answer is use a real file format… copying memory into disk file and back is just bad for so many reasons, especially since it’s almost 2024 🙂
---
### Author:  [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [November 28, 2023, 4:47pm UTC](https://discourse.julialang.org/t/lazy-read-string-from-iostream/106782/6 "2023-11-28T16:47:24Z")
> [@ryofurue](#):
>
> But I don’t do that. I just use mmap by default. In that way, I don’t have to care about whether the file is big or not. Then, what is your default way for String? .
You can `mmap` a string using StringViews.jl
There are also ways to do `eachline` with a pre-allocated buffer, so that you don’t allocate a new string for each line. e.g. using the [ViewReader](https://github.com/rickbeeloo/ViewReader) package, or the upcoming [`copyuntil`](https://github.com/JuliaLang/julia/pull/48273) function in Julia ~~1.10~~ 1.11.
---
### Author:  [@ryofurue](https://discourse.julialang.org/u/ryofurue)
#### Post date: [November 29, 2023, 3:15am UTC](https://discourse.julialang.org/t/lazy-read-string-from-iostream/106782/7 "2023-11-29T03:15:13Z")
> [@jling](#):
>
> my answer is use a real file format
Of course! But, for temporary files, plain binary is extremely convenient. You produce temporary files in one program, use them in other programs, and delete them after all processings. Writing a plain binary is a one-liner and reading it as an mmapped array is a two-line code (plus `using Mmap`). There is no advantage in “a real file format” for this use.
In addition, some of my colleagues still produce plain binary files, which I sometimes have to use.
> [@jling](#):
>
> since it’s almost 2024 🙂
I agree! In our field, “netCDF” is the de facto standard and it’s so much better than plain binary. The problem is (or used to be) that its Fortran interface is so, so tedious to use that you are very reluctant to write a program to produce netCDF files in Fortran. I shrink from it.
Now that netCDF’s Python and Julia interfaces are so much better, there is no excuse to use plain binary, except as temporary files (see above), as long as you use Python or Julia. But, there are still Fortran-only people, even among young scientists, in our field . . .