# Changing file on disk not affecting IOStream data, is this a bug?

**URL:** <https://discourse.julialang.org/t/changing-file-on-disk-not-affecting-iostream-data-is-this-a-bug/53359>\
**Category:** General Usage\
**Created:** [January 14, 2021, 9:55pm UTC](https://discourse.julialang.org/t/changing-file-on-disk-not-affecting-iostream-data-is-this-a-bug/53359 "2021-01-14T21:55:50Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![tlnagy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tlnagy/32/5815_2.png) [@tlnagy](https://discourse.julialang.org/u/tlnagy)\
**Post date:** [January 14, 2021, 9:55pm UTC](https://discourse.julialang.org/t/changing-file-on-disk-not-affecting-iostream-data-is-this-a-bug/53359/1 "2021-01-14T21:55:50Z")

</div>

I’m extending my support for memory-mapped TIFF images in my package [`TiffImages.jl`](https://github.com/tlnagy/TiffImages.jl) to support writing in addition to reading. One thing I’m running into is that my structs contain a file handle that I read/write to, but this `IOStream` object behaves unexpectedly when a file changes on disk.

```julia
julia> filepath = "test.txt";

julia> io = open(filepath, read=true, append=true);

julia> write(io, "test")
4

julia> flush(io)

julia> seekstart(io);

julia> read(io, String)
"test"

```

So far, it behaves as expected. But now if I open `test.txt` on disk and change the text to `test123` that isn’t reflected in the iostream?

```julia
julia> seekstart(io);

julia> read(io, String)
"test"

```

Opening a new IOStream shows the changes:

```julia
julia> io = open(filepath, read=true, append=true);

julia> seekstart(io);

julia> read(io, String)
"test123\n"

```

**How do I get the IOStream to reflect the file on disk?**

This is on Julia 1.5.3

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [January 14, 2021, 10:05pm UTC](https://discourse.julialang.org/t/changing-file-on-disk-not-affecting-iostream-data-is-this-a-bug/53359/2 "2021-01-14T22:05:18Z")

</div>

Can’t confirm this behaviour on windows:

```julia
julia> versioninfo()
Julia Version 1.5.3
Commit 788b2c77c1 (2020-11-09 13:37 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: AMD Ryzen 9 3900X 12-Core Processor
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, znver2)

```

The changes I do with an editor are shown everytime in the REPL when I do a

```julia
julia> seekstart(io);read(io, String)
"test123"

```

The file is on a local SSD (d:\temp), of course, not a network file system or a RAID or something like that, where some buffers with higher latency may be causing such an issue.

---

<div class="post-metadata">

**Author:** ![tlnagy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tlnagy/32/5815_2.png) [@tlnagy](https://discourse.julialang.org/u/tlnagy)\
**Post date:** [January 14, 2021, 10:06pm UTC](https://discourse.julialang.org/t/changing-file-on-disk-not-affecting-iostream-data-is-this-a-bug/53359/3 "2021-01-14T22:06:09Z")

</div>

I’m on Fedora 33. Writing to a local SSD.

```julia
julia> versioninfo()
Julia Version 1.5.3
Commit 788b2c77c1* (2020-11-09 13:37 UTC)
Platform Info:
  OS: Linux (x86_64-redhat-linux)
  CPU: Intel(R) Core(TM) i7-10710U CPU @ 1.10GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)

```

---

<div class="post-metadata">

**Author:** ![tlnagy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tlnagy/32/5815_2.png) [@tlnagy](https://discourse.julialang.org/u/tlnagy)\
**Post date:** [January 14, 2021, 10:21pm UTC](https://discourse.julialang.org/t/changing-file-on-disk-not-affecting-iostream-data-is-this-a-bug/53359/4 "2021-01-14T22:21:13Z")

</div>

This also happens on a different machine (Ubuntu 20.04.01 with Julia 1.4.2)

```julia
julia> filepath = "test.txt";

julia> io = open(filepath, read=true, append=true);

julia> write(io, "test")
4

julia> flush(io)

julia> seekstart(io);

julia> read(io, String)
"test"

shell> cat test.txt
test
shell> vim test.txt

shell> cat test.txt
test123

julia> seekstart(io); read(io, String)
"test"

```

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [January 14, 2021, 11:27pm UTC](https://discourse.julialang.org/t/changing-file-on-disk-not-affecting-iostream-data-is-this-a-bug/53359/5 "2021-01-14T23:27:11Z")

</div>

can’t repro on Linux (Arch), with Julia 1.5.3

---

<div class="post-metadata">

**Author:** ![sethaxen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sethaxen/32/35604_2.png) [@sethaxen](https://discourse.julialang.org/u/sethaxen)\
**Post date:** [January 15, 2021, 12:09am UTC](https://discourse.julialang.org/t/changing-file-on-disk-not-affecting-iostream-data-is-this-a-bug/53359/6 "2021-01-15T00:09:01Z")

</div>

I was able to reproduce this

```julia
julia> versioninfo()
Julia Version 1.5.3
Commit 788b2c77c1 (2020-11-09 13:37 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.7.0)
  CPU: Intel(R) Core(TM) i5-3470S CPU @ 2.90GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, ivybridge)
Environment:
  JULIA_EDITOR = code

```

---

<div class="post-metadata">

**Author:** ![pixel27](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pixel27/32/8902_2.png) [@pixel27](https://discourse.julialang.org/u/pixel27)\
**Post date:** [January 15, 2021, 4:11am UTC](https://discourse.julialang.org/t/changing-file-on-disk-not-affecting-iostream-data-is-this-a-bug/53359/7 "2021-01-15T04:11:38Z")

</div>

> <https://stackoverflow.com/questions/39347275/fread-isnt-reading-an-updated-value-despite-the-file-has-been-changed-by-another>

I believe Julia uses fread/fwrite when accessing IOStream files and the standard library would be buffering the data in that case. Why everyone’s not seeing the buffering is beyond me.

Another thread that might be helpful:

> **[What happens fwrite/fread at the same time?](https://www.unix.com/programming/154286-what-happens-fwrite-fread-same-time.html)**
>
> Hello, I have a question about what happens when I copy the file which is being written by another process on Solaris 9/SPARC, UFS file system. in particular, I want to know what happens while some pr | The UNIX and Linux Forums

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [January 15, 2021, 11:59am UTC](https://discourse.julialang.org/t/changing-file-on-disk-not-affecting-iostream-data-is-this-a-bug/53359/8 "2021-01-15T11:59:59Z")

</div>

I was able to reproduce and analyze this behaviour on a debian system.

The reason for this, in my experiments, is, that changing the content of the file while it is opened within Julia, e.g. with vim, does not only change the file, but deletes it and writes a new version. The deletion just deletes the inode to the original data, whereas the file descriptor opened within Julia still points to the original data on the filesystem which still exists. The changed data (the new test.txt) is now saved at a different place on the filesystem and test.txt is a new inode to the new data.

> if any process has the file open when this happens, deletion is postponed until all processes have closed the file  
> [Deleting Files (The GNU C Library)](https://www.gnu.org/software/libc/manual/html_node/Deleting-Files.html)

See the following:

```julia
julia> filepath = "test.txt"; io = open(filepath, read=true, append=true);

julia> seekstart(io);read(io, String)
"test\n"

```

Now we change in another shell the file test.txt and see what happens to the file descriptor:

```julia
root:~# ps -efl | grep julia | grep -v grep
4 S root 1001 29194 10 80 0 - 125484 ep_pol 12:30 pts/1 00:00:00 julia-1.5.3/bin/julia

```

pid of our julia process is 1001 :

```julia
root:~# ll /proc/1001/fd/
...
lrwx------ 1 root root 64 Jan 15 12:31 20 -> /root/test.txt
...

```

You see an open file descriptor called `20` which points to the open file (and some more which I have removed here).  
Now we edit the content of the file with vim and save it, now we get:

```julia
root:~# ll /proc/1001/fd/
...
lrwx------ 1 root root 64 Jan 15 12:31 20 -> (deleted)/root/test.txt~
...

```

Which means, the original file, the one which is still opened in Julia, has been deleted and what is left is a symbolic link to the original place of the (unchanged) data on the filesystem.  
You can try to edit /proc/1001/fd/20 (the 20 is my special case, you will have some other name) and you will see these changes reflected with the Julia command.

Ok, lets check, what happens, if we don’t change the file with an editor, but just append some data:

```julia
julia> close(io)

julia> filepath = "test.txt"; io = open(filepath, read=true, append=true);

julia> seekstart(io);read(io, String)
"test\n"

```

Second shell:

```julia
root:~# ll /proc/1001/fd/
...
lrwx------ 1 root root 64 Jan 15 12:31 20 -> /root/test.txt
...
root:~# echo "123" >> test.txt
root:~# ll /proc/1001/fd/
...
lrwx------ 1 root root 64 Jan 15 12:31 20 -> /root/test.txt
...

```

```julia
julia> seekstart(io);read(io, String)
"test\n123\n"

```

Voila, changes show up.

Of course, Windows is behaving differently, so it couldn’t be reproduced. What Linux (Arch) does, I don’t know, but apparently also something else or @jling didn’t edit with an editor.  
The complete story is quite complex and can be searched with keywords: “linux delete unlink inodes”.

---

<div class="post-metadata">

**Author:** ![pixel27](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pixel27/32/8902_2.png) [@pixel27](https://discourse.julialang.org/u/pixel27)\
**Post date:** [January 15, 2021, 3:50pm UTC](https://discourse.julialang.org/t/changing-file-on-disk-not-affecting-iostream-data-is-this-a-bug/53359/9 "2021-01-15T15:50:47Z")

</div>

I feel I should point out that reading from a file while another process is writing to the file is risky at best. Unless you have control of both programs and are able to synchronize your reads and writes there is a very real possibility that when reading you only get half the changes because the other program is still writing the changes.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [January 15, 2021, 5:02pm UTC](https://discourse.julialang.org/t/changing-file-on-disk-not-affecting-iostream-data-is-this-a-bug/53359/10 "2021-01-15T17:02:46Z")

</div>

I did edit with an editor (`vim` in my case)

```julia
julia> io = open("test.txt", read=true, append=true);

julia> seekstart(io);

julia> read(io, String)
"test123\n"

shell> vim test.txt

julia> seekstart(io);

julia> read(io, String)
"test123\n123\n"

```

---

<div class="post-metadata">

**Author:** ![tlnagy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tlnagy/32/5815_2.png) [@tlnagy](https://discourse.julialang.org/u/tlnagy)\
**Post date:** [January 18, 2021, 6:39pm UTC](https://discourse.julialang.org/t/changing-file-on-disk-not-affecting-iostream-data-is-this-a-bug/53359/11 "2021-01-18T18:39:22Z")

</div>

Hi all, thanks for the feedback. Thanks to @oheil’s careful sleuthing, I’m pretty sure the delete/write new file is the reason for the behavior I’m seeing, but it is curious that @jling can’t reproduce on Arch. Maybe they’re doing something different?

I guess in that case, what’s the best solution for my strategy for memory-mapping? I don’t think I can use the built-in memory-mapping facilities because of the complex layout of TIFFs on disk so my plan was to just read/write to disk on getindex/setindex calls, but now I’m questioning if that’s the right strategy given the behavior observed here. Any one have thoughts?

---

<div class="post-metadata">

**Author:** ![pixel27](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pixel27/32/8902_2.png) [@pixel27](https://discourse.julialang.org/u/pixel27)\
**Post date:** [January 19, 2021, 12:23am UTC](https://discourse.julialang.org/t/changing-file-on-disk-not-affecting-iostream-data-is-this-a-bug/53359/12 "2021-01-19T00:23:35Z")

</div>

I think your best bet will be to watch the file/directory with:

[https://docs.julialang.org/en/v1/stdlib/FileWatching/](https://docs.julialang.org/en/v1/stdlib/FileWatching/)

When you notice a change reopen the file and read the contents. That will ensure that you get the latest data from the file.