# Safe overwriting of files

**URL:** <https://discourse.julialang.org/t/safe-overwriting-of-files/117758>\
**Category:** General Usage\
**Tags:** question, filesystem, io\
**Created:** [August 2, 2024, 7:00am UTC](https://discourse.julialang.org/t/safe-overwriting-of-files/117758 "2024-08-02T07:00:29Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [August 2, 2024, 7:00am UTC](https://discourse.julialang.org/t/safe-overwriting-of-files/117758/1 "2024-08-02T07:00:29Z")

</div>

I have a workflow where I repeatedly need to overwrite files with new content. I think the easiest way to overwrite a file is to use `write(filename,newcontent)`.

My question is how safe this function is against data loss when the julia process calling it gets interrupted unexpectedly during the write operation (e.g. someone hitting CTRL-C, a cluster manager killing process because of timeout etc…). I basically want to make sure that the file always either contains the old content or the new one but it should ideally never end up in a state where both the old and the new content of the file are lost. Is it safer to do something like:

```julia
mv(filename, filename * ".old")
write(filename,newcontent)
rm(filename * ".old")

```

so I could recover the data manually or are there already existing protection mechanisms against data loss in the former short version?

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [August 2, 2024, 8:48am UTC](https://discourse.julialang.org/t/safe-overwriting-of-files/117758/2 "2024-08-02T08:48:45Z")

</div>

The short version is indeed not data loss safe (but very convenient when that is not a significant issue).

The usual approach to improve safety is to write the new content into a temporary file and once that has succeeded (and possibly been verified by reading it back or computing a hash), move it into its target location.

---

<div class="post-metadata">

**Author:** ![sgaure](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sgaure/32/14779_2.png) [@sgaure](https://discourse.julialang.org/u/sgaure)\
**Post date:** [August 2, 2024, 9:06am UTC](https://discourse.julialang.org/t/safe-overwriting-of-files/117758/3 "2024-08-02T09:06:08Z")

</div>

The `mv` operation is atomic when used within a file system (e.g. within a directory), so you should use that. The typical pattern is as Gunnar says,

```julia
tfilename = filename * ".tmp"
write(tfilename, newcontent)
mv(tfilename, filename, force=true)

```

This will ensure that `filename` always contains valid content.

---

<div class="post-metadata">

**Author:** ![fabiangans](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/fabiangans/32/2624_2.png) [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Post date:** [August 2, 2024, 9:13am UTC](https://discourse.julialang.org/t/safe-overwriting-of-files/117758/4 "2024-08-02T09:13:17Z")

</div>

Thanks a lot to both of you, exactly the information I needed.

---

<div class="post-metadata">

**Author:** ![nhz2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nhz2/32/44428_2.png) [@nhz2](https://discourse.julialang.org/u/nhz2)\
**Post date:** [August 2, 2024, 12:08pm UTC](https://discourse.julialang.org/t/safe-overwriting-of-files/117758/5 "2024-08-02T12:08:55Z")

</div>

The `mv(...; force=true)` function in Julia is not atomic. It calls `rm` before renaming, so there can be a time where the destination file is missing. [julia/base/file.jl at c6732a79494f604e0f320ea451856a1c10511659 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/blob/c6732a79494f604e0f320ea451856a1c10511659/base/file.jl#L348)

---

<div class="post-metadata">

**Author:** ![nhz2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nhz2/32/44428_2.png) [@nhz2](https://discourse.julialang.org/u/nhz2)\
**Post date:** [August 2, 2024, 1:20pm UTC](https://discourse.julialang.org/t/safe-overwriting-of-files/117758/6 "2024-08-02T13:20:31Z")

</div>

There is a function in Python called [`os.replace`](https://docs.python.org/3/library/os.html#os.replace) that tries to be atomic but I don’t know how to do this in Julia (outside of using PythonCall)

---

<div class="post-metadata">

**Author:** ![sgaure](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sgaure/32/14779_2.png) [@sgaure](https://discourse.julialang.org/u/sgaure)\
**Post date:** [August 2, 2024, 1:49pm UTC](https://discourse.julialang.org/t/safe-overwriting-of-files/117758/7 "2024-08-02T13:49:17Z")

</div>

Wow, what a mess. Then you can replace the `mv` with

```julia
@ccall rename(tfilename::Cstring, filename::Cstring)::Cint

```

or

```julia
run(`mv $tfilename $filename`)

```

---

<div class="post-metadata">

**Author:** ![eldee](https://avatars.discourse-cdn.com/v4/letter/e/b5a626/32.png) [@eldee](https://discourse.julialang.org/u/eldee)\
**Post date:** [August 3, 2024, 12:13pm UTC](https://discourse.julialang.org/t/safe-overwriting-of-files/117758/8 "2024-08-03T12:13:49Z")

</div>

To make everything a bit more messy, both of the methods sgaure suggests above, will sadly not work on Windows, as `rename` does not overwrite and `mv` does not exist.

On Windows you can instead use  
`run(`cmd /C MOVE /Y $old_filename $new_filename`)`

I’m not sure if this is in fact atomic though, as [a quick search](https://stackoverflow.com/questions/167414/is-an-atomic-file-rename-with-overwrite-possible-on-windows) yields contradictory information.

---

<div class="post-metadata">

**Author:** ![stephancb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stephancb/32/14243_2.png) [@stephancb](https://discourse.julialang.org/u/stephancb)\
**Post date:** [August 3, 2024, 1:16pm UTC](https://discourse.julialang.org/t/safe-overwriting-of-files/117758/9 "2024-08-03T13:16:16Z")

</div>

Another possible strategy is to put your contents into a database, for example SQLite with file based storage (DuckDB also qualifies, I guess). Databases are often chosen for handling larger amounts of content with non-trivial structuring, and may feel to be an overkill in this respect for simpler data. But overlooked is frequently another advantage: SQLite provides save inserts and updates=ACID transactions, across all supported OSs, and is, owing to its enormous popularity, very thouroughly tested. Trying to overwrite files safely but “manually” is to some extend [reinventing the wheel](https://www.sqlite.org/draft/atomiccommit.html).

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [August 3, 2024, 3:17pm UTC](https://discourse.julialang.org/t/safe-overwriting-of-files/117758/10 "2024-08-03T15:17:39Z")

</div>

> [@nhz2](#):
>
> It calls `rm` before renaming, so there can be a time where the destination file is missing.

A possible low-tech workaround is to do a two-step `mv` dance where the original file is temporarily moved to a backup name and then removed after the new file is in place.

---

<div class="post-metadata">

**Author:** ![StevenSiew](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevensiew/32/218393_2.png) [@StevenSiew](https://discourse.julialang.org/u/StevenSiew)\
**Post date:** [August 4, 2024, 3:18am UTC](https://discourse.julialang.org/t/safe-overwriting-of-files/117758/11 "2024-08-04T03:18:53Z")

</div>

If you are SUPER paranoid. Then try this

> [@sgaure](#):
>
> ```julia
> tfilename = filename * ".tmp"
> write(tfilename, newcontent)
> checksum = getchecksum(filename)
> cp(filename,filename * ".old")
> verifychecksum(checksum,filename * ".old")
> checksum = getchecksum(tfilename)
> mv(tfilename, filename)
> verifychecksum(checksum,filename)
> 
> ```

---

<div class="post-metadata">

**Author:** ![nhz2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nhz2/32/44428_2.png) [@nhz2](https://discourse.julialang.org/u/nhz2)\
**Post date:** [August 6, 2024, 12:48pm UTC](https://discourse.julialang.org/t/safe-overwriting-of-files/117758/12 "2024-08-06T12:48:13Z")

</div>

I made a PR to make `mv` more atomic for this pattern to work.

> <https://github.com/JuliaLang/julia/pull/55384>
>
> As noted in https://github.com/JuliaLang/julia/issues/41584 and https://discours…e.julialang.org/t/safe-overwriting-of-files/117758/3 \`mv\` is usually expected to be "best effort atomic".
> 
> Currently calling \`mv\` with \`force=true\` calls \`checkfor\_mv\_cp\_cptree(src, dst, "moving"; force=true)\` before renaming. \`checkfor\_mv\_cp\_cptree\` will delete \`dst\` if exists and isn't the same as \`src\`.
> 
> If \`dst\` is an existing file and julia stops after deleting \`dst\` but before doing the rename, \`dst\` will be removed but will not be replaced with \`src\`.
> 
> This PR changes \`mv\` with \`force=true\` to first try rename, and only delete \`dst\` if that fails. Assuming file system support and the first rename works, julia stopping will not lead to \`dst\` being removed without being replaced.
> 
> This also replaces a stopgap solution from https://github.com/JuliaLang/julia/pull/36638#discussion\_r453820564

---

<div class="post-metadata">

**Author:** ![nhz2](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nhz2/32/44428_2.png) [@nhz2](https://discourse.julialang.org/u/nhz2)\
**Post date:** [August 16, 2024, 12:39pm UTC](https://discourse.julialang.org/t/safe-overwriting-of-files/117758/13 "2024-08-16T12:39:37Z")

</div>

I made a new PR to document `Base.rename` which is a more cross-platform version of `@ccall rename(tfilename::Cstring, filename::Cstring)::Cint`

> <https://github.com/JuliaLang/julia/pull/55503>
>
> Fixes #41584. Follow up of #55384 
> 
> This marks \`rename\` as public.