Safely accessing API tokens while automatically updating

New to Julia
1

Hello.

I’m working with setting up an API in Julia. It requires tokens, which periodically expire. I currently have my tokens saved in “tokens.json” file, which I read when initializing the code. However, I wanted to create a function that automatically updates the tokens when necessary.

I’m new to Julia, and I haven’t really used threading before, so I was wondering if this was a possibility to safely access the tokens. I.e, what’s the best way to have another process that runs in the background and automatically updates these tokens. Is it safe to load the json file in another thread, check/update, and then write back to that json file? And then how would it work to access the updated json file in other parts of my code where I’m using the tokens.

I tried Threads.Atomic but it wouldn’t let me store complex objects like Dict(). Should I do something with Threads.@spawn? Thanks.

2

When writing the json file to not end up with a partially written file.

You can do something like:

# safely update the json file `file_name` in `dir_name` directory
mktemp(dir_name) do temp_path, temp_out
    nb = write(temp_out, data)
    if nb != length(data)
        error("short write of $(repr(file_name)) data")
    end
    close(temp_out)
    mv(temp_path, joinpath(dir_name, file_name); force=true)
end

Though this may throw an error on Windows sometimes.

I would also recommend storing a checksum in the json file so you can detect if you are reading invalid data.

3

When you need the tokens just read the file.

using JSON3

json_string = read(joinpath(dir_name, file_name), String)
tokens = JSON3.read(json_string)
# check checksum in tokens here
4

What error would this throw on windows?

5

On windows if the file is being read in another thread at the same time mv will throw an error. You might have to retry mv in this case.

julia> dir_name = tempdir()
"C:\\Users\\nzimm\\AppData\\Local\\Temp"

julia> file_name = "data.txt"
"data.txt"

julia> data = "stuff"
"stuff"

julia> mktemp(dir_name) do temp_path, temp_out
           nb = write(temp_out, data)
           if nb != length(data)
               error("short write of $(repr(file_name)) data")
           end
           close(temp_out)
           mv(temp_path, joinpath(dir_name, file_name); force=true)
       end
"C:\\Users\\nzimm\\AppData\\Local\\Temp\\data.txt"

julia> read(joinpath(dir_name, file_name), String)
"stuff"

julia> data = "new stuff"
"new stuff"

julia> f = open(joinpath(dir_name, file_name))

julia> mktemp(dir_name) do temp_path, temp_out
           nb = write(temp_out, data)
           if nb != length(data)
               error("short write of $(repr(file_name)) data")
           end
           close(temp_out)
           mv(temp_path, joinpath(dir_name, file_name); force=true)
       end
ERROR: IOError: unlink("C:\\Users\\nzimm\\AppData\\Local\\Temp\\data.txt"): resource busy or locked (EBUSY)
Stacktrace:
  [1] uv_error
    @ .\libuv.jl:100 [inlined]
  [2] unlink(p::String)
    @ Base.Filesystem .\file.jl:978
  [3] rm(path::String; force::Bool, recursive::Bool)
    @ Base.Filesystem .\file.jl:283
  [4] rm
    @ .\file.jl:273 [inlined]
  [5] checkfor_mv_cp_cptree(src::String, dst::String, txt::String; force::Bool)
    @ Base.Filesystem .\file.jl:332
  [6] checkfor_mv_cp_cptree
    @ .\file.jl:318 [inlined]
  [7] #mv#15
    @ .\file.jl:427 [inlined]
  [8] mv
    @ .\file.jl:426 [inlined]
  [9] (::var"#9#10")(temp_path::String, temp_out::IOStream)
    @ Main .\REPL[25]:7
 [10] mktemp(fn::var"#9#10", parent::String)
    @ Base.Filesystem .\file.jl:738
 [11] top-level scope
    @ REPL[25]:1

julia> close(f)

julia> mktemp(dir_name) do temp_path, temp_out
           nb = write(temp_out, data)
           if nb != length(data)
               error("short write of $(repr(file_name)) data")
           end
           close(temp_out)
           mv(temp_path, joinpath(dir_name, file_name); force=true)
       end
"C:\\Users\\nzimm\\AppData\\Local\\Temp\\data.txt"

julia> read(joinpath(dir_name, file_name), String)
"new stuff"

Also, as you can see in the error, calling mv can also lead to a call to rm, so it isn’t fully atomic. Instead of mv you can use:

err = ccall(:jl_fs_rename, Int32, (Cstring, Cstring), temp_path, joinpath(dir_name, file_name))

To hopefully avoid accidentally deleting the token file if the windows computer loses power while calling mv and read all at the same time.

6

That’s exactly what I’m trying to avoid. Is there a way to safely access it across multiple threads in a way that doesn’t throw an error? I.e. there’s a threading process that updates but pushes to a “safe” latest version which the other parts access.

7

Maybe GitHub - JuliaDatabases/SQLite.jl: A Julia interface to the SQLite library would do what you want. I haven’t used it before, but it supports safe access and atomic updates from multiple processes.

If you need to use a JSON file, you could try to write a temp file, then rename it to the original file, and if that fails because the original file is open, write a tokens-v2.json file. Then when reading, always try and read the latest version of the file, using readdir(dir_name) to see the available versions of the JSON file.