I have a package (well, several really) which allow you to pull data from online repositories. They make my workflow easier so I tend to not actually store the data, I just repeatedly fetch it. However, this is suboptimal for me and for the servers, so I am thinking of implementing a cache, wherein the first call to fetch would also cache the file in a scratch space and all subsequent calls over a period of time (perhaps 24 hours) would just load the cached file.
I’m wondering if someone could provide a MWE example using Scratch.jl of how to implement a cache for this type of workflow. I understand that the package readme does cover some of this, but I’m still a bit lost. It seems like this is a simple enough workflow that perhaps an additional package could provide this functionality?
@get_scratch!("some_key") gets you the scratch space directory, what you do with the directory after that is up to you. You can save a timestamp in a file in the directory and delete content at some point when the package gets loaded based on that timestamp. It seems like you want to get rid of the scratch space when it isn’t used though. There is not currently any mechanism for time-based expiration of scratch spaces—as long as some version of your package that asked for the name "some_key" remains installed, the corresponding scratch space directory will be kept around. An time-based expiration mechanism could maybe be added, but that would require design and implementation.
Not necessarily. If the time limit has elapsed, all I would want is that the next call to fetch that file updates the cache in the scratch space as well.
Ok, that would be doable. You would want to record the timestamp in a file in the scratch space (I wouldn’t rely on file system timestamps, they’re notoriously problematic) and if enough time has passed, redo the download. Scratch spaces doesn’t give you anything here except a place for the data to live.
I had the thought to use JuliaHub to look up dependencies on Scratch.jl and it did not find long to find a few packages already doing something similar. I will base my implementation on what I’ve found so far.