# Disk backed Dict?

**URL:** <https://discourse.julialang.org/t/disk-backed-dict/7719>\
**Category:** General Usage\
**Tags:** question\
**Created:** [December 12, 2017, 8:06pm UTC](https://discourse.julialang.org/t/disk-backed-dict/7719 "2017-12-12T20:06:42Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![jw3126](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jw3126/32/3086_2.png) [@jw3126](https://discourse.julialang.org/u/jw3126)\
**Post date:** [December 12, 2017, 8:06pm UTC](https://discourse.julialang.org/t/disk-backed-dict/7719/1 "2017-12-12T20:06:42Z")

</div>

Is there a disk backed `Dict` somewhere? I would like to be able to do the following:

```julia
d = DiskBackedDict{K,V}(mypath)::Associative{K,V} # load or create new one if not existing
x = d[foo] # this is as fast as an ordinary Dict lookup.
d[bar] = y # this may be dog slow, but the result must be stored at mypath

d2 = DiskBackedDict{K,V}(mypath) # error mypath already in use

```

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [December 12, 2017, 9:05pm UTC](https://discourse.julialang.org/t/disk-backed-dict/7719/2 "2017-12-12T21:05:24Z")

</div>

Wouldn’t `d[foo]` be slower since it’s disk-based?

---

<div class="post-metadata">

**Author:** ![jw3126](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jw3126/32/3086_2.png) [@jw3126](https://discourse.julialang.org/u/jw3126)\
**Post date:** [December 12, 2017, 9:10pm UTC](https://discourse.julialang.org/t/disk-backed-dict/7719/3 "2017-12-12T21:10:46Z")

</div>

I would like `DiskBackedDict` to cache stuff in memory for fast access.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [December 12, 2017, 9:21pm UTC](https://discourse.julialang.org/t/disk-backed-dict/7719/4 "2017-12-12T21:21:57Z")

</div>

Wondering how big is the dictionary of data you are thinking about using? If it’s smaller one can roll a simple solution fairly quickly in a couple of hours.  
Not sure if one already exists

---

<div class="post-metadata">

**Author:** ![jw3126](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jw3126/32/3086_2.png) [@jw3126](https://discourse.julialang.org/u/jw3126)\
**Post date:** [December 12, 2017, 9:33pm UTC](https://discourse.julialang.org/t/disk-backed-dict/7719/5 "2017-12-12T21:33:40Z")

</div>

Not big, maybe `100mb` at most. I guess its indeed not too hard to implement this. However if there is an existing solution I would prefer that. Also there are some implementation issues I am unsure about:

1. For storage of arbitrary julia objects, JLD is the only option I can think of. But I don’t know if there is a way to modify a value in a JLD file. The only way I am aware of is delete the file and save the modified version.
2. I am not sure how to do the locking of a path that belongs to an active `DiskBackedDict`.

---

<div class="post-metadata">

**Author:** ![xiaodai](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xiaodai/32/15937_2.png) [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Post date:** [December 12, 2017, 9:38pm UTC](https://discourse.julialang.org/t/disk-backed-dict/7719/6 "2017-12-12T21:38:05Z")

</div>

1. Is why I think it’s only good for a small dict. Maybe there are solutions that let you append to a file but then you have to design the format.
2. I think you can acquire an exclusive lock on the file when you open it. It should be in an `openfile` function somewhere maybe by another name.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [December 13, 2017, 8:13am UTC](https://discourse.julialang.org/t/disk-backed-dict/7719/7 "2017-12-13T08:13:49Z")

</div>

I recently needed something like this and thought about using an `mmap`ed array for storage of the values. The problem with that is that

1. you cannot grow mmapped arrays, so they have to be reallocated occasionally,
2. only bits types would work.

But otherwise it should be simple to adapt the code from `DataStructures.jl` or `Base`.

---

<div class="post-metadata">

**Author:** ![slowbrain](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/slowbrain/32/2224_2.png) [@slowbrain](https://discourse.julialang.org/u/slowbrain)\
**Post date:** [December 13, 2017, 8:36am UTC](https://discourse.julialang.org/t/disk-backed-dict/7719/8 "2017-12-13T08:36:50Z")

</div>

just to correct your statement regarding JLD value modification.  
It is possible to delete single dataset in JLD file and replace it with new one:

```julia
using JLD

# create JLD with two datasets
file = jldopen("mydata.jld", "w")
file["a"] = [1:100]
file["b"] = [1:10]
close(file)

#replace content of dataset "a"
file = jldopen("mydata.jld", "r+")
delete!(file["a"])
file["a"] = "changed"
close(file)

@load(mydata.jld)
show(a)
show(b)

```

---

<div class="post-metadata">

**Author:** ![jw3126](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jw3126/32/3086_2.png) [@jw3126](https://discourse.julialang.org/u/jw3126)\
**Post date:** [December 13, 2017, 6:12pm UTC](https://discourse.julialang.org/t/disk-backed-dict/7719/9 "2017-12-13T18:12:29Z")

</div>

I put something together [here](https://github.com/jw3126/DiskBackedDicts.jl), based on @slowbrain’s remark.

---

<div class="post-metadata">

**Author:** ![ScottPJones](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/scottpjones/32/146_2.png) [@ScottPJones](https://discourse.julialang.org/u/ScottPJones)\
**Post date:** [December 14, 2017, 1:24pm UTC](https://discourse.julialang.org/t/disk-backed-dict/7719/10 "2017-12-14T13:24:15Z")

</div>

What you are essentially doing is making something like key-value based store - and it’s not that difficult to make a Julia for them. I made one back in 2015 for Aerospike, that we use for our product at [http://www.dynactionize.com](http://www.dynactionize.com).

---

<div class="post-metadata">

**Author:** ![tobias.knopp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tobias.knopp/32/7551_2.png) [@tobias.knopp](https://discourse.julialang.org/u/tobias.knopp)\
**Post date:** [December 14, 2017, 1:29pm UTC](https://discourse.julialang.org/t/disk-backed-dict/7719/11 "2017-12-14T13:29:27Z")

</div>

I have tried several solutions to this problem, including JDL.

For me, in the end TOML was the best solution:  
[https://github.com/wildart/TOML.jl](https://github.com/wildart/TOML.jl)

It is an INI like file but it fits much better to Julia since the type of the values can be specified. If you don’t have to complicated types this could be a very goos solution. One big plus for me was that I can hand edit the files.