# Readers–writer lock using DataFrame

**URL:** <https://discourse.julialang.org/t/readers-writer-lock-using-dataframe/116550>\
**Category:** Performance\
**Tags:** question\
**Created:** [July 2, 2024, 9:16pm UTC](https://discourse.julialang.org/t/readers-writer-lock-using-dataframe/116550 "2024-07-02T21:16:15Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![JesperMartinsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jespermartinsson/32/34098_2.png) [@JesperMartinsson](https://discourse.julialang.org/u/JesperMartinsson)\
**Post date:** [July 2, 2024, 9:16pm UTC](https://discourse.julialang.org/t/readers-writer-lock-using-dataframe/116550/1 "2024-07-02T21:16:16Z")

</div>

Hi,

I have an application where one async task updates a DataFrame with new data every minute, while multiple tasks read the DataFrame at random times. I need to make this thread-safe, possibly by using a lock, to prevent reads during updates.

A ReentrantLock() could work, but it might block concurrent reads, which DataFrame can handle inherently. I’m considering using semaphores or a perhaps a suitable readers-writer lock to address this.

Any suggestions, pointers, or examples to achieve this without compromising the performance of concurrent reads?

---

<div class="post-metadata">

**Author:** ![era127](https://avatars.discourse-cdn.com/v4/letter/e/eb8c5e/32.png) [@era127](https://discourse.julialang.org/u/era127)\
**Post date:** [July 2, 2024, 9:59pm UTC](https://discourse.julialang.org/t/readers-writer-lock-using-dataframe/116550/2 "2024-07-02T21:59:05Z")

</div>

Here is an example of concurrent readers and writers with [duckdb](https://duckdb.org/docs/api/julia.html#concurrency) in case you want to persist the data as well.

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [July 2, 2024, 10:47pm UTC](https://discourse.julialang.org/t/readers-writer-lock-using-dataframe/116550/3 "2024-07-02T22:47:43Z")

</div>

I needed a RW lock for [AllocArrays.jl](https://github.com/ericphanson/AllocArrays.jl) and settled on [the one in ConcurrentUtilities.jl](https://github.com/JuliaServices/ConcurrentUtilities.jl/blob/main/src/rwlock.jl)

---

<div class="post-metadata">

**Author:** ![JesperMartinsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jespermartinsson/32/34098_2.png) [@JesperMartinsson](https://discourse.julialang.org/u/JesperMartinsson)\
**Post date:** [July 3, 2024, 8:10am UTC](https://discourse.julialang.org/t/readers-writer-lock-using-dataframe/116550/4 "2024-07-03T08:10:51Z")

</div>

Thanks for the pointer. That looks very interesting.

---

<div class="post-metadata">

**Author:** ![JesperMartinsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jespermartinsson/32/34098_2.png) [@JesperMartinsson](https://discourse.julialang.org/u/JesperMartinsson)\
**Post date:** [July 3, 2024, 8:14am UTC](https://discourse.julialang.org/t/readers-writer-lock-using-dataframe/116550/5 "2024-07-03T08:14:10Z")

</div>

Thanks, that looks very interesting. I see that julia v1.11 may have some of [these functionalities](https://github.com/JuliaLang/julia/blob/master/base/lock.jl) in base, but perhaps not the `ReadWriteLock()` part.

---

<div class="post-metadata">

**Author:** ![foobar\_lv2](https://avatars.discourse-cdn.com/v4/letter/f/ee59a6/32.png) [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)\
**Post date:** [July 3, 2024, 10:47am UTC](https://discourse.julialang.org/t/readers-writer-lock-using-dataframe/116550/6 "2024-07-03T10:47:58Z")

</div>

Depending on your specific requirements, also consider copy-on-write.

The idea of copy-on-write is that on updates, you construct a new dataframe (reusing unchanged arrays of the old one), and you have some object like

```julia
mutable struct AtomicContainer
@atomic contents::Any
end

```

and after your new dataframe is constructed, you insert it into the AtomicContainer. Readers have a long-lived reference to the AtomicContainer, and can then read the contents and dispatch (function barrier to cure the type instability!) with the current immutable snapshot of the data.

The relevant considerations for this are:

1. If you have a reader who wants to do stuff and an update is underway, is it preferable to block until the update is done or is it preferable to do your stuff on a consistent-but-potentially-stale version?
2. What is your relation between readers and writers, in terms of volume?
3. Can you afford the additional GC pressure from copy-on-write? Especially, how real-time-ish are your readers?

The big problems with Reader-writer-locks are that:

1. Multiple concurrent readers don’t block each other. But the responsible cpu cores still need to play tug-of-war on which core owns the cacheline of the lock
2. If your update is big, then you either have a long critical section, i.e. long blocking of readers, or your readers can see inconsistent states (because you relinquish the lock in the middle). Whether this is a problem depends on your answer to (1).
3. There is a big question for your readers: Take the lock for a long time (long critical section) or take it often for short times. Taking it for a long time may block the writer, taking it often causes tug-of-war on the cache-line between multiple readers. Your critical section of course needs to be long enough to span the required consistency (i.e. the entire “transaction”).

PS. The above example uses `Any` as type for the contents. This is super defensive programming of me, because consider the following:

```julia
mutable AtomicContainerYolo{T}
@atomic contents::T 
end

```

can introduce a lock in `AtomicContainerYolo((1,2,3))` in some julia versions. Because your hardware only supports 128 bit atomics, and julialang made the imo extremely ill-considered design decision to imitate C++ in implementing atomic variables that are hardware-impossible by hidden locks instead of boxing / copy-on-write. C++ has the excuse of “no GC in the runtime, cannot cow”, but julialang doesn’t. Making the `contents` field abstractly typed forces the compiler to box it, which is exactly what you want for anything larger than 128 bit.

PPS. I am not recommending to use persistent (aka functional, aka non-overwriting) datastructures. Instead, use that your problem is very specific: Your updates/writes come in large batches. So for every update/write you need to identify which parts of your dataframe are modified; and you might want to modify your data layout to minimize the modified parts.

---

<div class="post-metadata">

**Author:** ![quinnj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/quinnj/32/11_2.png) [@quinnj](https://discourse.julialang.org/u/quinnj)\
**Post date:** [July 4, 2024, 4:49am UTC](https://discourse.julialang.org/t/readers-writer-lock-using-dataframe/116550/7 "2024-07-04T04:49:51Z")

</div>

Yeah, we only added Lockable to Base, but not the ReadWriteLock.