Compare and swap operation on mmap/shared-memory

General Usage
1

Hi all :slight_smile:

How can I do an cas (compare and swap operation) on memory mapped arrays in Julia.

julia> v = SharedArray{UInt8}(abspath("test.bytes"), (1000,));

julia> using SharedArrays

julia> v = SharedArray{UInt8}(abspath("test.bytes"), (1000,));

julia> if v[1] == 0x01 # I wanna do this atomically - atomic_cas!
           v[1] = 0x00
       end

Thanks

2

For some reason atomics.jl decides against exporting that functionality. You can get it by copy-pasting and adjusting the code for Threads.atomic_cas!, like e.g.

my_atomic_cas!(ptr::Ptr{Int64}, cmp::Int64, new::Int64) =  Core.Intrinsics.llvmcall("""
                            %ptr = inttoptr i64 %0 to i64*
                            %rs = cmpxchg i64* %ptr, i64 %1, i64 %2 acq_rel acquire
                            %rv = extractvalue { i64, i1 } %rs, 0
                            ret i64 %rv
                            """, Int64, Tuple{Ptr{Int64},Int64,Int64},
                            ptr, cmp, new)

Then you can use that via e.g. my_atomic_cas!(pointer(a,2), 3, 4) (you may need to take care to sprinkle GC.@preserve at appropriate positions).

Does anyone know why we don’t export atomic operations on pointers and array elements?

1 Like
3

We do. It is still in Core.Intrinsics though, since we have not finalized a Base API yet

4

I just edited the post to make it more clear. Is your llvm code valid for the this?

5

I think your example wants to be

if v[1] == 0x01 # I wanna do this atomically - atomic_cas!
           v[1] = 0x00
       end

i.e. if the array element is as expected, then replace it with a different value (and if it is unexpected, e.g. because somebody updated it concurrently, then fail and write nothing – your code will have a branch to handle it).

The thing you wrote (if v[1] != 0x00 v[1] = 0x00 end) is rather weird and not CAS. You’d need to explain the desired effect of that and how that differs from plain v[1] = 0x00 (memory mapped IO? Avoid marking the page dirty?).

Otherwise you need to adjust the types from Int64 to UInt8 to use the example code I wrote:

my_atomic_cas!(ptr::Ptr{UInt8}, cmp::UInt8, new::UInt8) =  Core.Intrinsics.llvmcall("""
                            %ptr = inttoptr i64 %0 to i8*
                            %rs = cmpxchg i8* %ptr, i8 %1, i8 %2 acq_rel acquire
                            %rv = extractvalue { i8, i1 } %rs, 0
                            ret i8 %rv
                            """, UInt8, Tuple{Ptr{UInt8},UInt8,UInt8},
                            ptr, cmp, new)

@jameson Thanks for referring me to julia/multi-threading.md at 81813164963f38dcd779d65ecd222fad8d7ed437 · JuliaLang/julia · GitHub

When trying to use the new fancy atomic things, I am somewhat stumped? For some reason replacefield! does not support index exchanges on pointers/arrays; and even replacefield!(Ref(4), :x, 4, 5, :acquire_release) fails with ERROR: ConcurrencyViolationError(“replacefield!: non-atomic field cannot be written atomically”)`? Why that instead of the obvious and almost verbatim requested LOCK CMPXCHG8b?

So, what is the official recommendation? I’d guess that llvmcall is more stable than manually using a barely documented intrinsic? The main point I see for intrinsics as opposed to llvmcall is that the core team can put better aliasing info on the pointer than us mere end-users can access in llvmcall?

2 Likes
6

@Impressium We don’t currently support arrays (or sharedarray) with a high level API. Mainly because atomics on arrays are pretty slow, so there is often a better performing implementation with locks possible. We will likely add it at some point, however, since it does have its uses anyways. Currently, there is an experimental package though for experimenting with it: https://juliaconcurrent.github.io/Atomix.jl/dev/

instead of the obvious and almost verbatim requested

The standard memory models for CPUs (e.g. in use by C11) declare it to be undefined behavior to cast a non-atomic field pointer for use with atomics. The standards authors demonstrated that it lead to mis-compilations and incorrect execution, which is not something that Julia wants to accidentally expose to users.

llvmcall is more stable than manually

llvmcall IR frequently changes meaning between LLVM releases. It is explicitly not stable, per llvm’s policies. In contrast, Core.Intrinsics are part of the Julia API. As you mentioned, there are also a variety of correctness improvements possibly by using intrinsics which not expressible in a naked llvmcall. With enough annotations, the user can recover most of the correctness, and convert them into simple inefficiencies instead. But since intrinsics express the actual user intent, they do not suffer from those deficiencies.

7

Thank you all :slight_smile:

@jameson
To clarify what I want (don’t know if it’s the right way or even possible this why): I wanna use the shared-array within two separate processes. The cas on the array/file is used to lock it via a flag for the other process. Is that possible this way and does atomic operation in shared-array work like that?

8

You probably just want a mmap pointer (from the Mmap) package, since the array functionality is not desirable in that case. You can use that to get memory for a normal pthread_mutex and configure it with pthread_mutexattr_setpshared to be PTHREAD_PROCESS_SHARED. The equivalent on Windows is CreateMutex.

1 Like