MemViews.jl - practical, simple, low-level views into Memory

Package Announcements
1

ANN: MemViews.jl - low-level view into Memory{T}

MemViews.jl is a small package that introduces two new types:

  • MemView is a view into Memory{T}
  • MemKind(T::Type) is a trait signalling if T can be represented as memory.

MemViews.jl requires Julia the last Julia 1.11 release branch or master (it does not work on Julia 1.11-beta2).
It is currently undergoing registration.

The MemView type has the following important features compared to SubArray:

  • It has a concrete, simple memory layout, because it is not generic over the underlying AbstractArray or index type.
    This makes it much easier to reason about low-level considerations such as whether you can take a pointer to the view, and what stride it has, and what dimensionality it has, and whether it uses one-based indexing, and much else.
    Practically speaking, this makes it easier to write correct code.
  • The MemKind trait makes it easier to write specialized implementations for dense, memory backed arrays. In contrast, it’s difficult to select the correct subtypes of SubArray (or AbstractArray) that are dense and memory-backed.
  • Mutability or immutability is statically encoded into the type as either MemView{T, :mutable} or MemView{T, :immutable}. This makes it more difficult to e.g. accidentally mutate a string, even in code where you regularly manipulate strings with pointers or ccalls.

Please see the documentation for more information:

Also, see the related Julia issue

I’ve already been dogfooding MemViews.jl by developing the upcoming XAMAuxData.jl and PairwiseMappingFormat.jl and I must say it’s been quite nice to use, API-wise.

I’d like to express my appreciation for the work of @Oscar_Smith and @jameson , whose work on the underlying Julia types Memory and MemoryRef has made this package possible.

Please give feedback on the package’s design etc

Example

julia> MemView([1, 2, 3]) |> typeof
MutableMemView{Int64} (alias for MemView{Int64, :mutable})

julia> MemView(view(codeunits("abc"), Base.OneTo(0x02))) |> typeof
ImmutableMemView{UInt8} (alias for MemView{UInt8, :immutable})

julia> immutable = MemView("abc");

julia> unsafe_copyto!(immutable, MemView([0x01, 0x02, 0x03]))
ERROR: MethodError: no method matching unsafe_copyto!(::ImmutableMemView{UInt8}, ::MutableMemView{UInt8})
21 Likes
2

For some low-level IO package applications, I have long been struggling with exactly this:

So I immediately tried this:

julia> a = ImmutableMemView([1, 2, 3])
3-element ImmutableMemView{Int64}:
 1
 2
 3

julia> reinterpret(Int32, a)
6-element reinterpret(Int32, ::ImmutableMemView{Int64}):

this currently doesn’t have special implementation, but what I want essentially is for this to return a ImmutableMemView{Int32} – this would solve the problem of “difficult to pre-select correct types”

3

See the Limitations section of the README and also this Discourse thread

I also tried to make that work, but couldn’t. The reason is that it’s not possible to construct a MemoryRef{B} from a MemoryRef{A}.
It can be done incurring an allocation by making a new Memory object that aliases the old one, and then taking a view of that.
However, the docs of unsafe_wrap states:

Unlike unsafe_load and unsafe_store!, the programmer is responsible also
for ensuring that the underlying data is not accessed through two arrays of different element type, similar to the strict aliasing rule in C.

I believe it might be UB to create a MemView{A} object pointing to `Memory{B}. The proposed docs on UB in Julia states that it’s UB to cause:

Violations of TBAA guarantees (e.g. using unsafe_wrap)

1 Like
4

yeah. that would be UB

1 Like
5

Are those links placeholders? they seem to be broken.

Fantastic work @jakobnissen! Thanks for all your efforts to make BioJulia even better.

1 Like
6

Thanks for the kind words!
The repos were private. I’ve now made them public as they are almost complete.

3 Likes