# Efficient data access of wrapped C libraries and structs

**URL:** <https://discourse.julialang.org/t/efficient-data-access-of-wrapped-c-libraries-and-structs/4716>\
**Category:** General Usage\
**Tags:** question, package\
**Created:** [July 7, 2017, 3:27pm UTC](https://discourse.julialang.org/t/efficient-data-access-of-wrapped-c-libraries-and-structs/4716 "2017-07-07T15:27:30Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![klowrey](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/klowrey/32/2266_2.png) [@klowrey](https://discourse.julialang.org/u/klowrey)\
**Post date:** [July 7, 2017, 3:27pm UTC](https://discourse.julialang.org/t/efficient-data-access-of-wrapped-c-libraries-and-structs/4716/1 "2017-07-07T15:27:30Z")

</div>

I’ve been doing RL research recently using the [Mujoco physics engine](http://mujoco.org/) as my robotics simulator. Up until now I’ve just written my own library that exposed just the functionality I needed as a subset of all that Mujoco can do, but I figure it may be a good time to do a re-write with all functionality exposed for the community. The issue I’m trying to wrap around is Mujoco contains all its information in **two structs** , mjModel and mjData, and how to efficiently interact with the values they contain when they are both pointers. The following is a example of what they look like, but there are a _few hundred fields_ in each struct in the actual library:

```julia
// in C:
struct _mjModelExample {
  int nposition;
  int nvelocity;

  mjOption opt; // other c struct
  mjVis vis; // other c struct
}
struct _mjDataExample {
  int nbuffer;
  double* time; 
  void* buffer;
  double* positions; // pointers into the void* buffer field
  double* velocity;
}
mjModel* m = mj_loadXML("file.xml");
mjData* d = mj_makeData(m);
mj_step(m, d); // advance the model one time-step; values are stored in d

```

I can get around the struct-within-struct issue of mjModel by making everything **immutable** and not changing fields for the time being, but the bigger issue is accessing and swapping values in mjData’s fields, as Julia seems them as `Ptr{Cdouble}` types. What I’ve thought about doing is replicating the struct’s structure as a Julia type and start doing **unsafe\_wraps** to access the fields.

```julia
type mjDataJulia
  nbuffer::Integer
  time::Float64
  positions::Vector{Float64}
  velocity::Vector{Float64}
end
d = unsafe_load(d_pointer_from_c)
m = unsafe_load(m_pointer_from_c)
j_d = mjDataJulia(0, 0.0,
                  unsafe_wrap(Array, d.positions, m.npositions),
                  unsafe_wrap(Array, d.velocity, m.nvelocity))
j_d.positions = j_d.positions .* 0.5
mj_step(m_pointer_from_c, d_pointer_from_c) // Advance time; fields in mjData can be modified and accessed

```

This way I could ‘write’ into `positions` and `velocity` without having to do unsafe\_wraps later on, but doesn’t address how the non-pointer fields could be mirrored. The structs are too large to always do copies whenever I need values, so having **direct memory access** is ideal.

I feel like there’s something I’m missing, however, in that there’s a simple solution staring me in the face that I’m not seeing. Any advice from folks that have encountered these kinds of data structures or this situation before would be tremendously appreciated!

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [July 7, 2017, 10:13pm UTC](https://discourse.julialang.org/t/efficient-data-access-of-wrapped-c-libraries-and-structs/4716/2 "2017-07-07T22:13:48Z")

</div>

Given the large number of fields, have you tried [Clang.jl](https://github.com/ihnorton/Clang.jl)?

Second, replacing individual items will be tricky unless you access the memory locations directly (e.g., with `pointer`). But [in-progress work](https://github.com/JuliaLang/julia/pull/21912) may soon make that easier.

---

<div class="post-metadata">

**Author:** ![klowrey](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/klowrey/32/2266_2.png) [@klowrey](https://discourse.julialang.org/u/klowrey)\
**Post date:** [July 7, 2017, 10:50pm UTC](https://discourse.julialang.org/t/efficient-data-access-of-wrapped-c-libraries-and-structs/4716/3 "2017-07-07T22:50:29Z")

</div>

I did indeed use Clang.jl to generate some files that have been useful to play around with.

What I figure at this point is the correct course of action (albeit tedious), is like I mentioned above, in having a struct/immutable that is the same memory layout as the c-struct, and then doing unsafe\_wraps around the immutable into a more julia friendly struct for actual work. I have not see many other wrapped libraries that are as data-heavy as Mujoco can be, so was not confident in the approach I thought of.

Thanks for your response!

---

<div class="post-metadata">

**Author:** ![Michael\_Eastwood](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/michael_eastwood/32/669_2.png) [@Michael\_Eastwood](https://discourse.julialang.org/u/Michael_Eastwood)\
**Post date:** [July 7, 2017, 11:19pm UTC](https://discourse.julialang.org/t/efficient-data-access-of-wrapped-c-libraries-and-structs/4716/4 "2017-07-07T23:19:14Z")

</div>

I think [Cxx.jl](https://github.com/Keno/Cxx.jl) might be able to help you here. I honestly haven’t had the opportunity to play with Cxx.jl as much as I would like, but I think you should be able to wrap a C++ pointer in a Julia struct. Then you can write Julia functions that access the fields directly using something like `icxx"mjmodel->nposition"`.

---

<div class="post-metadata">

**Author:** ![klowrey](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/klowrey/32/2266_2.png) [@klowrey](https://discourse.julialang.org/u/klowrey)\
**Post date:** [July 8, 2017, 1:21am UTC](https://discourse.julialang.org/t/efficient-data-access-of-wrapped-c-libraries-and-structs/4716/5 "2017-07-08T01:21:15Z")

</div>

Wow, Cxx.jl provides a pretty attractive interface; you can just evaluate C code as if it were native with what seems to be a slightly separate memory space (still need to unsafe\_wrap pointers, etc). I think in my case I’d still have to be clever about exposing the appropriate data fields in the structs, but It looks like the perf hit of parsing the `icxx"mjmodel->nposition"` line can be exposed as a function:

```julia
set_npos(np) = icxx"mjmodel->nposition = $np;"
step() = icxx"mj_step(m, d);"

```

Definitely worth some investigation! Thanks!

---

<div class="post-metadata">

**Author:** ![ihnorton](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ihnorton/32/26_2.png) [@ihnorton](https://discourse.julialang.org/u/ihnorton)\
**Post date:** [July 8, 2017, 4:27am UTC](https://discourse.julialang.org/t/efficient-data-access-of-wrapped-c-libraries-and-structs/4716/6 "2017-07-08T04:27:43Z")

</div>

Cxx is amazing and is great for local use or in a controlled environment. But if you want to distribute code more broadly: be aware that Windows isn’t quite supported, and as far as I can tell there isn’t a good solution for package distribution yet.

A lightweight Julia object with wrappers makes sense. It should be fast enough for most purposes. If you can’t afford the extra indirect, then reflecting the structs is still going to be the best option. You can use `unsafe_store!` and `unsafe_load` and avoid some manual pointer math with `sizeof` and `fieldoffset`.

---

<div class="post-metadata">

**Author:** ![klowrey](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/klowrey/32/2266_2.png) [@klowrey](https://discourse.julialang.org/u/klowrey)\
**Post date:** [July 10, 2017, 8:16pm UTC](https://discourse.julialang.org/t/efficient-data-access-of-wrapped-c-libraries-and-structs/4716/7 "2017-07-10T20:16:52Z")

</div>

That’s a good point. I’ve already used Clang.jl to help with the conversion but there’s lots of manual tweaking involved, which I wanted to lazily avoid with CXX 🙂

Better to do the right thing from the start, however. Thanks!