# Discussion of ccall( (:function, "library"), ...)

**URL:** <https://discourse.julialang.org/t/discussion-of-ccall-function-library/53887>\
**Category:** General Usage\
**Created:** [January 25, 2021, 6:47am UTC](https://discourse.julialang.org/t/discussion-of-ccall-function-library/53887 "2021-01-25T06:47:16Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![lyonsquark](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lyonsquark/32/4483_2.png) [@lyonsquark](https://discourse.julialang.org/u/lyonsquark)\
**Post date:** [January 25, 2021, 6:47am UTC](https://discourse.julialang.org/t/discussion-of-ccall-function-library/53887/1 "2021-01-25T06:47:16Z")

</div>

Hi, I’d like to hear people’s thoughts about the use of `ccall( (:function, "library"), ...)` as opposed to `ccall( :function, ...)` (see [Calling C and Fortran Code · The Julia Language](https://docs.julialang.org/en/v1/manual/calling-c-and-fortran-code/)) and whether the former variant should be discouraged or the underlying code changed.

The problem is that specifying the shared-object library is very non-C like and prevents one from using profilers and other tools that rely on `LD_PRELOAD` to inject code. When writing C or C++, the library for a function is not specified in the source code. The shared-object is dynamically linked into the code at runtime and a function for a particular call is found by searching through the exported functions from all such shared-objects. This method is equivalent to the Julia `ccall( :function, ...)` variant. If the user has specified a shared-object with the `LD_PRELOAD` environment variable at runtime, that library’s functions go to the top of the function search list, potentially overriding functions of the same name in other shared-objects. This is a common technique of profiler tools that, say, wrap code around `malloc` to track memory allocations.

Use of `LD_PRELOAD` works with Julia so long as you do not specify the library in `ccall`. See [this gist](https://gist.github.com/lyon-fnal/e7e31452b82dc65e8fb9fbaa26022fb7) for a simple example.

However, the Julia documentation encourages specifying the library with `ccall( (:function, "library"), ...)` and so tools that rely on `LD_PRELOAD` do not work. I find that many Julia packages that call C/C++ functions use this variant of `ccall`. My specific problem is that I want to use a tool called [Darshan](https://www.mcs.anl.gov/research/projects/darshan/) to profile HDF5 I/O from my Julia application. Darshan uses `LD_PRELOAD` to wrap MPI-IO and HDF5 C functions to add counters and timers. To make this work, I have my own versions of `MPI.jl` and `HDF5.jl` where I’ve changed the `ccall` statements to use the non-explicit-library variant (I’ve submitted PRs to both packages … for `MPI.jl` all `ccall` statements were already prefixed by a macro and the authors have a PR to change that macro to remove the library argument from the `ccall`).

I do understand the rationale behind `ccall( (:function, "library"), ...)` as it loads the shared object for you. I’m wondering if that can be changed so that the library is loaded with the correct options to make `LD_PRELOAD` work (e.g. on Linux on needs to do `Libdl.dlopen(shared_object_file, RTLD_GLOBAL)`. Or, the documentation be changed to encourage the non-explicit-library variant. Or, at least have a warning that the explicit-library variant has some consequences.

I thought I’d bring this up here before making an issue or a PR to see if anyone else has thought about this concern.

Thanks! – Adam

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [January 25, 2021, 3:55pm UTC](https://discourse.julialang.org/t/discussion-of-ccall-function-library/53887/2 "2021-01-25T15:55:03Z")

</div>

I think the point of explicitly calling into the library is to avoid situations like

```julia
shell> cat foo.c
int test() {
    return 1;
}

shell> gcc -shared -fPIC -o foo.so foo.c

shell> cat bar.c
int test() {
    return 2;
}

shell> gcc -shared -fPIC -o bar.so bar.c

julia> dlopen("./foo.so", RTLD_LAZY | RTLD_DEEPBIND | RTLD_GLOBAL)
Ptr{Nothing} @0x000056086a4c5680

julia> dlopen("./bar.so", RTLD_LAZY | RTLD_DEEPBIND | RTLD_GLOBAL)
Ptr{Nothing} @0x000056086acf1530

julia> ccall(:test, Cint, ()) # can you guess which function will be called?

```

---

<div class="post-metadata">

**Author:** ![lyonsquark](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lyonsquark/32/4483_2.png) [@lyonsquark](https://discourse.julialang.org/u/lyonsquark)\
**Post date:** [January 25, 2021, 4:08pm UTC](https://discourse.julialang.org/t/discussion-of-ccall-function-library/53887/3 "2021-01-25T16:08:00Z")

</div>

`Libdl.dllist()` will tell you the shared object search order.

C/C++ has the same behavior. I’m certainly not saying this behavior is a good idea or defending it, but it’s what C/C++ does and some tools take advantage of it.

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [January 25, 2021, 4:45pm UTC](https://discourse.julialang.org/t/discussion-of-ccall-function-library/53887/4 "2021-01-25T16:45:02Z")

</div>

> [@lyonsquark](#):
>
> some tools take advantage of it

I mean, also a malicious library could do that.

---

<div class="post-metadata">

**Author:** ![lyonsquark](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lyonsquark/32/4483_2.png) [@lyonsquark](https://discourse.julialang.org/u/lyonsquark)\
**Post date:** [January 25, 2021, 5:05pm UTC](https://discourse.julialang.org/t/discussion-of-ccall-function-library/53887/5 "2021-01-25T17:05:41Z")

</div>

Well, yes, but C/C++ has the same vulnerability. I think the thing that saves us is that it’s hard to trick people into setting the `LD_PRELOAD` environment variable that would allow injection of malicious code.

---

<div class="post-metadata">

**Author:** ![staticfloat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/staticfloat/32/34_2.png) [@staticfloat](https://discourse.julialang.org/u/staticfloat)\
**Post date:** [January 26, 2021, 12:42am UTC](https://discourse.julialang.org/t/discussion-of-ccall-function-library/53887/6 "2021-01-26T00:42:03Z")

</div>

I will point out, perhaps pedantically, that this is not really a C/C++ feature, but more a feature of the underlying libc. This doesn’t really matter much to the discussion, but I wanted to point out that this feature was likely not in the minds of the C developers when they were initially designing the language; it’s tooling that has grown up around the dynamic linker on various platforms.

> I do understand the rationale behind `ccall( (:function, "library"), ...)` as it loads the shared object for you.

I would argue that is not, in fact, the primary reason for `ccall((:function, library))` (here I’m not making a distinction between `library::String` and `library::Ptr{Cvoid}`, such as the result from `dlopen()`). I would argue instead that the primary reason is so that `ccall()`, by default, ignores namespace pollution.

For the vast majority of C/C++ programs, which libraries you are going to load and what symbols they contain is a very static table. At compile time you list `-lfoo -lbar` and these get embedded into your executable’s import list, and this is not expected to change over time. From the compiler’s perspective, if I have two symbols with the same name trying to be embedded within the same namespace (e.g. the same shared library/executable) it will error, because it will create an unresolvable ambiguity. The dynamic linker doesn’t error and has platform-specific behavior; Linux, FreeBSD and MacOS all allow you to set it into a mode where the first symbol entered into the table wins, while Windows doesn’t maintain a global symbol table at all; you must scope your `dlsym()` queries to a specific namespace.

In practice, this often works quite well because the global namespace in C/C++ applications is a hand-crafted thing that developers more or less have control over. When you build an application, you know what is going to be going on within your namespace, and in the rare cases where you have, e.g. `libfoo` which is getting loaded within `giant_application`, and `libfoo` may conflict with `libbar`, some OS’s provide facilities such as multi-level namespaces that can solve this problem with specific flags given to `dlopen()`.

Julia, on the other hand, generally has a much more hostile namespace environment. Users interactively run `using Foo` all over the place, the order in which libraries are loaded is not guaranteed, and in general the community acts like a giant fuzzer for library loading combinations. 🙂 It is quite common for us to run into symbol name clashes.

See, for instance, [this issue](https://github.com/JuliaPy/PyCall.jl/issues/443), where if you use `PyCall` to load `numpy` on a Julia that uses MKL as its backing BLAS library, you get a segfault. This happens because MKL spits its symbols into the global symbol table, it provides ILP64 versions of those symbols, and Numpy is expecting a non-ILP64 version because it’s trying to call the functions in the BLAS library it shipped with. If `numpy` were to use the python equivalent of the `ccall((:function, library), ...)` syntax, it wouldn’t have these problems. (I will point out for completeness sake that we have requested Intel to change their naming policy, like we do with OpenBLAS to move ILP64 symbols to a different name such as `dgemm64_` instead of just `dgemm`, but it’s an uphill battle).

In our experience, the incidence rate of users wanting to override symbol lookup through `LD_PRELOAD` is much, much lower than the incidence rate of two completely unrelated libraries containing conflicting symbol names. Additionally, since users are building larger and larger applications all the time, it becomes only more and more likely that the namespacing issue will become easier and easier to trigger.

Now, all that being said, the best of both worlds would be to allow explicit overloading of symbol resolution. It seems to me that one way this can be done is by using the facility in Julia 1.6 to parameterize the `library` argument to `ccall`. E.g. you could do the following:

```julia
$ cat foo.c
int always_foo() {
    return 1;
}

int sometimes_foo() {
    return 2;
}
$ cat bar.c
int sometimes_foo() {
    return 5040;
}
$ gcc -o foo.so -shared foo.c
$ gcc -o bar.so -shared bar.c

```

```julia
julia> function get_overridden_library(func_name::Symbol)
           if func_name in (:sometimes_foo,)
               return "./bar.so"
           end
           return "./foo.so"
       end

julia> ccall((:always_foo, get_overridden_library(:always_foo)), Cint, ())
1
julia> ccall((:sometimes_foo, get_overridden_library(:sometimes_foo)), Cint, ())
5040

```

Note that this only works on Julia v1.6+, in Julia v1.5- you can’t dynamically compute the library name. And for each `ccall()` site, your `ccall()` choice gets baked into the generated code, so you can’t dynamically switch at runtime; the result of the function that you called to determine the library path gets cached.

---

<div class="post-metadata">

**Author:** ![lyonsquark](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lyonsquark/32/4483_2.png) [@lyonsquark](https://discourse.julialang.org/u/lyonsquark)\
**Post date:** [January 27, 2021, 6:43am UTC](https://discourse.julialang.org/t/discussion-of-ccall-function-library/53887/7 "2021-01-27T06:43:52Z")

</div>

Thanks for your detailed and interesting post! The new features for dynamically computing the library name in Julia 1.6+ look interesting.

I’ll just point out two things… You wrote that C/C++ applications hand-craft the global namespace. Counter-examples are many of the C++ frameworks used in code to process and analyze High Energy Physics experiment data. Such frameworks, like [art](https://art.fnal.gov), rely on dynamic loading at runtime of many modules to produce, select, and analyze data. The framework code itself handles the dynamic loading, the I/O loop, and other infrastructure stuff while researchers write the modules with the science code. So in this sense, the application itself is not defined until runtime. Of course, name collisions are a problem and most frameworks will error if loading a shared object creates a name conflict.

On the other hand, `LD_PRELOAD` works “above” the application where naming collisions are intensional for tools that do profiling. We use such tools often to help us make our code performant. The nice thing about `LD_PRELOAD` is that you can use such tools without changing or recompiling the code to be profiled. Would be nice if that worked for Julia code too (e.g. like for what I’m trying to do by profiling HDF5 C calls).

---

<div class="post-metadata">

**Author:** ![staticfloat](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/staticfloat/32/34_2.png) [@staticfloat](https://discourse.julialang.org/u/staticfloat)\
**Post date:** [January 27, 2021, 7:42am UTC](https://discourse.julialang.org/t/discussion-of-ccall-function-library/53887/8 "2021-01-27T07:42:35Z")

</div>

> Counter-examples are …

Good point, there are always exceptions to the rule.

> Of course, name collisions are a problem and most frameworks will error if loading a shared object creates a name conflict.

A valid approach, but not the one we would ultimately choose if we can avoid it.

> Would be nice if that worked for Julia code too

Yes… having thought about this a bit more and talked it over with @vchuravy on the Julia HPC community call today, I think it might not be too difficult to emulate `LD_PRELOAD` in Julia. Here’s how I would do it:

- First, we could make this either a global or an opt-in semantic. If it’s global, it will need to be merged into Julia base, and you’ll need to convince more than just me that this is a good idea. ;). If it’s opt-in, it can be done in a package.

- The basic idea is to reimplement the necessary logic of `LD_PRELOAD`, which, when you boil it down, becomes essentially:

```julia
function allow_ld_preload(func_name::Symbol, lib_path::String)
    preload_libs = split(get(ENV, "LD_PRELOAD", ""), ":")
    for prelib in preload_libs
        preload_lookup = dlsym(prelib, func_name; throw_error=false)
        if preload_lookup !== nothing
            return (func_name, prelib)
        end
    end
    return (func_name, lib_path)
end

```

Ignore for the moment the fact that `dlsym()` needs a handle and not a `String`, and that doing `dlsym()` again and again for each call site may not be the most efficient way to do this; in the end, this is quite a simple set of logic. For the opt-in usecase, this can be used to generate that libname argument on the fly:

```julia
ccall(allow_ld_preload(:foo, "libfoo"), ...)

```

For this to be the default, we’d have to do that equivalent logic within `ccall()` itself. That’s not impossible, but there may be difficulties I haven’t thought of yet. Also, the benefit of doing this in Julia itself is that this approach would work on all platforms (including Windows), so we could become the first programming language to support `LD_PRELOAD` on Windows! 😉

---

<div class="post-metadata">

**Author:** ![lyonsquark](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lyonsquark/32/4483_2.png) [@lyonsquark](https://discourse.julialang.org/u/lyonsquark)\
**Post date:** [January 29, 2021, 8:04pm UTC](https://discourse.julialang.org/t/discussion-of-ccall-function-library/53887/9 "2021-01-29T20:04:06Z")

</div>

Thanks! Sorry I missed the Julia HPC call (I always have a conflicting meeting over that time). Would be nice to somehow cache the library result (sounds like in Julia 1.6 this will be possible).

In general, I think this is a good approach. Though the “spirit” of `LD_PRELOAD` is that you don’t have to change your code to bring in the profiler or whatever.

A suggestion in my [HDF5 PR](https://github.com/JuliaIO/HDF5.jl/pull/791) from @musm was to use Cassette.jl to do the injection (e.g. remove the library specification from `ccall` statements - or change the library name to the one in `LD_PRELOAD` - I guess like the macro in MPI.jl but could go deeper). I like that idea, because package code wouldn’t need to change. Though I wonder what the performance implications would be. I also don’t know enough about Cassette.jl to try something like that. But from what I’ve seen I think it is possible. Would be interested to hear thoughts on that approach.