Clang.jl large library

[mopg]

I am trying to link Julia to a large C library. I was thinking that that wouldn’t be an issue given that everyone always says it is straightforward to link C to Julia with minimal performance loss. My problem is that I have never linked different programming languages together, so I am a bit confused by the whole process.

On someone’s recommendation, I started using Clang.jl, but the documentation for that only treats very simple examples (and is not always up-to-date). Clang.jl seems to be exactly what I am looking for, but I’m just struggling how to use it.

Now this is what I have so far (it renders a lot of deprecated warnings, but this is what was in the documentation):

top = cindex.parse_header("<path to .h header file from C library>")

context = wrap_c.init(; output_file="<packagename>.jl", header_library=x->"<packagename>", common_file="<packagename>_h.jl", clang_diagnostics=true)

wrap_c.wrap_c_headers(context, ["<path to .h header file from C library>"])

This generates the <packagename>.jl and <packagename>_h.jl. Now 3 questions:

1. Is this at all a correct way to go about this? The library has a bunch of header files in its include directory, but I am only using the one that you would use if included it from C. The library also has an .so file but I am not doing anything with that.
2. I get the following error, how do I fix that? The documentation talks about it, but that is incomprehensible to me.
   /usr/include/stdio.h:33:11 fatal error: 'stddef.h' file not found
3. What am I supposed to do with the <packagename>.jl and <packagename>_h.jl files once they are generated? Does including them give me access to the C library?

Sorry for the very basic questions, any help is much appreciated.

[ChrisRackauckas]

Take a look at Sundials.jl for a full example of Clang.jl.

[randyzwitch]

Out of curiosity Chris, how many passes over the code did it take you to write all that code? Did you see things you didn’t like and keep writing the code?

[ChrisRackauckas]

@randyzwitch

how many passes over the code did it take you to write all that code? Did you see things you didn’t like and keep writing the code?

Quite a few. I started from an older wrap_sundials.jl file that I inherited. For the latest Sundials update though, there were a lot of changes that had to be done to match the new API. So I would run it, get some headers, see what’s wrong, and then make a few changes, run it again, repeat, until the auto-wrapper was setup perfectly.

[mopg]

Thanks. So when you run that script, you just use the generated files? You don’t need to “manually post process” them?

[ChrisRackauckas]

Those directly give the files in the wrapped API folder.

https://github.com/JuliaDiffEq/Sundials.jl/tree/master/src/wrapped_api

and that’s most of the wrapper.

[mopg]

Ok so I now have something that doesn’t throw any errors when I include the generated .jl files. However, when I run Clang.jl it only generates a types_and_consts.jl and an include.jl file, not a .jl for every header file. So when I try to use any function (any CCall) it throws an error that it cannot find the associated library file.

So specific questions:

1. What is happening after line 133 in wrap_sundials.jl? Is that specific to the Sundials C library?
2. What part of the code ensures that all header files are wrapped in different julia files and link to different library files? Right now, I am only using this (egads is the name of the library):

 const contxt = wrap_c.init(
     common_file = joinpath(outpath, "types_and_consts.jl"),
     # clang_args = [
         # "-D", "__STDC_LIMIT_MACROS",
         # "-D", "__STDC_CONSTANT_MACROS",
         # "-v"
     # ],
     clang_diagnostics = true,
     clang_includes = [clang_includes; incpath],
     header_outputfile = julia_file,
     header_library = library_file#,
     # header_wrapped=wrap_header#,
     # cursor_wrapped=wrap_cursor
 )
contxt.headers = egads_headers
 
run(contxt)

[ChrisRackauckas]

@mopg

However, when I run Clang.jl it only generates a types_and_consts.jl and an include.jl file, not a .jl for every header file. So when I try to use any function (any CCall) it throws an error that it cannot find the associated library file.

I don’t think that last part is related? It should work no matter what .jl file it’s in. For example, the call

https://github.com/JuliaDiffEq/Sundials.jl/blob/master/src/wrapped_api/cvode.jl#L5-L7

knows what to call by the constant libsundials_cvode which is set

SciML/Sundials.jl/blob/master/src/Sundials.jl#L19-L24

here via a string in the deps/deps.jl file. For example, my deps.jl is:

## This file autogenerated by BinaryProvider.@write_deps_file.
## Do not edit.
const libsundials_kinsol = "C:\\Users\\Chris\\.julia\\v0.6\\Sundials\\deps\\usr\\bin\\libsundials_kinsol.dll"
const libsundials_idas = "C:\\Users\\Chris\\.julia\\v0.6\\Sundials\\deps\\usr\\bin\\libsundials_idas.dll"
const libsundials_nvecserial = "C:\\Users\\Chris\\.julia\\v0.6\\Sundials\\deps\\usr\\bin\\libsundials_nvecserial.dll"
const libsundials_cvodes = "C:\\Users\\Chris\\.julia\\v0.6\\Sundials\\deps\\usr\\bin\\libsundials_cvodes.dll"
const libsundials_sunlinsolspfgmr = "C:\\Users\\Chris\\.julia\\v0.6\\Sundials\\deps\\usr\\bin\\libsundials_sunlinsolspfgmr.dll"
const libsundials_sunmatrixdense = "C:\\Users\\Chris\\.julia\\v0.6\\Sundials\\deps\\usr\\bin\\libsundials_sunmatrixdense.dll"
const libsundials_sunlinsolspbcgs = "C:\\Users\\Chris\\.julia\\v0.6\\Sundials\\deps\\usr\\bin\\libsundials_sunlinsolspbcgs.dll"
const libsundials_sunlinsoldense = "C:\\Users\\Chris\\.julia\\v0.6\\Sundials\\deps\\usr\\bin\\libsundials_sunlinsoldense.dll"
const libsundials_sunlinsolspgmr = "C:\\Users\\Chris\\.julia\\v0.6\\Sundials\\deps\\usr\\bin\\libsundials_sunlinsolspgmr.dll"
const libsundials_sunlinsolpcg = "C:\\Users\\Chris\\.julia\\v0.6\\Sundials\\deps\\usr\\bin\\libsundials_sunlinsolpcg.dll"
const libsundials_sunlinsolsptfqmr = "C:\\Users\\Chris\\.julia\\v0.6\\Sundials\\deps\\usr\\bin\\libsundials_sunlinsolsptfqmr.dll"
const libsundials_sunmatrixsparse = "C:\\Users\\Chris\\.julia\\v0.6\\Sundials\\deps\\usr\\bin\\libsundials_sunmatrixsparse.dll"
const libsundials_sunlinsolband = "C:\\Users\\Chris\\.julia\\v0.6\\Sundials\\deps\\usr\\bin\\libsundials_sunlinsolband.dll"
const libsundials_sunmatrixband = "C:\\Users\\Chris\\.julia\\v0.6\\Sundials\\deps\\usr\\bin\\libsundials_sunmatrixband.dll"
const libsundials_arkode = "C:\\Users\\Chris\\.julia\\v0.6\\Sundials\\deps\\usr\\bin\\libsundials_arkode.dll"
function check_deps()
    global libsundials_kinsol
    if !isfile(libsundials_kinsol)
        error("$(libsundials_kinsol) does not exist, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    if Libdl.dlopen_e(libsundials_kinsol) == C_NULL
        error("$(libsundials_kinsol) cannot be opened, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    global libsundials_idas
    if !isfile(libsundials_idas)
        error("$(libsundials_idas) does not exist, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    if Libdl.dlopen_e(libsundials_idas) == C_NULL
        error("$(libsundials_idas) cannot be opened, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    global libsundials_nvecserial
    if !isfile(libsundials_nvecserial)
        error("$(libsundials_nvecserial) does not exist, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    if Libdl.dlopen_e(libsundials_nvecserial) == C_NULL
        error("$(libsundials_nvecserial) cannot be opened, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    global libsundials_cvodes
    if !isfile(libsundials_cvodes)
        error("$(libsundials_cvodes) does not exist, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    if Libdl.dlopen_e(libsundials_cvodes) == C_NULL
        error("$(libsundials_cvodes) cannot be opened, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    global libsundials_sunlinsolspfgmr
    if !isfile(libsundials_sunlinsolspfgmr)
        error("$(libsundials_sunlinsolspfgmr) does not exist, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    if Libdl.dlopen_e(libsundials_sunlinsolspfgmr) == C_NULL
        error("$(libsundials_sunlinsolspfgmr) cannot be opened, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    global libsundials_sunmatrixdense
    if !isfile(libsundials_sunmatrixdense)
        error("$(libsundials_sunmatrixdense) does not exist, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    if Libdl.dlopen_e(libsundials_sunmatrixdense) == C_NULL
        error("$(libsundials_sunmatrixdense) cannot be opened, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    global libsundials_sunlinsolspbcgs
    if !isfile(libsundials_sunlinsolspbcgs)
        error("$(libsundials_sunlinsolspbcgs) does not exist, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    if Libdl.dlopen_e(libsundials_sunlinsolspbcgs) == C_NULL
        error("$(libsundials_sunlinsolspbcgs) cannot be opened, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    global libsundials_sunlinsoldense
    if !isfile(libsundials_sunlinsoldense)
        error("$(libsundials_sunlinsoldense) does not exist, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    if Libdl.dlopen_e(libsundials_sunlinsoldense) == C_NULL
        error("$(libsundials_sunlinsoldense) cannot be opened, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    global libsundials_sunlinsolspgmr
    if !isfile(libsundials_sunlinsolspgmr)
        error("$(libsundials_sunlinsolspgmr) does not exist, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    if Libdl.dlopen_e(libsundials_sunlinsolspgmr) == C_NULL
        error("$(libsundials_sunlinsolspgmr) cannot be opened, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    global libsundials_sunlinsolpcg
    if !isfile(libsundials_sunlinsolpcg)
        error("$(libsundials_sunlinsolpcg) does not exist, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    if Libdl.dlopen_e(libsundials_sunlinsolpcg) == C_NULL
        error("$(libsundials_sunlinsolpcg) cannot be opened, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    global libsundials_sunlinsolsptfqmr
    if !isfile(libsundials_sunlinsolsptfqmr)
        error("$(libsundials_sunlinsolsptfqmr) does not exist, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    if Libdl.dlopen_e(libsundials_sunlinsolsptfqmr) == C_NULL
        error("$(libsundials_sunlinsolsptfqmr) cannot be opened, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    global libsundials_sunmatrixsparse
    if !isfile(libsundials_sunmatrixsparse)
        error("$(libsundials_sunmatrixsparse) does not exist, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    if Libdl.dlopen_e(libsundials_sunmatrixsparse) == C_NULL
        error("$(libsundials_sunmatrixsparse) cannot be opened, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    global libsundials_sunlinsolband
    if !isfile(libsundials_sunlinsolband)
        error("$(libsundials_sunlinsolband) does not exist, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    if Libdl.dlopen_e(libsundials_sunlinsolband) == C_NULL
        error("$(libsundials_sunlinsolband) cannot be opened, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    global libsundials_sunmatrixband
    if !isfile(libsundials_sunmatrixband)
        error("$(libsundials_sunmatrixband) does not exist, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    if Libdl.dlopen_e(libsundials_sunmatrixband) == C_NULL
        error("$(libsundials_sunmatrixband) cannot be opened, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    global libsundials_arkode
    if !isfile(libsundials_arkode)
        error("$(libsundials_arkode) does not exist, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

    if Libdl.dlopen_e(libsundials_arkode) == C_NULL
        error("$(libsundials_arkode) cannot be opened, Please re-run Pkg.build(\"Sundials\"), and restart Julia.")
    end

end

So I could move all of the functions to a single file and it would still work (and libraries which aren’t installed won’t matter if they aren’t called).

@mopg

What is happening after line 133 in wrap_sundials.jl? Is that specific to the Sundials C library?

Yeah, that’s a bunch of fancy Sundials-specific parsing. For example, it has its own Vector type, so instead generating the call directly on that it generates two functions, the outer one automatically applying the type conversion:

https://github.com/JuliaDiffEq/Sundials.jl/blob/master/src/wrapped_api/cvode.jl#L117-L124

@mopg

What part of the code ensures that all header files are wrapped in different julia files and link to different library files? Right now, I am only using this (egads is the name of the library):

I believe that’s mostly from parsing where the header is from and binning it to the right file:

https://github.com/JuliaDiffEq/Sundials.jl/blob/master/src/wrap_sundials.jl#L16-L33

[mopg]

Awesome, thanks that seemed to work so far.

However, when I am now trying to use it, I am struggling with translating my C code script that calls the library to an equivalent Julia script.

For example, the pointers are giving a bit of a problem.

In my include.jl file that was generated using Clang.jl I have the following line:

mutable struct egObject
     ... # declarations (generated by Clang.jl)
     egObject() = new() # added by me
end
     
const ego = Ptr{egObject}

Now in the C script, one needs to initialize several of these ego types, through

ego context, ebody;

Now how do I do the same thing in Julia?
context = ego() does not work, which throws an error no method matching Ptr{egObject}().