I am embedding Julia in a C++ project. This work fine on Unix platform, but I am having some trouble with the windows build available for download on julialang.org
The main blocker is that while a LLVM.dll shared library is available, no .lib file is provided for linking.
The LLVM dll does not follow the same naming scheme as on other plaforms
{PREFIX}LLVM-{VERSION){SUFFIX}
In other word, I would have expected LLVM-3.7.1.dll instead of LLVM-3.7.dll.
The name of base julia library to link with has a .a suffix instead of the windows standard .lib which forces me to add it to CMAKE_FIND_LIBRARY_SUFFIXES in my cmake to be able to find it.
.lib would only be standard if Julia were built with MSVC. It is not, we use MinGW. Embedding Julia in a MSVC-built application may work, but it’s not recommended.
I don’t think you should usually need to link directly against libLLVM, since libjulia already does. LLVM’s build system appears to create a libLLVM.dll.a import library but not install it.
@barche the use case is the Pure C++ tests of xtensor-julia. I have two test suites
One driven by Julia, with the standard Pkg.test
One driven by C++ which instantiate the Julia interpreter and does more thorough testing. (and uses the FindJulia.cmake that we discussed on another channel).
Both work well on Unix and OS-X, they do require linking with LLVM.
It’s pretty large, and there are tools you can use to create import libraries from existing dll’s. Building the embedding example works fine with mingw without needing a .dll.a.
How big is the .dll.a? Generally speaking, .lib for dynamic linking on Windows are quite small since they contained the exported symbols only. Does mingw have a different behavior (something similar to gcc that exports all the symbols)?
Besides I don’t think that it is sane to make packages depend on tools to create import libraries from existing dll’s; such tools should be reserved for fixing issues, they should not be incorporated in build processes.
When I did this sort of thing for Julia 0.5.0 I needed to link with libjulia and libopenlibm but not with LLVM.
I’m not sure it’s really needed anymore but at the time I generated a .lib file from the .dll file with this code:
# The Julia Windows binary package provides libjulia.dll and
# libopenlibm.dll but not the corresponding .lib files needed to link
# to them (unless you do dynamic loading at runtime). It is however
# possible to build such files from the corresponding dll yourself,
# which is what we do here.
#
# Note: This is hopefully a temporary solution since it should be
# possible to use .dll.a files in place of .lib files and Julia
# provides a libjulia.dll.a file but not a libopenlibm.dll.a file though
# that is expected to change for a later 0.5.x release.
function build_lib_files()
juliadir = joinpath(PATH_TO_PROJECT, "3rdparty/Julia")
dumpbin, lib = find_vc_binaries()
build_lib_file(juliadir, dumpbin, lib, "libjulia")
build_lib_file(juliadir, dumpbin, lib, "libopenlibm")
end
# This tries to find the dumpbin.exe and lib.exe binaries in a
# hopefully general way.
function find_vc_binaries()
reg_query = readstring(`reg query "HKEY_LOCAL_MACHINE\\SOFTWARE\\Wow6432Node\\Microsoft\\VisualStudio\\12.0" /v InstallDir`)
vc_install_dir = strip(split(reg_query, "REG_SZ")[end])
vc_bin_dir = joinpath(vc_install_dir, "../../VC/bin")
return joinpath(vc_bin_dir, "dumpbin.exe"), joinpath(vc_bin_dir, "lib.exe")
end
function build_lib_file(juliadir, dumpbin, lib, libname)
dllname = joinpath(juliadir, "bin", "$(libname).dll")
dump_output = readstring(`$(dumpbin) /exports $(dllname)`)
symbols = String[]
for line in split(dump_output, "\r\n")
m = match(r"^\s+\d+\s+[0-9A-F]+\s+[0-9A-F]+\s+(\w+)", line)
if m != nothing
push!(symbols, m.captures[1])
end
end
def_filename = joinpath(juliadir, "bin", "$(libname).def")
lib_filename = joinpath(juliadir, "lib", "$(libname).lib")
f = open(def_filename, "w")
print(f, "EXPORTS\r\n", join(symbols, "\r\n"), "\r\n")
close(f)
run(`$(lib) /def:$(def_filename) /OUT:$(lib_filename)`)
end
To enable users to build upon julia by using it as a library, I don’t think that requiring them to recompose import libraries from dlls is a good approach.
Besides being fragile and requiring tools such as dumpbin at build time, it also creates a huge barrier to using the interpreter as a library.
Already, the absence of cmake config files and the need to use FindJulia.cmake scripts that make use of the Julia runtime to get the required information complicates the picture. This would bring a much higher level of complexity.
tl;dr: there are two things you might try to get you further on windows:
make sure Julia is on the path;
switch from julia_init to julia_init_with_image, passing the right paths as arguments.
When embedding on windows, Julia needs to either be on the path or all of the DLLs in the bin directory need to live in the same directory as your executable (or it may be sufficient that the Julia bin directory is the current directory when you try to initialize the Julia runtime – a hack I exploit in mexjulia to eliminate the requirement for users that Julia be on the path). In 0.5 at least, I am able to build mexjulia against MSVC, as well as mingw, using only the .dll.a files that ship with Julia. I have yet to test it, but I certainly hope the embedding changes in 0.6 haven’t brought a regression on that front.
I have also been unable in the past to use the julia_init function. I have always had to use julia_init_with_image. That has always seemed unnecessary to me, as julia_init lives in libjulia and can therefore always find the path of the library in which it lives. The “path guessing” in julia_init (at least through 0.5) was based on the calling executable path (and not libjulia), which means it was basically useless for the embedding case unless you happen to be building your executable into the same directory as the julia executable (not a common or, I would suggest, desirable thing). Maybe the path guessing works for the general embedding case now. That would be a welcome improvement.
There are certainly issues with the Julia embedding story but having to link with a .dll.a rather than a smaller .lib file is a very minor one (I’m assuming the absence of the libopenlibm.dll.a file is solved by now and I have never experienced any need to link explicitly with LLVM myself). The major issues I have encountered are the gotchas around correctly protecting Julia objects from garbage collection (JL_GC_PUSH with friends), getting paths right, and the need to bundle the Julia runtime to make sure that it’s compatible to what you built with.
With the caveats that it’s a couple months since I worked actively with embedding, I still had some unsolved issues, and I haven’t checked out the changes that have gone into 0.6, I tend to think that the most sane approach to embedding is using cfunction pointers for all your interaction with your Julia code and limiting the use of the Julia C API to the minimum needed for initialization and acquiring the cfunction pointers. That way it becomes quite manageable to dynamically load libjulia and do away with the build time linking trouble.
One factor that helped with the success of e.g. the Lua interpreter is how easy it was to embed it into an application. A clear C API and best practices for the build process.
In comparison, requiring users to recompose import libraries from dlls is a much higher barrier for adoption.
More generally, using a different compiler than the one of the platform for the official binaries (mingw) and not providing cmake project config files are already obstacles, but which developers can overcome.
I’m not saying this is not needed in general, but I tested the cmake version of the embedding example yesterday using the standard Julia 0.5.1 binaries and visual studio 2015, and it worked without requiring an LLVM import lib. So I still wonder under what exact circumstances the LLVM import lib is needed.
libLLVM.dll.a appears to be 26 MB, around the same size as the largest dll’s in our current binaries. There may be LLVM build system issues contributing to it being larger than it should be. If we were absolutely certain that it was required for embedding we could consider whether it’s worth it, but so far much of the evidence is indicating that it should not be necessary.
Better support for cmake, and any support at all for visual studio, will need to be contributed by the community. At the moment both are very low priorities because they introduce as many or more problems than they solve. Priorities can change, but that’s where we are today.