Ccall c++ sort vector of String

@dmbates , I’ll try to answer your questions. If I miss some please do not hesitate to tell me.
Q: As for creating an “Rcpp for Julia”, the first question is what would it be used for?
A: It would be used to call existing code or library from Julia. Some people might have the skills and the time to re-invest the wheel and test that is actually working like a wheel. I personally do not. Julia itself exist because C/C++ and other projects written in those languages exist. These languages have been around forever and will be around in the next 50 years. So in my view being able to interface smoothly with these languages is key. Julia exists because it is standing on the shoulders of giants.

You:If you have a C++ library that you really want to access from within Julia, the best way is to write a C-callable wrapper.
Me: Yes that is exactly what I am trying to do, unfortunately I did not find a lot of example to learn from. I think that @Ronis_BR and @Gnimuc examples are very good and should be advertised much more. I actually thank them for sharing their knowledge. I wish there are more example out there.

You: I would say, again not entirely frivolously, that the awkwardness in the interface is not because Julia is deficient but because C++ is weird.
Me: I am not going to comment on “C++” being weird. I just find your comment poor and sad. I feel sorry that you cannot appreciate how amazing that language is. I believe all language are unique and amazing in their own way whether it is R, Python, C, C++ or Julia.

I should have been more careful with my last comment about C++ being weird. I was thinking of the linkage, which is how different languages must communicate with each other. If you have a C or Fortran header file you can determine how to call a function. To do the same with a C++ header file you essentially need to have a C++ compiler, because of name mangling.

2 Likes

I think it’s pretty much optimal. Note that it is also possible to just continue using the (C++) StdVector and convert to (Julia) Vector only when needed. The trade-off is that conversion to Vector copies the data, but index operations become much faster than on a StdVector. So the best approach depends on what you need to do with the result.

4 Likes

I have tried on a windows machine and I get the following error.
Any idea how to solve that?

D:\Code\test>clang++ --std=c++17 -shared -undefined dynamic_lookup -o libsort.dll 
-I C:\Users\gitboy\.julia\artifacts\ddaec29ab3e1def8127a1757203ee4a75c1bbead\include\
-ID:\julia-1.7.1-win64\julia-1.7.1\include\julia
-LD:\julia-1.7.1-win64\julia-1.7.1\lib\julia sort.cpp

In file included from sort.cpp:1:
In file included from C:\Users\gitboy\.julia\artifacts\ddaec29ab3e1def8127a1757203ee4a75c1bbead\include\jlcxx/jlcxx.hpp:13:
In file included from C:\Users\gitboy\.julia\artifacts\ddaec29ab3e1def8127a1757203ee4a75c1bbead\include\jlcxx/julia_headers.hpp:15:
In file included from D:\julia-1.7.1-win64\julia-1.7.1\include\julia\julia.h:12:
In file included from D:\julia-1.7.1-win64\julia-1.7.1\include\julia\julia_fasttls.h:14:
  D:\julia-1.7.1-win64\julia-1.7.1\include\julia/dirpath.h:10:9: warning: 'PATH_MAX' macro
    redefined [-Wmacro-refedined]
  #define PATH_MAX MAX_PATH
          ^
  D:\llvm-mingw-20211002-msvcrt-x86_64\llvm-mingw-20211002-msvcrt-x86_64\include\limits.h:20:9:
      previous definition is here
  #define PATH_MAX   260
          ^
  1 warning generated
ld.lld error undefined symbol: __declspec(dllimport) jl_symbol
>>> referenced by C:\Users\gitboy\AppData\Local\Temp\2\sort-3bb21c.o: (jlcxx::FunctionWrapperBase& jlcxx::
Module::method<void, std::__1::vector<std::_1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator
<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1
...etc

Calling C++ to sort a vector of strings is more complicated than it seems. The issue is that C++ vectors and Julia vectors are different types, and so are C++ strings and Julia strings. You first need to decide whether you want to use C++ vectors or Julia vectors, C++ strings or Julia strings, or whether you want to convert them as you call C++. Converting needs to copy all data and is thus a bit slower, and if you pass the converted vector back to Julia you get a new vector, you didn’t sort the original vector. You can use C++ vectors and strings from Julia (that’s straightforward). You can also use Julia vector or strings from C++, but that’s more complicate since these new types are not std::vector or std::string, so they won’t have any of the functionality you expect.

Another complication comes form the fact that C++ uses manual memory management, whereas Julia is garbage collected. For any object you handle in C++, you need to decide who “owns” the object, i.e. who is responsible for destroy it when it is no longer needed. A lot of C++'s low-level complexity shines through here. You can use std::shared_ptr<std::vector<...>> or std::shared_ptr<std::string> here to simplify this, if your library allows you to do that.

If you want to use a large, existing C++ library, then I assume you’d be using C++ vectors and C++ strings, and use these from Julia. That’s straightforward; they type CxxVector{T} is a subtype of AbstractVector, and CxxString is a subtype of AbstractString, so Julia will just handle these types “as usual”.

CxxWrap has a nice story for this. It’s a bit more complex that one would like, but there is a good reason for this. I am not aware of a tutorial for what you are trying to do.

If you want to make something really simple work, then sort a C array of C ints instead, i.e. call the equivalent of

int arr[10] = { ... };
std::sort(&arr[0], &arr[10], std::less<int>());

from Julia.

-erik

7 Likes

I haven’t tried using clang on Windows yet, but it looks to me like you are not linking to libjulia and libcxxwrap-julia. Even if you link in those, you probably need to compile libcxxwrap-julia from source. I recommend using CMake to set up a project similar to the one here:

That way CMake should set up all link instructions correctly.

3 Likes

6 posts were split to a new topic: Windows development without admin privileges

Hi @barche, thank you , i have tried to use your cxxwrap-template. Here are the steps I did:

  • download portable version of VS Code
    -install the cmake extension
    -downloaded cmake 3.22 portable and added it to PATH
    -I cloned your cxxwrap-template repo with git on my local drive
    -I have opened the folder in VS Code
    -I have opened the file called CMakeLists.txt
    -I have run the file using the button at the bottom of VS code (like you describe on github)
    -I got an error that the variable JlCxx_DIR was not set, so I did set it to C:\Users\gitboy\.julia\artifacts\...\lib\cmake\JlCxx and re-run

I got the following message:

Cmake Warning at CMakeLists.txt:8 (find_package):
Could not find a configuration file for package "JlCxx" that is compatible with requested version "".
The following configuration files were considered but not accepted:
C:\Users\gitboy\.julia\artifacts\...\lib\cmake\JlCxx\JlCxxConfig.cmake, version: 0.9.0 (64bit)

-- Configuration incomplete, errors occured!
CMake Error at CMakeLists.txt:9 (get_target_property):
get_target_property() called with non-existent target "JlCxx::cxxwrap_julia".

CMake Error at CMakeLists.txt:10 (get_file_name_component):
get_file_name_component called with incorrect number of arguments

I am not sure to understand what is missing, would you have detailed steps on how I can make this work? Thank you

In the workspace settings of cxxwrap-template you should add something like this:

{
    "cmake.configureSettings": {
        "JlCxx_DIR": "C:/Users/user/source/repos/libcxxwrap-julia/build",
        "Julia_EXECUTABLE": "C:/Users/user/AppData/Local/Programs/Julia-1.7.1/bin/julia.exe"
    }
}

Here, C:/Users/user/source/repos/libcxxwrap-julia/build is the build directory of the libcxxwrap that you compiled with the same compiler as you are using to develop the wrapper.

Please note that this topic is closed, we should continue in Windows development without admin privileges (@stevengj you may want to move this post and the one above it, thanks :slight_smile: )

2 Likes

Thank you @barche for the details. I’ll continue posting here until they move your post to the other thread so people who are looking for help can follow the discussion.
I did what you said and it seemed to have work as I got the following messages:

[cmake] -- Configuration done
[cmake] -- Generating done
[cmake] -- Build files have been written to: D:/cxxwrap-template/build

But I don’t see any .dll or library being created that I can use with @wrapmodule.

Does your CMakeLists.txt have an install target? By default, those binaries should be located in $CMAKE_INSTALL_PREFIX/bin.

This just completed the build file generation phase, you still need to compile by hitting the “Build” button at the bottom or executing the VScode command CMake: Build. The compiled files should then appear in a subdir build

2 Likes

CMakeLists.txt files that I use exactly the same as the one in the git repo of libcxxwrap-julia and cxxwrap-template.
Following instruction of @barche , I can see now the dll in the folder D:\libcxxwrap-julia\build\bin
Then I tried tried to build the dll for cxxwrap-template and got a lot warning messages like that one:

D:\libbcxxwrap-julia\include\jlcxx\module.hpp(183,91): warning C4251: 'jlcxx:::FunctionWrapperBase::m_return_type': struct 'std::pair<jl_datatype_t *, jl_datatype_t *>' needs to have dll-interface to be used by clients of class 'jlcxx::FunctionWrapperBase' [D:\cxxwrap-template\build\foo.vcxproj]

It looks like the dlls I just created for libcxxwrap-julia are not found when I build cxxwrap-template.

Only warnings and no errors? These should be harmless, so there should be a new dll in the cxxwrap-template/build directory.

1 Like

Seems there is no error. You are right there is a dll in the folder called D:\cxxwrap-template\build\Debug\foo.dll. There is no bin folder like libcxxwrap-julia. When I try to use the dll in the file testfoo.jl in module ModFoo I have the following error in Julia:

Could not load library "D:\cxxwrap-template\build\Debug\foo.dll"
The specified procedure  could not be found.
Stacktrace:
 [1] dlopen(s::String, flag::UInt32; throw_error::Bool)
...etc

Is this after loading CxxWrap? Most likely this means that the libcxxwrap-julia DLLs can’t be found. This happens if either CxxWrap is not loaded, or loaded but not using an Overrides.toml to direct to the libcxxwrap-julia you built yourself.

1 Like

Here is what I have tried after creating the dlls.

Attempt 1:
-Install CxxWrap by typing add CxxWrap without Overrides.toml file created.
-load Cxxwrap with using CxxWrap which loads ok.
-try to run the module in testfoo.jl file
(I get the error message I sent before)

Attempt 2:
-After removing CxxWrap from attempt 1, I created the Overrides.toml file as described in GitHub - JuliaInterop/libcxxwrap-julia: C++ library for backing CxxWrap.jl
-I install again CxxWrap and then I try to load it with using CxxWrap and I get the following error message in Julia:

ERROR: InitError: could not load library "C:\Users\gitboy\.julia\artifacts\...\bin\libcxxwrap_julia_stl.dll"
The specified procedure could not be found.
Stacktrace:
[1] dlopen(s::String, flag::UInt32; throw_error::Bool)
...etc

Some similarity here : InitError: could not load library "...libcxxwrap_julia_stl.dll" on windows

I followed the steps suggested by @hiemstar and got a different error.

Let’s continue the discussion about the Overrides.toml in the issue linked below, can you please post your full Overrides.toml and the paths to the built DLLs there?

1 Like

I finally manage to make it work with both MS compiler and LLVM on windows. I can’t explain what the issue was but basically I needed to have a clean installation of Julia ,i.e. the folder C:\Users\gitboy\.julia must be empty except for the Overrides.toml file in the folder artifacts.
Thank you all for your help.