# UNICODE string from C++ to Julia and vice versa

**URL:** https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272
**Category:** General Usage
**Tags:** question, embedding, examples, cxx
**Created:** [April 18, 2017, 7:32pm UTC](https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272 "2017-04-18T19:32:10Z")
**Posts on this page:** 20
**Page:** 1

<div class="post-metadata">

### Author: ![Martin\_Florek](https://avatars.discourse-cdn.com/v4/letter/m/f4b2a3/32.png) [@Martin\_Florek](https://discourse.julialang.org/u/Martin_Florek)
#### Post date: [April 18, 2017, 7:32pm UTC](https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272/1 "2017-04-18T19:32:10Z")

</div>

I’m using [embedding Julia](https://docs.julialang.org/en/stable/manual/embedding/) in C++ project. Now I need to get any unicode string to Julia, e.g…  
`std::wstring test = L"šČô_φ_привет_일보";`  
Tips & hints or a model example for solutions, please. Thanks.

---

<div class="post-metadata">

### Author: ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [April 19, 2017, 1:52am UTC](https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272/2 "2017-04-19T01:52:03Z")

</div>

> [@Martin\_Florek](#):
>
> “šČô\_φ\_привет\_일보”

Just pasting `"šČô_φ_привет_일보"` into Julia code works fine. Julia strings are all unicode.

(More precisely Julia strings are UTF8-encoded Unicode. Any decent editor should be able to edit UTF8 text, and if you paste Unicode into a UTF8 document it should convert the encoding to UTF8 as needed.)

---

<div class="post-metadata">

### Author: ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [April 19, 2017, 1:57am UTC](https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272/3 "2017-04-19T01:57:33Z")

</div>

Sorry, I just realized that because you are embedding Julia, you probably have a `wchar_t` string in C++ that you want to convert to a Unicode `String` in Julia at _runtime._

Since your `std::wstring` is a `wchar_t*` string (UTF-16 on Windows and UTF-32 everywhere else), it a different encoding than the UTF-8 encoding used by the Julia `String` type, so you need to either convert it (the `transcode` function in Base can do this) or use the `WString` type in the `LegacyStrings` package (to use it in Julia without conversion, e.g. to share memory for large strings).

e.g. suppose you pass the data as a `wchar_t *p` (or `p::Ptr{Cwchar_t}` in Julia) and a length `L`. Then you could call `transcode(String, unsafe_wrap(p, L))` to get a Julia `String` object.

---

<div class="post-metadata">

### Author: ![Martin\_Florek](https://avatars.discourse-cdn.com/v4/letter/m/f4b2a3/32.png) [@Martin\_Florek](https://discourse.julialang.org/u/Martin_Florek)
#### Post date: [April 19, 2017, 9:18am UTC](https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272/4 "2017-04-19T09:18:44Z")

</div>

Thanks @stevengj. I’m starting to understand, but how sensibly to get the data as a `wchar_t * p` to Julia with the help of `julia.h`? I have got

> jl\_function\_t _funcTransc = jl\_get\_function(jl\_base\_module, “transcode”);  
> std::wstring testS = L"a\_šČô\_φ\_привет\_일보";  
> const wchar\_t_ s = testS.c\_str();

> uint64\_t Len = testS.length();  
> jl\_value\_t\* L = jl\_box\_uint64(Len);

> jl\_value\_t\* p = ???  
> jl\_call2(funcTransc, …);

---

<div class="post-metadata">

### Author: ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [April 19, 2017, 12:24pm UTC](https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272/5 "2017-04-19T12:24:38Z")

</div>

> [@Martin\_Florek](#):
>
> jl\_function\_t \*funcTransc = jl\_get\_function(jl\_base\_module, “transcode”);

I would write as much as possible of your glue/conversion code in Julia rather than in C/C++. Will be a lot easier. You can even call cfunction in Julia to return a C function pointer to C, so that you don’t have to mess around with jl\_call and jl\_value\_t\* conversions.

---

<div class="post-metadata">

### Author: ![Martin\_Florek](https://avatars.discourse-cdn.com/v4/letter/m/f4b2a3/32.png) [@Martin\_Florek](https://discourse.julialang.org/u/Martin_Florek)
#### Post date: [April 19, 2017, 1:00pm UTC](https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272/6 "2017-04-19T13:00:36Z")

</div>

I create the API for application executables on each PC. It is clear to me that I want to work as little as possible with C ++, but I need a C++ connector to Julia.  
Some idea or another way how get unicode string from C++ to Julia?  
Thanks.

---

<div class="post-metadata">

### Author: ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [April 19, 2017, 4:43pm UTC](https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272/7 "2017-04-19T16:43:31Z")

</div>

> [@Martin\_Florek](#):
>
> I create the API for application executables on each PC. It is clear to me that I want to work as little as possible with C ++, but I need a C++ connector to Julia.

You can write the glue code in Julia and then call it from C++.

For example, you could write:

```c
jl_value_t *wstr2jl = jl_eval_string("wstr2jl(wptr, len) = transcode(String, unsafe_wrap(Ptr{Cwchar_t}(wptr), len))");

```

and then do

```c
std::wstring test = L"šČô_φ_привет_일보";
jl_value_t *jtest = jl_call2(wstr2jl, jl_box_voidpointer(test.c_str()), jl_box_uint64(test.size()));

```

---

<div class="post-metadata">

### Author: ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)
#### Post date: [April 19, 2017, 5:23pm UTC](https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272/8 "2017-04-19T17:23:14Z")

</div>

> [@stevengj](#):
>
> jl\_value\_t \*jtest = jl\_call2(wstr2jl, jl\_box\_voidpointer(test.c\_str()), jl\_box\_uint64(test.size()));

Note that the result of `jl_box_voidpointer` has to be rooted before calling `jl_box_uint64`.

---

<div class="post-metadata">

### Author: ![Martin\_Florek](https://avatars.discourse-cdn.com/v4/letter/m/f4b2a3/32.png) [@Martin\_Florek](https://discourse.julialang.org/u/Martin_Florek)
#### Post date: [April 20, 2017, 7:52am UTC](https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272/9 "2017-04-20T07:52:39Z")

</div>

Thanks guys @stevengj and @yuyichao , but I got error with jl\_box\_voidpointer. Probably I can not get a pointer on a string to Julia correctly. First I copy your code a got error

```julia
Error (active) argument of type "const wchar_t *" is incompatible with parameter of type "void *" 
Error	C2664	'jl_value_t *jl_box_voidpointer(void *)': cannot convert argument 1 from 'const wchar_t *' to 'void *'

```

For the Second I modify code for void pointer and add `println()`

```julia
jl_value_t *wstr2jl = jl_eval_string("function wstr2jl(wptr, len) println(1); out = transcode(String, unsafe_wrap(Ptr{Cwchar_t}(wptr), len)); println(2); out end");
std::wstring testString = L"šČô_φ_привет_일보";
void *my_ptr1 = static_cast <void*>(const_cast <wchar_t*>(testString.c_str()));
jl_value_t *jtest = jl_call2(wstr2jl, jl_box_voidpointer(my_ptr1), jl_box_uint64(testString.size()));

```

but the `println("2")` has not happened. How to get a correct pointer `const wchar_t *` to Julia?

---

<div class="post-metadata">

### Author: ![barche](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/barche/32/79_2.png) [@barche](https://discourse.julialang.org/u/barche)
#### Post date: [April 20, 2017, 4:31pm UTC](https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272/10 "2017-04-20T16:31:17Z")

</div>

The `unsafe_wrap` function should take 3 arguments, the first being `Array` in this case:

```julia
unsafe_wrap(Array,Ptr{Cwchar_t}(wptr), len)

```

Inspired by this thread, I’m also attempting to add wstring support to [CxxWrap.jl](https://github.com/JuliaInterop/CxxWrap.jl), though it fails on Windows right now. The conversion functions that get called are here:  
[https://github.com/JuliaInterop/CxxWrap.jl/blob/master/src/CxxWrap.jl#L405-L406](https://github.com/JuliaInterop/CxxWrap.jl/blob/master/src/CxxWrap.jl#L405-L406)

edit: Actually, it only fails on MSVC, not on MinGW. The failing string compare is here:  
[https://ci.appveyor.com/project/barche/cxxwrap-jl/build/1.0.164/job/xmitpqjk6s1ne63v#L1278](https://ci.appveyor.com/project/barche/cxxwrap-jl/build/1.0.164/job/xmitpqjk6s1ne63v#L1278)

No idea why this fails…

---

<div class="post-metadata">

### Author: ![Martin\_Florek](https://avatars.discourse-cdn.com/v4/letter/m/f4b2a3/32.png) [@Martin\_Florek](https://discourse.julialang.org/u/Martin_Florek)
#### Post date: [April 21, 2017, 6:39am UTC](https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272/11 "2017-04-21T06:39:47Z")

</div>

Thanks @barche. This is Julia part, but how does your part look in c ++ ? i.e. How do you get the `wchar_t` pointer to Julie? If using a generic function ` jl_box_voidpointer()`, then I must probably cast type `Ptr{Void}` to type `Ptr{Cwchar_t}`!?  
Correct me if I’m wrong. Thank you very much for taking time for me.  
With add `Array` to `unsafe_wrap` this get to `println(2)` 🙂 (Windows 8.1), now just check.

---

<div class="post-metadata">

### Author: ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [April 21, 2017, 1:29pm UTC](https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272/12 "2017-04-21T13:29:38Z")

</div>

> [@Martin\_Florek](#):
>
> hen I must probably cast type Ptr{Void} to type Ptr{Cwchar\_t}

You do this in the Julia code, as I showed in my example. (I forgot to add the `Array` first argument, as @barche pointed out, and @yuyichao pointed out that you need to root the pointer). Then you call the Julia code from C++. The point, as I’ve been trying to emphasize, is to do as little as possible in C++; write all the glue in Julia.

---

<div class="post-metadata">

### Author: ![Martin\_Florek](https://avatars.discourse-cdn.com/v4/letter/m/f4b2a3/32.png) [@Martin\_Florek](https://discourse.julialang.org/u/Martin_Florek)
#### Post date: [April 21, 2017, 1:47pm UTC](https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272/13 "2017-04-21T13:47:23Z")

</div>

Thanks from C++ to Julia it works perfectly and how I get unicode back (in reverse)? From Julia to C++. I have got

```julia
jl_value_t *jl2wstr = jl_eval_string("function jl2wstr() s = \"a_šČô_φ_привет_일보\"; out = transcode(Cwchar_t, s); println(out); out end");
jl_array_t *jl2wstrOut = (jl_array_t*)jl_call0(jl2wstr);
	if (jl2wstrOut)
	{
		wchar_t* xData2 = (wchar_t*)jl_array_data(jl2wstrOut);
		f << xData2;
		printf("\noutStr = [");
		for (size_t i = 0; i < jl_array_len(jl2wstrOut); ++i)
			printf("%X ", xData2[i]);
		printf("]\n");
	}

```

But conversion is not good. I get  
`UInt16[0x0061,0x005f,0x009a,0x003f,0xdb37,0xdf9f,0x003f,0x003f,0x003f,0x003f,0x003f,0x003f,0x005f,0x003f,0x003f`  
Instead of this  
`UInt16[0x0061,0x005f,0x0161,0x010c,0x00f4,0x005f,0x03c6,0x005f,0x043f,0x0440,0x0438,0x0432,0x0435,0x0442,0x005f,0xc77c,0xbcf4]`

---

<div class="post-metadata">

### Author: ![barche](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/barche/32/79_2.png) [@barche](https://discourse.julialang.org/u/barche)
#### Post date: [April 21, 2017, 2:49pm UTC](https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272/14 "2017-04-21T14:49:26Z")

</div>

> [@Martin\_Florek](#):
>
> How do you get the wchar\_t pointer to Julie?

I use `jl_new_bits` to box the pointer in the correct type:

```C
jl_value_t* wchar_dt = jl_get_global(jl_base_module, jl_symbol("Cwchar_t"));
jl_value_t* ptr_dt = jl_apply_type((jl_value_t*)jl_pointer_type, jl_svec1(wchar_dt));
jl_value_t* ptr = jl_new_bits(ptr_dt, (void*)&x); // Boxed Ptr{Cwchar_t} to pass to Julia, x is the C pointer

```

This is pieced together a bit from different places around CxxWrap.jl, so not tested in the form above (and is missing GC rooting, and is v0.5-only). The objective is also different, since this code is used to make wrapping C++ functions returning or consuming std::wstrings, so if you’re using the embedding interface directly I fully support @stevengj 's advice to do as much as possible in Julia (obviously way simpler, too).

---

<div class="post-metadata">

### Author: ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [April 21, 2017, 6:27pm UTC](https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272/15 "2017-04-21T18:27:24Z")

</div>

> [@Martin\_Florek](#):
>
> `jl_eval_string("function jl2wstr() s = \"a_šČô_φ_привет_일보\";`

The problem is probably that you are entering this string in C++, not in Julia. i.e. the argument to `jl_eval_string` is a C++ string… how is that encoded by your compiler? (The same Julia code works fine for me in the REPL, so I think the problem is just the encoding of the string by your C++ compiler.)

---

<div class="post-metadata">

### Author: ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [April 21, 2017, 6:36pm UTC](https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272/16 "2017-04-21T18:36:15Z")

</div>

> [@stevengj](#):
>
> `jl_eval_string("...")`

If your compiler supports C++11, you should do `jl_eval_string(u8"...")` so that the Julia code is UTF8-encoded.

---

<div class="post-metadata">

### Author: ![barche](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/barche/32/79_2.png) [@barche](https://discourse.julialang.org/u/barche)
#### Post date: [April 23, 2017, 9:29pm UTC](https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272/17 "2017-04-23T21:29:57Z")

</div>

Not sure if it’s related, but for CxxWrap I had to change the encoding of the C++ source file from UTF-8 to UTF-8-BOM for MSVC to do the right thing.

---

<div class="post-metadata">

### Author: ![Martin\_Florek](https://avatars.discourse-cdn.com/v4/letter/m/f4b2a3/32.png) [@Martin\_Florek](https://discourse.julialang.org/u/Martin_Florek)
#### Post date: [April 24, 2017, 7:58am UTC](https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272/18 "2017-04-24T07:58:16Z")

</div>

I use in MSVC (File \>\>Advanced Save options… \>\> Unicode (UTF-8 without signature)) for correct showing.  
But I still do not know how to interpret pointer to `Array{Cwchar_t}` **from Julia back to C++**.??  
Can not I use jl\_unbox\_voidpointer()?  
Is this the right way?  
`jl_array_t *jl2wstrOut = (jl_array_t*)jl_call0(jl2wstr);`  
`wchar_t* xData2 = (wchar_t*)jl_array_data(jl2wstrOut);`

---

<div class="post-metadata">

### Author: ![barche](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/barche/32/79_2.png) [@barche](https://discourse.julialang.org/u/barche)
#### Post date: [April 24, 2017, 8:58am UTC](https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272/19 "2017-04-24T08:58:49Z")

</div>

> [@Martin\_Florek](#):
>
> Is this the right way?  
> jl\_array\_t _jl2wstrOut = (jl\_array\_t_)jl\_call0(jl2wstr);  
> wchar\_t\* xData2 = (wchar\_t\*)jl\_array\_data(jl2wstrOut);

That’s basically what I’m doing too, does it give an error?

---

<div class="post-metadata">

### Author: ![Martin\_Florek](https://avatars.discourse-cdn.com/v4/letter/m/f4b2a3/32.png) [@Martin\_Florek](https://discourse.julialang.org/u/Martin_Florek)
#### Post date: [April 26, 2017, 7:32am UTC](https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272/20 "2017-04-26T07:32:53Z")

</div>

@barche If I assign the pointer instead of X, the program fail. See the code.

```julia
jl_value_t* wchar_dt = jl_get_global(jl_base_module, jl_symbol("Cwchar_t"));
jl_value_t* ptr_dt = jl_apply_type((jl_value_t*)jl_pointer_type, jl_svec1(wchar_dt));
//jl_value_t* ptr = jl_new_bits(ptr_dt, (void*)&testString); //pointer to std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > *
const wchar_t* ap = testString.c_str();
jl_value_t* ptr = jl_new_bits(ptr_dt, (void*)&ap); // pointer to const wchar_t **

```

But If assign e.g. `const wchar_t **` or `std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > *` program run. If assign `ap` = `const wchar_t *`, `testString.c_str()` or `testString.data()` the program fail. This is not good, because the function `jl_new_bits` expects `void * data`.  
Why I need to assign pointer to pointer instead of pointer?  
Thanks.

[Next page](https://discourse.julialang.org/t/unicode-string-from-c-to-julia-and-vice-versa/3272.md?page=2)
