# Calling c function strtok in julia

**URL:** <https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738>\
**Category:** New to Julia\
**Created:** [May 29, 2019, 2:51pm UTC](https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738 "2019-05-29T14:51:32Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![mwolff](https://avatars.discourse-cdn.com/v4/letter/m/c2a13f/32.png) [@mwolff](https://discourse.julialang.org/u/mwolff)\
**Post date:** [May 29, 2019, 2:51pm UTC](https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738/1 "2019-05-29T14:51:32Z")

</div>

Hello,

i just wrapped the C function strtok to Julia to practice. I wanted to try it myself only with the help of the documentation.  
Can I leave this approach as it is or are there other possibilities? I didn’t know exactly how to handle the NULL better.

The Original example comes from [http://www.cplusplus.com/reference/cstring/strtok/](http://www.cplusplus.com/reference/cstring/strtok/)

```julia
/* strtok example */
#include <stdio.h>
#include <string.h>

int main ()
{
  char str[] ="- This, a sample string.";
  char * pch;
  printf ("Splitting string \"%s\" into tokens:\n",str);
  pch = strtok (str," ,.-");
  while (pch != NULL)
  {
    printf ("%s\n",pch);
    pch = strtok (NULL, " ,.-");
  }
  return 0;
}

```

in Julia it looks like this…

```julia
function myStrTok(str::String, delim::String)
	t = ccall((:strtok, LIB), Cstring, (Cstring,Cstring), str, delim)
	return unsafe_string(t)
end

function myStrTok(delim::String)
	t = ccall((:strtok, LIB), Cstring, (Ptr{Nothing}, Cstring), C_NULL, delim)
	(t != C_NULL) ? (return unsafe_string(t)) : return ""
end

stok = "- This, a sample string."
p = myStrTok(stok, " ,.-")
while(p != "")
	global p
	println("Token : $p")
	p=myStrTok(" ,.-")
end

```

best regards

Michael

---

<div class="post-metadata">

**Author:** ![Ronis\_BR](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ronis_br/32/50999_2.png) [@Ronis\_BR](https://discourse.julialang.org/u/Ronis_BR)\
**Post date:** [May 29, 2019, 4:09pm UTC](https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738/2 "2019-05-29T16:09:41Z")

</div>

It seems fine. However, if you want to replicate that C behavior (with pointer to NULL), you can do:

```julia
function strtok(str::Union{Nothing,String}, delim::String)
    ptr = str == nothing ? Cstring(C_NULL) : convert(Cstring, pointer(str))
    ptr_tok = ccall((:strtok, "libc"), Cstring, (Cstring, Cstring), ptr, delim)
    tok = ptr_tok != C_NULL ? unsafe_string(ptr_tok) : ""
    return tok
end

stok = "- This, a sample string."
p = strtok(stok, " ,.-")

while(p ≠ "")
	global p
	println("Token : $p")
	p = strtok(nothing, " ,.-")
end

```

```julia
julia> include("strok.jl")
Token : This
Token : a
Token : sample
Token : string

```

---

<div class="post-metadata">

**Author:** ![mwolff](https://avatars.discourse-cdn.com/v4/letter/m/c2a13f/32.png) [@mwolff](https://discourse.julialang.org/u/mwolff)\
**Post date:** [May 29, 2019, 4:19pm UTC](https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738/3 "2019-05-29T16:19:24Z")

</div>

Okay, thanks Ronis\_BR 👍

a nice approach to the solution.

best regards

Michael

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [May 29, 2019, 4:24pm UTC](https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738/4 "2019-05-29T16:24:11Z")

</div>

1. This is undefined behavior since `String` in julia cannot be mutated. You must use a `Vector` or something mutable instead.
2. The function returns a pointer to the argument so you must `GC.@preserve` the first argument before you convert the returned pointer to string.
3. This is very bad.

> [@Ronis\_BR](#):
>
> ptr = str == nothing ? Cstring(C\_NULL) : convert(Cstring, pointer(str))

You need to` GC.@preserve` `str` at the least. Or the pointer you get from here can just be garbage.

---

<div class="post-metadata">

**Author:** ![Ronis\_BR](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ronis_br/32/50999_2.png) [@Ronis\_BR](https://discourse.julialang.org/u/Ronis_BR)\
**Post date:** [May 29, 2019, 4:34pm UTC](https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738/5 "2019-05-29T16:34:09Z")

</div>

> [@yuyichao](#):
>
> You need to `GC.@preserve` `str` at the least. Or the pointer you get from here can just be garbage.

Yes! Thanks to point this out, I was about to write to @mwolff that in this case the user must ensure that the str is not deallocated by the GC. It works in the example because the string is stored in a global variable, right?

---

<div class="post-metadata">

**Author:** ![mwolff](https://avatars.discourse-cdn.com/v4/letter/m/c2a13f/32.png) [@mwolff](https://discourse.julialang.org/u/mwolff)\
**Post date:** [May 29, 2019, 4:39pm UTC](https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738/6 "2019-05-29T16:39:06Z")

</div>

hmm… such things are missed in the documentary and are left out for simplicity’s sake.

---

<div class="post-metadata">

**Author:** ![Ronis\_BR](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ronis_br/32/50999_2.png) [@Ronis\_BR](https://discourse.julialang.org/u/Ronis_BR)\
**Post date:** [May 29, 2019, 4:41pm UTC](https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738/7 "2019-05-29T16:41:14Z")

</div>

> [@yuyichao](#):
>
> This is undefined behavior since `String` in julia cannot be mutated. You must use a `Vector` or something mutable instead.

@yuyichao, just to improve my understanding, I know that you cannot mutate a string in Julia. Thus, if I pass a string as argument to a function, it will not be modified after the function is called. However, in this example, it is indeed modified:

```julia
stok = "- This, a sample string."
println(stok)
p = strtok(stok, " ,.-")
println(stok)

```

```Julia
julia> include("strok.jl")
- This, a sample string.
- This a sample string.

```

Can you please explain to me please?

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [May 29, 2019, 4:43pm UTC](https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738/8 "2019-05-29T16:43:22Z")

</div>

It’ll also simply work if the compiler and GC doesn’t feel like messing with it. You are merely giving it a license to free it, which may happen at any time.

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [May 29, 2019, 4:44pm UTC](https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738/9 "2019-05-29T16:44:35Z")

</div>

You can certainly write code to mutate string in Julia, it’s just that doing that is undefined behavior. You cannot expect Julia to act in any sane way after this point.

---

<div class="post-metadata">

**Author:** ![Ronis\_BR](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ronis_br/32/50999_2.png) [@Ronis\_BR](https://discourse.julialang.org/u/Ronis_BR)\
**Post date:** [May 29, 2019, 4:51pm UTC](https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738/10 "2019-05-29T16:51:21Z")

</div>

Oh I see. Thanks @yuyichao

If you have a little spare time, can you take a look in this modified version? This integration is something I am really interested in:

```julia
function strtok(str::Union{Nothing,Vector{String}}, delim::String)
    if str == nothing
        ptr = Cstring(C_NULL)
    else
        if length(str) != 1
            error("The vector `str` must have only one element.")
        end

        GC.@preserve str (ptr = convert(Cstring,pointer(str[1])))
    end

    ptr_tok = ccall((:strtok, "libc"), Cstring, (Cstring, Cstring), ptr, delim)
    tok = ptr_tok != C_NULL ? unsafe_string(ptr_tok) : ""

    return tok
end

stok = ["- This, a sample string."]
p = strtok(stok, " ,.-")

while(p ≠ "")
	global p
	println("Token : $p")
	p = strtok(nothing, " ,.-")
end

```

---

<div class="post-metadata">

**Author:** ![mwolff](https://avatars.discourse-cdn.com/v4/letter/m/c2a13f/32.png) [@mwolff](https://discourse.julialang.org/u/mwolff)\
**Post date:** [May 29, 2019, 5:01pm UTC](https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738/11 "2019-05-29T17:01:26Z")

</div>

But in the dokumentary cstring(String) should be used for char\* in c function. Vector only if i have to allocate memory myself then Ptr{Uint8}. I think strtok does this internally, because the original string is broken and gives new strings back. Sorry i must translate everything from german to english…

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [May 29, 2019, 5:20pm UTC](https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738/12 "2019-05-29T17:20:00Z")

</div>

No, a vector of string doesn’t make any difference… You are still mutating the string.

---

<div class="post-metadata">

**Author:** ![kristoffer.carlsson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kristoffer.carlsson/32/22_2.png) [@kristoffer.carlsson](https://discourse.julialang.org/u/kristoffer.carlsson)\
**Post date:** [May 29, 2019, 5:21pm UTC](https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738/13 "2019-05-29T17:21:00Z")

</div>

The block of the GC preserve need to enclose all the uses of the pointer as well AFAIU.

---

<div class="post-metadata">

**Author:** ![Ronis\_BR](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ronis_br/32/50999_2.png) [@Ronis\_BR](https://discourse.julialang.org/u/Ronis_BR)\
**Post date:** [May 29, 2019, 5:49pm UTC](https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738/14 "2019-05-29T17:49:32Z")

</div>

But if I pass the pointer to the function instead of the string, then everything will be ok right? Like

```nohighlight
strtok(pointer(stok), " ,.-")

```

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [May 29, 2019, 6:00pm UTC](https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738/15 "2019-05-29T18:00:28Z")

</div>

No. Passing a pointer simply means the caller is now responsible to make sure the julia object doesn’t get free’d, there’s absolutely no way you can cheat on this. If you pass a pointer that come from a string then it’s wrong.

---

<div class="post-metadata">

**Author:** ![Ronis\_BR](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ronis_br/32/50999_2.png) [@Ronis\_BR](https://discourse.julialang.org/u/Ronis_BR)\
**Post date:** [May 29, 2019, 8:30pm UTC](https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738/16 "2019-05-29T20:30:08Z")

</div>

Now I see. That would answer tons of questions I had with random segmentation faults while creating [TextUserInterfaces.jl](https://github.com/ronisbr/TextUserInterfaces.jl). Some ncurses functions needs a null-terminated string to create menus (`char*`). That string must not be deallocated since it is not copied anywhere, only the pointer is stored.

In this case, what can I do?

---

<div class="post-metadata">

**Author:** ![Ronis\_BR](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ronis_br/32/50999_2.png) [@Ronis\_BR](https://discourse.julialang.org/u/Ronis_BR)\
**Post date:** [May 29, 2019, 8:49pm UTC](https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738/17 "2019-05-29T20:49:39Z")

</div>

Since I need to use a mutable type, then I came up with this solution, in which I convert the string to a const `UInt8` array. Of course, this will only work for ASCII characters:

```julia
const _vstr = UInt8[]

function strtok(str::Union{Nothing,String}, delim::String)
    GC.@preserve _vstr begin
        if str == nothing
            ptr = C_NULL
        else
            # Empty the current string vector.
            empty!(_vstr)

            # Create the new string vector.
            for c in str
                push!(_vstr, UInt8(c))
            end

            # We need a null-terminated string.
            push!(_vstr, '\0')

            # Get the pointer.
            ptr = convert(Ptr{Cvoid}, pointer(_vstr))
        end

        ptr_tok = ccall((:strtok, "libc"), Cstring, (Ptr{Cvoid}, Cstring), ptr, delim)
        tok = ptr_tok != C_NULL ? unsafe_string(ptr_tok) : ""

        return tok
    end
end

p = strtok("- This, a sample string.", " ,.-")

while(p ≠ "")
	global p
	println("Token : $p")
	p = strtok(nothing, " ,.-")
end

```

In this case, is everything right? (I know it is ugly, but is it at least right 😃)

---

<div class="post-metadata">

**Author:** ![yuyichao](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yuyichao/32/20_2.png) [@yuyichao](https://discourse.julialang.org/u/yuyichao)\
**Post date:** [May 29, 2019, 9:01pm UTC](https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738/18 "2019-05-29T21:01:52Z")

</div>

Why do you need \_vstr as global?

---

<div class="post-metadata">

**Author:** ![Ronis\_BR](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ronis_br/32/50999_2.png) [@Ronis\_BR](https://discourse.julialang.org/u/Ronis_BR)\
**Post date:** [May 29, 2019, 9:09pm UTC](https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738/19 "2019-05-29T21:09:08Z")

</div>

It was the very first thing that came up to my mind. I cannot declare it local, on the function, because it will get deallocated right? AFAIK, `strtok` stores the pointer and uses it if called with `NULL` as input.

---

<div class="post-metadata">

**Author:** ![Ronis\_BR](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ronis_br/32/50999_2.png) [@Ronis\_BR](https://discourse.julialang.org/u/Ronis_BR)\
**Post date:** [May 30, 2019, 12:32am UTC](https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738/20 "2019-05-30T00:32:14Z")

</div>

@yuyichao btw, I am wondering, if the C function does not change the string, only uses it, (ncurses does that) would then be wrong to pass the pointer to the string?

[Next page](https://discourse.julialang.org/t/calling-c-function-strtok-in-julia/24738.md?page=2)