Calling c function strtok in julia

This is an awful lot of trouble to go to call the strtok function. Is that just a proxy for something more complex? If you literally just need strtok then it’s not very hard to write a Julia version of the function that handles Unicode correctly and works on any string type.

There is, however, no reason why passing an array of bytes to C will only handle ASCII strings. UTF-8 is also represented by arrays of bytes and as long as the separators are ASCII, it will work just fine. If you need to handle Unicode, on the other hand, you’re better off implementing this in Julia.

Regarding the correct use of ccall, you do not need a global to prevent deallocation. Yes, a global will not be deallocated. But a local variable that is used after the ccall will also not be deallocated. Or if you wrap a local in GC.@preserve then it also will not be deallocated. You’re now doing all three which is significant overkill.

But again, relying on C for simple sting splitting is probably not the way to go.

3 Likes