The format of the protocol I’m trying to develop for specifies that when sending a file descriptor, “The file descriptor is not stored in the message buffer, but in the ancillary data of the UNIX domain socket message (msg_control).”
Is there any way to access that from Julia or will I have to dig into C libraries?
This doesn’t make sense to me, as far as I’m aware a file descriptor is like a pointer to memory, just something that is local to you own process. To try to send it via a socket would be non-sensical, you will end up with a pointer that now points to empty space/garbage.
I believe there are special conditions where file descriptors can be shared, if you have a process and fork new processes from that with specific fork options. But in general it doesn’t sound like you would normally want to share the file descriptors especially with Julia, unless you are creating a custom app, which forks multiple processes which then all launch julia.
You would probably do better to just share the full path file name (if still on the same machine) and then open the file again and/or send the url and open the file remotely if using a proper network connection from another machine.
UNIX domain socket messages are unix specific and we be problematic for portability thus I assume just from that line of reasoning, it will not be possible to do from Julia which is cross platform.
Your question indicates a lot of missing context, can you explain what you actually want to do?
Sorry, I’ve apparently had a brain fart while posting this. Fixed the question to actually mean what I want.
I want to develop a Julia library for the Wayland protocol, whose wire format (including the message data specification) is written here: https://wayland.freedesktop.org/docs/html/ch04.html#sect-Protocol-Wire-Format
I know file descriptors are process-local, but apparently there’s a standard way to pass them between processes through Unix domain sockets. In the context of Wayland this is presumably used for shared memory buffers, which it uses for high-performance window compositing.
In addition to sending data, processes may send file descriptors across a Unix domain socket connection using the sendmsg() and recvmsg() system calls. This allows the sending processes to grant the receiving process access to a file descriptor for which the receiving process otherwise does not have access.
From: Unix domain socket - Wikipedia
As this happens entirely through system calls, I’m not sure you can get around calling the appropriate functions via ccall. As far as I’m aware, it’s not possible to do syscalls directly through julia, but I may be wrong about that.
Wayland is very Unix specific though, so relying on C for this is not unreasonable.
Yeah, I was just hoping that I can implement its core 100% in Julia, without messing around with C libraries and all the related “fun”.
Thanks for the link though, apparently my google-fu failed me this time. Will mark as solved if nobody comes to correct you.
One possible way of dealing with this would be to write a thin wrapper package handling the relevant syscalls and relaying that appropriately to julia callers - for example, when receiving the file descriptor I think you have to do some more syscalls to open the right file (maybe this can be done in julia?) and return an IO object instead.
A file descriptor is for all intents and purposes an open file. After some more research it seems ccall’ing into libc and using its interface is the only option, but after I successfully retrieve the fd I can just wrap it in any IO I want without any additional syscalls. Marking as solved.
This is incorrect, you can do syscalls perfectly fine in Julia: ccall(:getpid, Cint, ()). As you’ve stated more recently, this calls into libc to do what you need. I’ve been using this mechanism to fiddle with psuedoterminals and fork/exec mechanisms for a project I’m working on, and it works perfectly fine: https://gitlab.com/Samantha.ai/sipty.jl
Btw, @Deuxis, what exactly are you doing with Wayland? I’ve been considering writing a wrapper around wlroots at some point so I can play with writing my own compositors.
EDIT: Ok, I guess it’s technically correct that you can’t (easily) do syscalls in Julia, but if you’re bypassing your libc in any language, you’re very likely doing things very wrong 
One additional thing I’d just like to mention:
Just because Julia is cross-platform, does not mean that every package should be cross-platform too. Wayland of course only (currently?) works on Linux and some of the open source *BSDs, but that’s OK; I don’t think anyone who worked on the Wayland specifications was giving much though to Windows or OSX in the first place. Some things just work on a limited number of platforms, because that’s the intent. Microsoft is about as likely to implement the Linux kernel’s DRM (Direct Rendering Manager) as pigs are likely to fly tomorrow, so the likelihood of Wayland every working there is close to never.
Well, I guess you could in theory do it manually by writing assembly to the stack and just jumping there to execute… but I’m not even sure we can do that in an easier way than just calling the libc wrapper And since libc is going to be around on any system that’s running wayland, why not use it? It’s not like you’re adding another dependency.
EDIT: Not to mention, you’d have to take care of each architecture you’re supposed to be running on to write the correct assembly…
And you shouldn’t forget to mention all the wonderful things that your average libc does for you, like handling actual memory allocation (malloc is not an actual syscall, it’s brk plus a bunch of logic for pooling memory), caching all sorts of stuff, etc.
It’s not about bypassing libc, but about not wanting to directly play with C itself. And about what I’m doing… the ultimate goal is to write a Wayland version of Openbox, as I love its performance and grew very tired of all of Xorg’s issues but none of the current window managers tickle my fancy, but I’m doing all this at the most basic level to learn the protocol and polish up on Julia at the same time.
So far I’ve created a WaylandCore module with all the wire-level stuff (except, as you can see, fds) and a WaylandScanner module that is supposed to read protocol XML files and generate proper, all-julia, bindings. The core works just as well as I can test it without having any advanced functionality that uses it, and the scanner is on the verge of working. Feedback is very much appreciated.
PS: Yes, I know I can just use libwayland. Consider the wheel as in the process of being reinvented for educational purposes.
Well, if you stick only to ccalls, then you’re avoiding C almost entirely, however that can get cumbersome very quickly for complicated tasks. That’s part of why I recommended wlroots, as it’s a rapidly-maturing, heavily-used Wayland server library that could make your job much, much easier. However, if you want to do all the low-level Wayland stuff with ccalls (and without libwayland even), be my guest, I’d still be willing to contribute. It sounds like a good challenge and learning experience
Actually, Wayland isn’t tied to DRM at all, it’s just that it’s the best back-end for a compositor. You could implement a Wayland compositor using whatever Windows uses as its windowing API as back-end if you were feeling particularly masochistic.
Welp. I’m trying, and this is just as annoying as expected. The second argument to recvmsg is a pointer to msghdr struct. Said struct contains a void pointer to a buffer for ancillary data. Said ancillary data is a densely packed sequence of cmsghdr structs. Said cmsghdr structs are:
cmsg_len - the byte length of the control message, of type socklen_t, which I don’t even know what it maps to (opaque integral value of at least 32 bits, but who knows how long exactly?). Define Int and cross my fingers? It’s not even defined in socket.h!ints with metadata, those aren’t a problemcmsg_data - a (verbatim, not a pointer to) buffer of length cmsg_len containing god knows whatHow does one even define the correct data types that won’t make Julia segfault multiple times trying to send and read all this? How does one even read it if that succeeds? Can I pass IOBuffers to C?
Source: the man page and socket.h header itself.
I like, respect and occasionally use C, but this kind of binary typing “fun” is precisely what I wanted to avoid.
I did tell you that it wouldn’t be as easy as just using a pre-existing C library 
My approach for stuff like this is to first get a PoC working in C for the interface you’re trying to work with, use some printf() and sizeof() statements to figure out the sizes of things in C-land, and then do something similar in Julia to figure out which of the C-compatible Julia types are the right sizes.
If you want to see how wlroots does this stuff, here’s where they do sendmsg(): https://github.com/swaywm/wlroots/blob/master/backend/session/direct-ipc.c#L80
wlroots is MIT-licensed, so it’s safe to read and re-implement in your library.