Porting Julia to other platforms. llvm libunwind

Internals & Design
1

Hi.

I’ve been working on Julia port for bitrig (I’ll start porting to openbsd as soon as they finish thread local storage support but it’s going to take some time).

As Bitrig(and openbsd) does not provide getcontext related system calls, libunwind library does not support bitrig. But bitrig’s base c++ standard library is libc++, built upon libc++abi linked against llvm’s libunwind implementation(they have minor local patch to compile libunwind on bitrig not yet ported to llvm libunwind upstream).

So I decided to use llvm libunwind for making stack traces and just realized julia also has it’s own enhanced version of libosxunwind(based on llvm libunwind).

If I port julia to bitrig with llvm libunwind, julia will have to support 3 different kinds of libunwind implementations and that seems unnecessary, considering llvm libunwind should work on every unix like os, as it’s implementation does not depend on system calls, thus os agnostic in most parts.

I began testing llvm libunwind as default libunwind implementation on my linux machines and for julia the only problem was that llvm libunwind does not have unw_get_proc_info_by_ip (used in debuginfo.cpp jl_getDylibFunctionInfo).

Considering the state of julia’s debugging facility, I don’t think we really need to use unw_get_proc_info_by_ip, and dropping it makes functioning julia executable for my local gentoo machine.

Before opening issue or pull requests I wanted to know what julia devs think of using llvm libunwind and what I might be missing here.

Thanks.

2

Yes we do. It makes a function julia executable in the same sense that a executable that provide no backtrace is also a function executable.

3

More importantly, the llvm libunwind is also missing the dynamic unwind info registration functions so we can’t use it as it. Having to support multiple libunwind is obviously not ideal although not too bad either given that they have similar api. IMO, the best way forward (and we are inevitably doing already) is to take over libunwind and merge all the patches that we need. Ideally we should also check with a few other projects that maintain their own libunwind forks.

5

Actually, simple patch to debuginfo.cpp gave me working stack trace with llvm’s libunwind on linux. Just directly manage eh frame like we already do in OSX. There’s no need to rely on libunwind’s old api.

6

The unw_get_proc_info_by_ip function is not for unwinding, but for getting base pointer to sysimg function.

7

Also the unw_get_proc_info_by_ip is not the hardest one. I have no doubt that we can use LLVM for it (and patch welcome for this part). The issue is for dynamic unwind info registration. AFAICT LLVM libunwind doesn’t have it and the patch you linked previously does not provide that either. You might not notice this on x86 because we turn off frame pointer omission and latest libunwind is reasonably good at doing frame pointer unwinding when there’s no unwind info.

8

Yes so I read the code.

#if defined(_OS_LINUX_)
        unw_proc_info_t pip;
        if (!saddr && unw_get_proc_info_by_ip(unw_local_addr_space,
                                              pointer, &pip, NULL) == 0)
            saddr = (void*)pip.start_ip;
#endif

(snip)

if (saddr) {
            for (size_t i = 0; i < sysimg_fvars_n; i++) {
                if (saddr == sysimg_fvars[i]) {
                    frame0->linfo = sysimg_fvars_linfo[i];
                    break;
                }
            }

my confusion was how is julia resolving sysimg symbols in OS X?

9

dladdr / dlinfo returns local symbol names on OSX but not linux.

10

thanks for pointing this out. I failed to find documentation but after reading implementation and experimenting that seems to be the case indeed. Unfortunately llvm libunwind resolves symbol with dladdr too so this might not be straightforward as I thought it would be. So without it, gdb and stack trace wouldn’t know what to do with sys.so’s local functions(but I guess it won’t be problem for most stack traces after inspecting readelf -Ws ./sys.so?).

on that point though if my understanding is correct, after generating code, julia emits eh information by (cgmemmgr.cpp’s )RTDyldMemoryManagerJL::finalizeMemory method and RTDyldMemoryManagerJL::registerEHFrames.

Both method uses (debuginfo.cpp’s) register_eh_frames, which works differently in os x and linux (ultimately it boils down to difference between llvm libunwind api and libunwind-dynamic api).

All I’m saying is I have working stack trace on linux with llvm libunwind api.

void _unw_add_dynamic_fde(unw_word_t fde) {
  CFI_Parser<LocalAddressSpace>::FDE_Info fdeInfo;
  CFI_Parser<LocalAddressSpace>::CIE_Info cieInfo;
  const char *message = CFI_Parser<LocalAddressSpace>::decodeFDE(
                           LocalAddressSpace::sThisAddressSpace,
                          (LocalAddressSpace::pint_t) fde, &fdeInfo, &cieInfo);
  if (message == NULL) {
    // dynamically registered FDEs don't have a mach_header group they are in.
    // Use fde as mh_group
    unw_word_t mh_group = fdeInfo.fdeStart;
    DwarfFDECache<LocalAddressSpace>::add((LocalAddressSpace::pint_t)mh_group,
                                          fdeInfo.pcStart, fdeInfo.pcEnd,
                                          fdeInfo.fdeStart);
  } else {
    _LIBUNWIND_DEBUG_LOG("_unw_add_dynamic_fde: bad fde: %s", message);
  }
}

back to the topic of porting it to bitrig, openbsd’s dl- related functions are just interface to ld.so, so I guess it won’t have problem “seeing” local symbols without the help of unw_get_proc_info_by_ip just like freebsd. I’ll stick with bitrig base’s libunwind implementation for now.

11

I was actually talking about using LLVM to do that, not llvm’s libunwind. We are already using LLVM to read debug info that that should provide enough information if the correct API is found.

Interesting, so they renamed the API, which is why the one we are using is commented out. Seems that the new API is better since there’s no need to parse FDE’s? It doesn’t seem to support ARM though. (Neither does upstream libunwind although we patched it to add support)

Looking more closely at llvm’s libunwind, it’s much more different from the original libunwind then what I assumed from the API change. It seems that they are focusing on being a C++ exception handling implementation which is probably what causing the difference.

  1. It seems to try only one unwinding method and it actually doesn’t use frame pointer at all.

    We can probably pass all the tests without this although it’ll be nice to have it for some obscure external C libraries

  2. I can’t find any info about how signal handler safe it is. There’s a NO_HEAP option but that seems to be incompatible with dynamic unwind info registration.

12

Hmm I’m reading LLVMSymbolizer class and LLVMObject library implementation. Hope I can find the exact API or API I can use as component to resolve local symbol from address.

Yes internal class CFI_Parser does all the work for us.

What makes you think so? If we stick to dwarf 2 eh of ARM not EHABI this should work regardless of processor type. that being said I just realized julia arm unwind uses exidx. Is there reason for it?

sjlj for 32bit arm. dwarf eh for everything else. EHABI support for arm if desired. It’s tied to how llvm handles exception. I don’t think EHABI section was meant to be dynamically generated tho.

I have no idea. so far all routines look like thread safe but there’s no documentation or comment hints about async signals. I have strong feeling that at least methods for unwinding local address space is signal safe tho.
I contacted llvm devs for confirmation. hope I get answers soon.

13

I find no way to convince any compiler (gcc/clang/llvmjit) to emit eh_frame. EHABI is also what c++ uses.

SJLJ is irrelevant here. We use SJLJ for all exceptions and use unwinding only for backtrace. This is why we want to try as hard as possible to unwind the stack even in the cases we cannot restore all registers.

Well, LLVM jit does that.

malloc and dl_iterate_phdr are not, which is why I mentioned the NO_HEAP option. We still have problem with original libunwind in the profiler on ARM for exactly this reason.

14

I guess my knowledge of arm architecture is seriously dated. I take it back.

I was talking about llvm libunwind’s implementation(as tied to llvm). not how Julia creates exception. If I remember correctly llvm backend uses llvm.eh.sljl intrinsics right?

are we talking about MCJit here? Now I understand what NotifyFunctionEmitted is doing for ARM target.

Then yes. but I’m pretty sure it should be simple to add support for arm. as we already have all the necessary information right after jitting.

15

No

MCJIT and OrcJIT.

Also note that other than the technical issue of making sure dynamic registration works and make sure we try multiple unwinding method instead of only one, it is also preferred to use the system provided libunwind on systems with a working package manager instead of always shipping our own. What libunwind version does openbsd ship with?

AFAICT all major linux distributions ships the original libunwind and since the llvm libunwind is not fully api and abi compatible this is unlikely to change. This is the main advantage of patching and maintaining the original libunwind and use it on linux.

16

whole point was that openbsd/bitrig does not ship with old libunwind. bitrig has llvm upstream’s libunwind and that’s all. openbsd doesn’t even have one.

I know llvm libunwind is not meant to be used this way and most people just need them for libcxxabi but as os x port of julia is already using it, I wanted to know devs’ opinion of using one on linux.

don’t worry though, I’m not going to make series of pull request to replace old libunwind with llvm libunwind on linux. I’ll keep experimenting to further my understanding of llvm(and libcxx)'s architecture but that’s all.

I’ll just submit patch to make julia compilable&usable under bitrig and if devs like it they can merge it.