Endianness doesn't seem to be conserved in FORTRAN shared libraries?

General Usage
,
1

Hello! I have a question regarding calling a FORTRAN shared library that needs to read a binary file that is big-endian.

This is the file in question: https://github.com/wrf-model/WRF/blob/master/run/RRTMG_LW_DATA.

I can write some fortran to read the file:

content = """
real(kind=selected_real_kind(12)) :: fracrefao(16)

OPEN(10,FILE='RRTMG_LW_DATA',                  &
    FORM='UNFORMATTED',STATUS='OLD')

READ (10) fracrefao

WRITE (*, *) fracrefao
"""

and then put it in a program:

prog = "program p\n" * content * "\nend program p\n"

open("read_file.f90", "w") do f
    write(f, prog)
end

and compile and run it:

run(`gfortran -O2 -fconvert=big-endian read_file.f90 -o read_file.exe`)

run(`./read_file.exe`)

This all works fine, but only if I use -fconvert=big-endian to do big-endian IO. If I don’t use that flag, I get this error:

At line 5 of file read_file.f90 (unit = 10, file = 'RRTMG_LW_DATA')
Fortran runtime error: End of file

What I really want to do, however, is to call a fortran shared library from julia to do the reading:

mod = """module m
contains
    subroutine readtxt
""" * content * """
    end subroutine readtxt
end module m
"""

open("read_file.f90", "w") do f
    write(f, mod)
end

run(`gfortran -shared -O2 -fPIC  -fconvert=big-endian read_file.f90 -o read_file.so`)

However, when I call the library:

ccall((:__m_MOD_readtxt, "./read_file.so"), Cvoid, ())

I get the error:

At line 7 of file read_file.f90 (unit = 10, file = 'RRTMG_LW_DATA')
Fortran runtime error: End of file

This is the same error that I get above when I don’t use -fconvert=big-endian, so my suspicion is that somehow that flag is not being honored when I call it from Julia. Additionally, if I use a text file instead of a big-endian binary file, calling the shared library also works fine, which further supports the idea that the problem has something to do with binary endianness.

Does anyone know how I can fix this? Thanks!

-Chris

2

Update: I’ve figured out that I can add ,convert='big_endian' to the file OPEN statement in the FORTRAN, but if it’s at all possible I would still like to find a way that doesn’t require editing the original fortran code. Thanks!

3

Maybe this helps:

https://github.com/traktofon/FortranFiles.jl

4

Perhaps the functions ntoh and hton in Base can be used. ntoh stands for “network to host”. Network is big endian, and host is little endian for all modern CPUs (but not some old ones, and not guaranteed for the future).

5

Just keep in mind that ,convert='big_endian' is a GNU Fortran extension and may not be supported by other compilers. You may want to fence it with the GNU Fortran preprocessor directive __GFORTRAN__

6

Have you by any chance built the library first without the -fconvert=big-endian, tried to call subroutine readtxt, then rebuilt the library with the conversion flag and ccalled it again within the same Julia session? If so, then you likely have not loaded the rebuilt library at all. Instead the original library is used, even though it has been rewritten on disk.

When using ccall, the first time a library function is called the library is Libdl.dlopened and a handle to that function found and saved in the background. Using ccall this way means you cannot then Libdl.dlclose that library and use a new one in the same place.

To dynamically load and unload a library, allowing for recompilation within the same Julia session, you can use the Libdl.dlopen(::Function, library_path) form of dlopen to open and then close the library, using Libdl.dlsym to get the function handle.

See the section on this in the manual.

An example:

julia> # Compile a library to print '1' to stdout, to file `lib.so`

julia> using Libdl: dlopen, dlsym

julia> dlopen("lib.so") do lib
           func = dlsym(lib, :test)
           ccall(func, Cvoid, ())
       end
1

julia> # Recompile the library to print '2' to screen

julia> dlopen("lib.so") do lib
           func = dlsym(lib, :test)
           ccall(func, Cvoid, ())
       end
2

If you had simply done ccall without the dlopens, both calls would have written 1 to stdout.

Once you have this working you can change your code back to ccalling the function you need without dlcloseing the library handle.

2 Likes
7

Thanks! However, in this case the Fortran runtime error: End of file error causes the entire Julia session to crash every time, so I don’t think it’s an issue of not reloading the library, because I have to start a new session with every attempt.

8

Thanks for letting me know!

9

Thanks! I’m not quite ready to translate the whole thing to julia right now, but this may be useful in the future.

10

So, it turns out that for gfortran shared libraries the endianness is specified at runtime, rather than at compile time. Therefore, all that needs to be done is:

ENV["GFORTRAN_CONVERT_UNIT"] = "big_endian"

before ccall and then it works correctly.

Reference here.

Thanks!

2 Likes