An experimental Julia library: copyout csv from postgresql using rust and Clang.jl

New to Julia
1

copyout csv from postgresql, using rust postgres/cbindgen and Clang.jl to interop rust and julia together, without jirs.

Any suggestions for sharing complex data structures from Rust to Julia? like:

#[repr(C)]
pub enum DTypes {
    I8 = 0,
    I32 = 1,
    I64 = 2,
    F32 = 3,
    F64 = 4,
    Str = 5,
}

#[repr(C)]
pub struct DFrame {
    pub width: u32,
    pub height: u32,
    pub fields: *mut *const u8,     // field names
    pub types: *mut DTypes,         // field types
    pub values: *mut *const c_void, // field values
}

Purpose:

Use the rust postgres library to generate a postgresql client that can be quickly imported, and provide a copyout method to quickly obtain csv results.

Prerequisites:

  1. Rust environment
  2. Julia environment with Clang.jl
  3. Postgresql database
  4. Modify the database link and test code in build.jl

Compile method:

using global julia env

  1. build step by step
include("build.jl")

build_rs()
build_jl()
test()
  1. or build once
julia build.jl

howto:

Refer to the test code in build.jl

changelog:

  • 2024-10-09 v0.1.1:

    • using Clang.jl to generate julia library
    • use mutable struct Copyout so that rust memory can be freed auto
    • refactor project name and interface
    • rewrite build.jl

  • 2024-10-03 v0.1.0: initial version
    https://github.com/bluebug/libpq_rsjl

2
  • 2024-10-10 v0.2.0:
    • add pq_query_native(rs)/pq_query(jl) function to get dframe from postgresql
    • add pq_free_dframe(rs) function to release dframe, but not sure if the rust code releases the memory correctly💣
    • add test code for dframe in build.jl
    • call unsafe_string only when call copyout.body

help need:

how to view Cstring without copy? using unsafe_string causing copy.

unsafe_string(s:Cstring)
3

I think you can use VectorStrings.jl for that!

4

Thanks, according to your suggestion, I found the library StringViews.jl, and now I can view CString without copying.

using StringViews

Base.length(cs::Cstring) = ccall(:strlen, Cint, (Cstring,), cs)
CStringView(cs::Cstring) = StringView(unsafe_wrap(Array, Ptr{UInt8}(cs), length(cs)))
Base.write(io::IO, cs::Cstring) = write(io, CStringView(cs))
Base.print(io::IO, cs::Cstring) = (write(io, CStringView(cs)); nothing)
Base.show(io::IO, cs::Cstring) = (write(io, CStringView(cs)); nothing)

Base.length(cs::Ptr{UInt8}) = ccall(:strlen, Cint, (Ptr{UInt8},), cs)
CStringView(cs::Ptr{UInt8}) = StringView(unsafe_wrap(Array, cs, length(cs)))
Base.write(io::IO, cs::Ptr{UInt8}) = write(io, CStringView(cs))
Base.print(io::IO, cs::Ptr{UInt8}) = (write(io, CStringView(cs)); nothing)
Base.show(io::IO, cs::Ptr{UInt8}) = (write(io, CStringView(cs)); nothing)

export CStringView