SQLite.jl and LibPQ.jl packages don't work when wrapped in a package

General Usage
1

Hello!

I found that the SQLite.jl and LibPQ.jl packages don’t work when wrapped in a package (not a module but a package with manifest and project.toml).

has anyone experienced this?

module N

import LibPQ
import Tables

const CONNECTION_STRING = "dbname=postgres user=postgres password=postgres"
const CONNECTION = LibPQ.Connection(CONNECTION_STRING)

function select_global_vars()
    res = LibPQ.execute(CONNECTION, "select * from mytable")
    return Tables.columntable(res)
end

end # module N

module M

import SQLite
import DBInterface
import Tables

const conn = SQLite.DB("./mybase.sqlite")

function select_global_vars()
    return DBInterface.execute(conn, "select * from mytable") |> Tables.columntable
end

end # module M

   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _  |  |
  | | |_| | | | (_| |  |  Version 1.9.2 (2023-07-05)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.9) pkg> activate .
  Activating project at `C:\N`

(N) pkg> instantiate
┌ Warning: The active manifest file has dependencies that were resolved with a different julia version (1.8.5). Unexpected behavior may occur.
└ @ C:\N\Manifest.toml:0
Precompiling project...
  1 dependency successfully precompiled in 3 seconds. 40 already precompiled.

(N) pkg> precompile

julia> using N

julia> N.select_global_vars()
[error | LibPQ]: UnknownError:
ERROR: UnknownError:
Stacktrace:
 [1] error(logger::Memento.Logger, exc::LibPQ.Errors.PQResultError{LibPQ.Errors.CUN, LibPQ.Errors.EUNOWN})
   @ Memento C:\Users\admin\.julia\packages\Memento\xnHxE\src\loggers.jl:463
 [2] handle_result(jl_result::LibPQ.Result{false}; throw_error::Bool)
   @ LibPQ C:\Users\admin\.julia\packages\LibPQ\CRtC5\src\results.jl:238
 [3] handle_result
   @ C:\Users\admin\.julia\packages\LibPQ\CRtC5\src\results.jl:230 [inlined]
 [4] _multi_execute(jl_conn::LibPQ.Connection, query::String; throw_error::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ LibPQ C:\Users\admin\.julia\packages\LibPQ\CRtC5\src\results.jl:305
 [5] _multi_execute
   @ C:\Users\admin\.julia\packages\LibPQ\CRtC5\src\results.jl:298 [inlined]
 [6] #execute#51
   @ C:\Users\admin\.julia\packages\LibPQ\CRtC5\src\results.jl:294 [inlined]
 [7] execute
   @ C:\Users\admin\.julia\packages\LibPQ\CRtC5\src\results.jl:290 [inlined]
 [8] select_global_vars()
   @ N C:\N\src\N.jl:10
 [9] top-level scope
   @ REPL[5]:1

julia>

   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _  |  |
  | | |_| | | | (_| |  |  Version 1.9.2 (2023-07-05)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.9) pkg> activate .
  Activating project at `C:\M`

(M) pkg> instantiate

(M) pkg> precompile
Precompiling project...
  1 dependency successfully precompiled in 1 seconds. 17 already precompiled.

julia> using M

julia> M.select_global_vars()
ERROR: SQLite.SQLiteException("out of memory")
Stacktrace:
  [1] sqliteerror(args::SQLite.DB)
    @ SQLite C:\Users\admin\.julia\packages\SQLite\aeqsS\src\SQLite.jl:34
  [2] macro expansion
    @ C:\Users\admin\.julia\packages\SQLite\aeqsS\src\base.jl:10 [inlined]
  [3] prepare_stmt_wrapper
    @ C:\Users\admin\.julia\packages\SQLite\aeqsS\src\SQLite.jl:109 [inlined]
  [4] SQLite.Stmt(db::SQLite.DB, sql::String; register::Bool)
    @ SQLite C:\Users\admin\.julia\packages\SQLite\aeqsS\src\SQLite.jl:146
  [5] Stmt
    @ C:\Users\admin\.julia\packages\SQLite\aeqsS\src\SQLite.jl:145 [inlined]
  [6] prepare
    @ C:\Users\admin\.julia\packages\SQLite\aeqsS\src\SQLite.jl:180 [inlined]
  [7] execute
    @ C:\Users\admin\.julia\packages\DBInterface\1Gmxx\src\DBInterface.jl:130 [inlined]
  [8] #execute#2
    @ C:\Users\admin\.julia\packages\DBInterface\1Gmxx\src\DBInterface.jl:152 [inlined]
  [9] execute
    @ C:\Users\admin\.julia\packages\DBInterface\1Gmxx\src\DBInterface.jl:152 [inlined]
 [10] select_global_vars()
    @ M C:\M\src\M.jl:10
 [11] top-level scope
    @ REPL[5]:1

julia>
2

This is because both these libraries wrap C libraries to provide the core functionality. Specifically, the Connection objects contain pointer “handles” to the C-library side of those objects and those aren’t valid to precompile in Julia modules. Instead, you’ll want to use the __init__ pattern to initialize these variables at runtime, something like:

module Foo

using SQLite

const CONN = Ref{SQLite.DB}()

function __init__()
    CONN[] = SQLite.DB(":memory:")
end

end

You can then pass CONN[] to various execute calls and it should work.

3 Likes
3

thank you