Trying to develop an extension

feribg · November 30, 2023, 2:02am

I find the process of developing an extension and all the steps extremely puzzling.
So far i have

these files:
– Package.jk
– – Utils.jl

Package is as follows:

module Package

include("Utils.jl")
export Utils.MarsagliaRng, Utils.next_u32, Utils.rand

end

and Utils as follows:

then runtests.jl is

using TestItemRunner

@run_package_tests

and UtilsTests.jl:

using Test
using Rndmwlk
using Rndmwlk.Utils.MarsagliaRng
using Rndmwlk.Utils.next_u32
using Rndmwlk.Utils.rand

@testitem "MarsagliaRng_nextu32" begin
    # Test for specific seed and expected results
    rng = MarsagliaRng(UInt8.([77, 0, 0, 0]))  # Seed = 77
    @test next_u32(rng) == 863531084
    @test next_u32(rng) == 2082884757
    @test next_u32(rng) == 788188464
    @test next_u32(rng) == 951323774
    @test next_u32(rng) == 744852019

    # Reset seed and test again
    rng = MarsagliaRng(UInt8.([77, 0, 0, 0]))  # Seed = 77
    @test next_u32(rng) == 863531084
    @test next_u32(rng) == 2082884757
    @test next_u32(rng) == 788188464
    @test next_u32(rng) == 951323774
    @test next_u32(rng) == 744852019

    # Test for seed 1
    rng = MarsagliaRng(UInt8.([1, 0, 0, 0]))  # Seed = 1
    @test next_u32(rng) == 1304893148
    @test next_u32(rng) == 840426891
    @test next_u32(rng) == 2480088888
    @test next_u32(rng) == 2020221409
    @test next_u32(rng) == 2766137397
end

@tesitem "Uniform Distribution Random Number Generation Tests" begin
    #Uniform distribution with seed 33
    rng = MarsagliaRng(UInt8.([33, 0, 0, 0]))  # Seed = 33
    @test isapprox(rand(rng), 0.68160327; atol=1e-8)
    @test isapprox(rand(rng), 0.37011177; atol=1e-8)
    @test isapprox(rand(rng), 0.00732552; atol=1e-8)
    @test isapprox(rand(rng), 0.53339489; atol=1e-8)
    @test isapprox(rand(rng), 0.79831550; atol=1e-8)


    #Uniform distribution with seed 1
    rng = MarsagliaRng(UInt8.([1, 0, 0, 0]))  # Seed = 1
    @test isapprox(rand(rng), 0.30381911; atol=1e-8)
    @test isapprox(rand(rng), 0.19567713; atol=1e-8)
    @test isapprox(rand(rng), 0.57744069; atol=1e-8)
    @test isapprox(rand(rng), 0.47036945; atol=1e-8)
    @test isapprox(rand(rng), 0.64404155; atol=1e-8)


    #Repeating tests for seed 33 to ensure repeatability
    rng = MarsagliaRng(UInt8.([33, 0, 0, 0]))  # Seed = 33, repeat to ensure repeatability
    @test isapprox(rand(rng), 0.68160327; atol=1e-8)
    @test isapprox(rand(rng), 0.37011177; atol=1e-8)
    @test isapprox(rand(rng), 0.00732552; atol=1e-8)
    @test isapprox(rand(rng), 0.53339489; atol=1e-8)
    @test isapprox(rand(rng), 0.79831550; atol=1e-8)

end

resulting in the most cryptic error message ever:
ERROR: LoadError: syntax: "module" at Package.jl:1 expected "end", got "."

So far its been a few hours trying to understand what’s the differnece between actIVATE . and dev . how it all plays out with VsCode, VsCode’s test runner vs Julia native and i think i almost have it in some kind of working state but this error makes no sense. Original package name was rndmwlk so there might be some refs to it…

Also is there any way to get around having to export every single thing twice or some kind of public private macro? I dont see any value of not having the visibility defined where the symbol is defined but instead having to double export. FYI if i dont export from Package.jl using the FQN in the tests still results in missing symbol…

filchristou · November 30, 2023, 7:49am

Just some random thoughts here and there.
Firstly, the correct term for your situation is “subpackage” instead of extension or
weak dependencies.
Subpackages are not super popular and they could get pretty complex too, so I would suggest you to follow such an architecture only if you are sure about it. Usually flatting everything in a single package works better.

ERROR: LoadError: syntax: "module" at Package.jl:1 expected "end", got "."

Probably you got a syntax error and that there is not a module ... end pair.
I am not sure where the problem is, but I am confident if you investigate your code a bit you will track this done. Is more information in the error message as per the location ? I usually comment large chunks of code to debug such syntax errors.
Also, keep in mind that include doesn’t do any magic and just concatenates text files; so it’s possible that the error lies inside Utils.jl

If you want to use some dependencies like using Dates in inside a module Mod, the the using Dates must be inside, i.e. after module Mod.

what’s the differnece between actIVATE . and dev .

activate fires up up a local environment. The default environment is the global.
develop adds a local dependency to the already started environment.

Find more in the docs

julia> import Pkg
help?> Pkg.activate
...
help?> Pkg.develop
...

Also is there any way to get around having to export every single thing twice

Have a look at Reexport.jl

Finally, I would suggest to get your priorities straight. It seems that your Package doesn’t even precompile. So, before going on doing the tests, make sure that at least Package precompiles successfully (e.g. with Pkg.precompile).
Also I don’t know why you keep Rndmwlk around in the tests, if that was the old name to your package.

feribg · November 30, 2023, 4:34pm

We can ignore the package naming for a second, It’s originally called Rndmwlk.jl, but thats not the main issue.

For example if i open a new julia shell and i do include(“src/Utils.jl”) it works just fine. There’s no syntax warnings in that file (shared in the pastebin) above from the Julia language server.

Is there any better way to include files that actually runs compiler checks and its not just a text append, as well as how come language server and including the file in the repl work just fine like so:

julia> include("src/Utils.jl")
Main.Utils

julia> next
nextfloat  nextind    nextpow    nextprod
julia> next
nextfloat  nextind    nextpow    nextprod
julia> Utils.next_u32
next_u32 (generic function with 1 method)

julia>

but this fails with compiler error:

module Rndmwlk

include("Utils.jl")
export Utils.MarsagliaRng, Utils.next_u32, Utils.rand

end

Failed to precompile Rndmwlk [36f4a8c9-f51c-45d8-8f56-ad14a47f3803] to "/home/feras/.julia/compiled/v1.9/Rndmwlk/jl_1nVwUe".
ERROR: LoadError: syntax: "module" at /home/Rndmwlk/src/Rndmwlk.jl:1 expected "end", got "."
Stacktrace:
 [1] top-level scope

the fact that this is so cryptic and so trivial makes me wonder what happens if this is the standard way of organizing things and i now have 10 submodules and multiple files flying around, it will be a colossal mess if there’s a typo somewhere, i struggle to believe this is the default way and if it is i think its bad enough to make me less excited about giving Julia a spin frankly ? Hope its not, since the other bits are quite neat.

Utils.jl:

0 issues in that file as reported by JLS:

I also manually concatenated the files like so:

module Rndmwlk

using Dates
using Random

module Utils

# Marsaglia Random Number Generator struct equivalent
mutable struct MarsagliaRng
    q::Vector{UInt32}
    carry::UInt32
    mwc256_initialized::Bool
    mwc256_seed::Int32
    i::UInt8

    function MarsagliaRng(seed::Vector{UInt8})
        q = zeros(UInt32, 256)
        carry = 362436
        mwc256_initialized = false
        mwc256_seed = reinterpret(Int32, seed)[1]
        i = 255
        new(q, carry, mwc256_initialized, mwc256_seed, i)
    end
end

# Default constructor using system time for seeding
function MarsagliaRng()
    ts_nano = UInt8(nanosecond(now()))
    seed = [ts_nano, ts_nano >>> 8, ts_nano >>> 16, ts_nano >>> 24]
    MarsagliaRng(seed)
end

# Random number generator functions
function next_u32(rng::MarsagliaRng)::UInt32
    a = UInt64(809430660)

    if !rng.mwc256_initialized
        j = UInt32(rng.mwc256_seed)
        c1 = UInt32(69069)
        c2 = UInt32(12345)
        rng.mwc256_initialized = true
        for k in 1:256
            j = (c1 * j + c2) % UInt32
            rng.q[k] = j
        end
    end

    rng.i = (rng.i + 1) % 256
    t = a * UInt64(rng.q[rng.i+1]) + UInt64(rng.carry)
    rng.carry = (t >> 32)
    rng.q[rng.i+1] = t % UInt32

    return rng.q[rng.i+1]
end



function next_u64(rng::MarsagliaRng)::UInt64
    UInt64(next_u32(rng))
end

# Sample function for generating a Float64
function rand(rng::MarsagliaRng)::Float64
    mult = 1.0 / typemax(UInt32)
    mult * next_u32(rng)
end

export MarsagliaRg, next_u32, rand

end

export Utils.MarsagliaRng

end

and it seems the error comes from:
export Utils.MarsagliaRng which is valid syntax and the recommended way of doing it. The error message saying error on line 1 is weird, why ?

feribg · November 30, 2023, 4:54pm

Ok thanks ChatGPT:

Nested Modules: You have a nested module structure where Utils is a module inside the Rndmwlk module. This is perfectly fine in Julia, but it does require careful handling of scope and export statements.

Export Statement: The line export Utils.MarsagliaRng at the end of your outer module Rndmwlk is intended to export MarsagliaRng from the Utils module. However, Julia's module system doesn't directly support exporting a name from a nested module in this way. That's why you are getting the error.

Error Message: The error message syntax: "module" at ... expected "end", got "." is a bit misleading. It suggests a syntax error related to the module keyword, but the real issue is how the export statement is used.

To achieve your intended functionality (i.e., making MarsagliaRng accessible from outside the Rndmwlk module), you need to adjust your approach:

Option 1: Export within the Nested Module: Inside the Utils module, you can use export MarsagliaRng. This will make MarsagliaRng available when you use Utils elsewhere.

Option 2: Qualified Export: In the outer module Rndmwlk, instead of export Utils.MarsagliaRng, you can just use export Utils. This will make the Utils module available outside of Rndmwlk, and you can access MarsagliaRng via Utils.MarsagliaRng.

Option 3: Using Statement: Another approach is to use a using statement. Inside Rndmwlk, after defining Utils, you can add using .Utils: MarsagliaRng. This will bring MarsagliaRng into the scope of Rndmwlk, and then you can use export MarsagliaRng.

This is okay now the tests somewhat work but i have to do this every time:

using Rndmwlk.Utils

@testitem "MarsagliaRng_nextu32" begin
    # Test for specific seed and expected results
    rng = Utils.MarsagliaRng(UInt8.([77, 0, 0, 0]))  # Seed = 77

and i cant find a way to bring all of Utils in scope a la Python from Utils import *. the only solution I find is Option 3 recomended by ChatGPT but thats not great as that makes it available for all users of the package. I want to qualify it as Utils.func sometimes and sometimes using Utils.func

Is there a way to accomplish that ? For example the recommended:
using Rndmwlk.Utils: MarsagliaRng, next_u32, rand doesn’t work. Function not defined error…

mkitti · November 30, 2023, 6:03pm

I think you either want to just export Utils or from outside the package do using Package: Utils.

Export example

julia> module Package
           export Utils
           module Utils
               const tau = 2π
           end
       end
Main.Package

julia> using .Package

julia> Utils.tau
6.283185307179586

using Example

julia> module PackageB
           module UtilsB
               const tau = 2π
           end
       end
Main.PackageB

julia> using .PackageB

julia> UtilsB
ERROR: UndefVarError: `UtilsB` not defined

julia> using .PackageB: UtilsB

julia> UtilsB.tau
6.283185307179586

feribg · November 30, 2023, 6:55pm

My issue is mostly how to:

using Test
using Rndmwlk.Utils

@testitem "MarsagliaRng_nextu32" begin
    # Test for specific seed and expected results
    rng = Utils.MarsagliaRng(UInt8.([77, 0, 0, 0]))  # Seed = 77
    @test Utils.next_u32(rng) == 863531084
    @test Utils.next_u32(rng) == 2082884757

Remove the Utils.next_u32 and make it just next_u32 in this file ONLY, without having to do using .Utils.next_u32 in the root module and exporting it as next_u32 everywhere. So in a sense just a shorthand alias for Utils.* for this file only to avoid boilerplate and improve readibility.