Problem with referencing modules

jlhouchin · June 27, 2023, 9:36pm

Hello. Not quite new to Julia, been lurking for years and played quite a bit with porting code from another project.

I have code I have ported that I have written tests for and one of them is failing and I can not figure out the error. Probably simple for the regular users. But it has me baffled. I have searched this forum and have read the posts that I felt most closely resembled my problem. But I still haven’t figured it out.

I would dearly love help is getting past this problem as it completely stops me from using Julia. And I have avoided writing this post for far too long.

Smallest example:

I have two modules Candles and TimeFrames. Candles is using TimeFrames.
But when I have tests/candlestest.jl and I run the test I get this error over and over for any test which references TimeFrame.

Got exception outside of a @test
  MethodError: no method matching 
  candles(::Vector{Main.CandlesTest.Candles.Candle}, 
    ::Main.CandlesTest.TimeFrames.TimeFrame)

Now if I copy the test code and place it inside of the Candles module, it works, all tests pass.

Inside of candlestest.jl I have

include("../src/core/Candles.jl")
using .Candles
include("../src/core/TimeFrames.jl")
using .TimeFrames

Any help in understanding what I am doing wrong and what I need to do to correct it would be greatly appreciated.

Thanks.

Jimmie

stevengj · June 27, 2023, 9:58pm

Did you forget to export candles?

algunion · June 27, 2023, 9:58pm

Can you also show the content of the test file?

A few things from what you already added:

you don’t seem to export candles
if you don’t want to export it, then you either use Candles.candles or import it specifically.

jlhouchin · June 27, 2023, 10:26pm

Responding to multiple replies. Thanks.

From Candles

export Candle, CandleFull, CandleJpy, CandleFullJpy, hcavg, hlavg, lcavg, 
    movement, ohlcavg, hlcavg, show, candles, csv

candletest.js below is long but all is here.

module CandlesTest

using Test, Dates, Printf

include("../src/core/Prices.jl")
using .Prices
include("../src/core/Candles.jl")
using .Candles
include("../src/core/TimeFrames.jl")
using .TimeFrames

dt = now(UTC)
dt_s = string(dt)
price = 1.00000
price_s = "1.00000"
s1   = TimeFrames.S1
s5   = TimeFrames.S5
s10  = TimeFrames.S10
s15  = TimeFrames.S15
s30  = TimeFrames.S30
m1   = TimeFrames.M1
m2   = TimeFrames.M2
m4   = TimeFrames.M4
m5   = TimeFrames.M5
m10  = TimeFrames.M10
m15  = TimeFrames.M15
m30  = TimeFrames.M30
h1   = TimeFrames.H1
h2   = TimeFrames.H2
h3   = TimeFrames.H3
h4   = TimeFrames.H4
h6   = TimeFrames.H6
h8   = TimeFrames.H8
h12  = TimeFrames.H12
dt_candles_start = DateTime(2017, 1, 1, 21,  0) 

@testset "Testing All Candles" begin
    @testset "Testing Candle" begin
        candle = Candle(dt, price, price, price, price)
        candle_s = Candle(dt_s, price_s, price_s, price_s, price_s)
        candle_jpy = CandleJpy(dt,100.000,100.000,100.000,100.000)

        @test candle.datetime == dt
        @test candle.open == price
        @test candle.high == price
        @test candle.low == price
        @test candle.close == price

        @test candle_s.datetime == dt
        @test candle_s.open == price
        @test candle_s.high == price
        @test candle_s.low == price
        @test candle_s.close == price

        @test hcavg(candle) == 1.0
        @test hlavg(candle) == 1.0
        @test lcavg(candle) == 1.0
        @test movement(candle) == 0.0
        @test ohlcavg(candle) == 1.0
        @test hlcavg(candle) == 1.0
        @test string(candle) == "$(candle.datetime) o:1.00000 h:1.00000 l:1.00000 c:1.00000"
        @test csv(candle) == "$(candle.datetime),1.00000,1.00000,1.00000,1.00000"
        @test csv(candle_jpy) == "$(candle.datetime),100.000,100.000,100.000,100.000"
    end
    
    @testset "Testing CandleFull" begin
        candle = CandleFull(dt, price, price, price, price, price, price, price, price, 100)
        candle_s = CandleFull(dt_s, price_s, price_s, price_s, price_s, 
            price_s, price_s, price_s, price_s, "100")
        candle_jpy = CandleFullJpy(dt,100.000,100.000,100.000,100.000,
            100.000,100.000,100.000,100.000, 100)

        @test candle.datetime == dt
        @test candle.bidopen == price
        @test candle.bidhigh == price
        @test candle.bidlow == price
        @test candle.bidclose == price
        @test candle.askopen == price
        @test candle.askhigh == price
        @test candle.asklow == price
        @test candle.askclose == price
        @test candle.volume == 100

        @test candle_s.datetime == dt
        @test candle_s.bidopen == price
        @test candle_s.bidhigh == price
        @test candle_s.bidlow == price
        @test candle_s.bidclose == price
        @test candle_s.askopen == price
        @test candle_s.askhigh == price
        @test candle_s.asklow == price
        @test candle_s.askclose == price
        @test candle_s.volume == 100

        @test hcavg(candle) == 1.0
        @test hlavg(candle) == 1.0
        @test lcavg(candle) == 1.0
        @test movement(candle) == 0.0
        @test ohlcavg(candle) == 1.0
        @test hlcavg(candle) == 1.0
        @test string(candle) == "$(candle.datetime) bid o:1.00000 h:1.00000 l:1.00000 c:1.00000" *
            " ask o:1.00000 h:1.00000 l:1.00000 c:1.00000 v:100"
        @test csv(candle) == "$(candle.datetime),1.00000,1.00000,1.00000,1.00000," *
            "1.00000,1.00000,1.00000,1.00000,100"
        @test csv(candle_jpy) == "$(candle.datetime),100.000,100.000,100.000,100.000," *
            "100.000,100.000,100.000,100.000,100"      
    end

    @testset "Converting candles" begin
        # We have to create a lot of M1 candles in order to have 50 tests per timeframe.
        # I think 50 tests provides a sufficient data set to confirm problems and fix them.
        # If something were to require larger data set it is always available to adjust
        # the   first(m1_candles, ??) for any testset.
        vsize = 72000
        m1_candles = Vector{Candle}(undef, vsize)
        startt = now(UTC)
        # println("Creating $vsize m1 candles at: $startt")
        for i in 1:vsize
            ct = dt_candles_start + Minute(i)
            open = 1.0 + (i * 0.00001)
            high = open + 0.00001
            low = open - 0.00001
            close = high
            m1_candles[i] = Candle(Dates.format(ct, ISODateTimeFormat), @sprintf("%0.5f", open), 
                @sprintf("%0.5f", high), @sprintf("%0.5f", low), @sprintf("%0.5f", close))
        end
        endt = now(UTC)
        println("printing from m1_candles")
        println(m1_candles[1])
        println(m1_candles[1000])
        # println("Finished creating $vsize m1 candles at: $(endt)")
        # println("Total time creating $vsize m1 candles: $(endt - startt)")

        # In normal usage we test for high >= low as they can be equal
        # But due to specific creation of candles and their properties.
        # For the purpose of these test the  high > low   should always be true.

        @testset "Testing m1m5 candles conversion" begin
            tfduration = m5.duration
            tfcandles = Candles.candles(first(m1_candles, 250), m5)
            for index in 2:(length(tfcandles))
                candle = tfcandles[index]
                dt = (dt_candles_start + ((index - 1) * tfduration))
                @test candle.datetime == floor(dt, tfduration)
                @test candle.open == round((1.0 + ((index - 1) * tfduration.value * 0.00001)), digits=5)
                @test candle.low == round((candle.open - 0.00001), digits=5)
                @test candle.high == candle.close
                @test (candle.high > candle.low) == true
            end
        end

        @testset "Testing m1m10 candles conversion" begin
            tfduration = m10.duration
            tfcandles = Candles.candles(first(m1_candles, 500), m10)
            for index in 2:(length(tfcandles))
                candle = tfcandles[index]
                dt = (dt_candles_start + ((index - 1) * tfduration))
                @test candle.datetime == floor(dt, tfduration)
                @test candle.open == round((1.0 + ((index - 1) * tfduration.value * 0.00001)), digits=5)
                @test candle.low == round((candle.open - 0.00001), digits=5)
                @test candle.high == candle.close
                @test (candle.high > candle.low) == true
            end
        end

        @testset "Testing m1m15 candles conversion" begin
            tfduration = m15.duration
            tfcandles = Candles.candles(first(m1_candles, 750), m15)
            for index in 2:(length(tfcandles))
                candle = tfcandles[index]
                dt = (dt_candles_start + ((index - 1) * tfduration))
                @test candle.datetime == floor(dt, tfduration)
                @test candle.open == round((1.0 + ((index - 1) * tfduration.value * 0.00001)), digits=5)
                @test candle.low == round((candle.open - 0.00001), digits=5)
                @test candle.high == candle.close
                @test (candle.high > candle.low) == true
            end
        end

        @testset "Testing m1m30 candles conversion" begin
            tfduration = m30.duration
            tfcandles = Candles.candles(first(m1_candles, 1500), m30)
            for index in 2:(length(tfcandles))
                candle = tfcandles[index]
                dt = (dt_candles_start + ((index - 1) * tfduration))
                @test candle.datetime == floor(dt, tfduration)
                @test candle.open == round((1.0 + ((index - 1) * tfduration.value * 0.00001)), digits=5)
                @test candle.low == round((candle.open - 0.00001), digits=5)
                @test candle.high == candle.close
                @test (candle.high > candle.low) == true
            end
        end

        @testset "Testing m1h1 candles conversion" begin
            tfduration = h1.duration
            tfcandles = Candles.candles(first(m1_candles, 3000), h1)
            for index in 2:(length(tfcandles))
                candle = tfcandles[index]
                dt = (dt_candles_start + ((index - 1) * tfduration))
                @test candle.datetime == floor(dt, tfduration)
                minutes = (index - 1) * Minute(tfduration).value
                @test candle.open == round((1.0 + (minutes * 0.00001)), digits=5)
                @test candle.low == round((candle.open - 0.00001), digits=5)
                @test candle.high == candle.close
                @test (candle.high > candle.low) == true
            end
        end

        @testset "Testing m1h2 candles conversion" begin
            tfduration = h2.duration
            tfcandles = Candles.candles(first(m1_candles, 6000), h2)
            offset = Minute(tfduration).value - (tfcandles[2].open * 100000 - 100000)
            for index in 2:(length(tfcandles))
                candle = tfcandles[index]
                dt = (dt_candles_start + ((index - 1) * tfduration))
                @test candle.datetime == floor(dt, tfduration)
                minutes = ((index-1) * Minute(tfduration).value) - offset
                @test candle.open == round((1.0 + (minutes * 0.00001)), digits=5)
                @test candle.low == round((candle.open - 0.00001), digits=5)
                @test candle.high == candle.close
                @test (candle.high > candle.low) == true
            end
        end

        @testset "Testing m1h3 candles conversion" begin
            tfduration = h3.duration
            tfcandles = Candles.candles(first(m1_candles, 9000), h3)
            offset = Minute(tfduration).value - (tfcandles[2].open * 100000 - 100000)
            for index in 2:(length(tfcandles))
                candle = tfcandles[index]
                dt = (dt_candles_start + ((index - 1) * tfduration))
                @test candle.datetime == floor(dt, tfduration)
                minutes = ((index-1) * Minute(tfduration).value) - offset
                @test candle.open == round((1.0 + (minutes * 0.00001)), digits=5)
                @test candle.low == round((candle.open - 0.00001), digits=5)
                @test candle.high == candle.close
                @test (candle.high > candle.low) == true
            end
        end

        @testset "Testing m1h4 candles conversion" begin
            tfduration = h4.duration
            tfcandles = Candles.candles(first(m1_candles, 12000), h4)
            offset = Minute(tfduration).value - (tfcandles[2].open * 100000 - 100000)
            for index in 2:(length(tfcandles))
                candle = tfcandles[index]
                dt = (dt_candles_start + ((index - 1) * tfduration))
                @test candle.datetime == floor(dt, tfduration)
                minutes = ((index-1) * Minute(tfduration).value) - offset
                @test candle.open == round((1.0 + (minutes * 0.00001)), digits=5)
                @test candle.low == round((candle.open - 0.00001), digits=5)
                @test candle.high == candle.close
                @test (candle.high > candle.low) == true
            end
        end

        @testset "Testing m1h6 candles conversion" begin
            tfduration = h6.duration
            tfcandles = Candles.candles(first(m1_candles, 18000), h6)
            offset = Minute(tfduration).value - (tfcandles[2].open * 100000 - 100000)
            for index in 2:(length(tfcandles))
                candle = tfcandles[index]
                dt = (dt_candles_start + ((index - 1) * tfduration))
                @test candle.datetime == floor(dt, tfduration)
                minutes = ((index-1) * Minute(tfduration).value) - offset
                @test candle.open == round((1.0 + (minutes * 0.00001)), digits=5)
                @test candle.low == round((candle.open - 0.00001), digits=5)
                @test candle.high == candle.close
                @test (candle.high > candle.low) == true
            end
        end

        @testset "Testing m1h12 candles conversion" begin
            tfduration = h12.duration
            tfcandles = Candles.candles(first(m1_candles, 36000), h12)
            offset = Minute(tfduration).value - (tfcandles[2].open * 100000 - 100000)
            for index in 2:(length(tfcandles))
                candle = tfcandles[index]
                dt = (dt_candles_start + ((index - 1) * tfduration))
                @test candle.datetime == floor(dt, tfduration)
                minutes = ((index-1) * Minute(tfduration).value) - offset
                @test candle.open == round((1.0 + (minutes * 0.00001)), digits=5)
                @test candle.low == round((candle.open - 0.00001), digits=5)
                @test candle.high == candle.close
                @test (candle.high > candle.low) == true
            end
        end

        @testset "Testing m1D candles conversion" begin
            tfduration = TimeFrames.D.duration
            tfcandles = Candles.candles(first(m1_candles, 72000), TimeFrames.D)
            offset = Minute(tfduration).value - (tfcandles[2].open * 100000 - 100000)
            for index in 2:(length(tfcandles))
                candle = tfcandles[index]
                dt = (dt_candles_start + ((index - 1) * tfduration))
                @test candle.datetime == floor(dt, tfduration)
                minutes = ((index-1) * Minute(tfduration).value) - offset
                @test candle.open == round((1.0 + (minutes * 0.00001)), digits=5)
                @test candle.low == round((candle.open - 0.00001), digits=5)
                @test candle.high == candle.close
                @test (candle.high > candle.low) == true
            end
        end
    end

    @testset "Converting prices to candles" begin
        #size set so as to not create a candle based on a single price. 
        # That messes up testing.
        vsize = 3059 
        prices = Vector{Price}(undef, vsize)
        startt = now(UTC)
        # println("Creating $vsize prices at: $startt")
        for i in 1:vsize
            ct = dt_candles_start + Second(i)
            price = round((1.0 + (i * 0.00001)), digits=5)
            prices[i] = Price(ct, price)
        end
        endt = now(UTC)
        # println("Finished creating $vsize prices at: $endt")
        @testset "Testing prices_m1 candles conversion" begin
            # println("Total time creating m1 candles")
            tfduration = m1.duration
            tfcandles = Candles.candles(prices, m1)
            for index in 2:(length(tfcandles))
                candle = tfcandles[index]
                @test candle.datetime == dt_candles_start + ((index - 1) * tfduration)
                seconds = (index - 1) * Second(tfduration).value
                @test candle.open == round((1.0 + (seconds * 0.00001)), digits=5)
                @test candle.low == candle.open
                @test candle.high == candle.close
                @test (candle.high > candle.low) == true 
            end
        end
    end

    @testset "Converting CandlesFull" begin
        #size set so as to not create a candle based on a single price. 
        # That messes up testing.
        vsize = 3059 
        prices = Vector{PriceFull}(undef, vsize)
        startt = now(UTC)
        # println("Creating $vsize pricesfull at: $startt")
        for i in 1:vsize
            ct = dt_candles_start + Second(i)
            price = round((1.0 + (i * 0.00001)), digits=5)
            prices[i] = PriceFull(ct, price, price)
        end
        endt = now(UTC)
        # println("Finished creating $vsize pricesfull at: $endt")
        
        @testset "Testing prices_m1 candles conversion" begin
            # println("Total time creating candlesfull")
            tfduration = m1.duration
            tfcandles = Candles.candles(prices, m1)
            for index in 2:(length(tfcandles))
                candle = tfcandles[index]
                @test candle.datetime == dt_candles_start + ((index - 1) * tfduration)
                seconds = (index - 1) * Second(tfduration).value
                @test candle.bidopen == round((1.0 + (seconds * 0.00001)), digits=5)
                @test candle.bidlow == candle.bidopen
                @test candle.bidhigh == candle.bidclose
                @test (candle.bidhigh > candle.bidlow) == true
                @test candle.askopen == candle.bidopen
                @test candle.asklow == candle.askopen
                @test candle.askhigh == candle.askclose
                @test (candle.askhigh > candle.asklow) == true
            end
        end
    end
end

end #module CandlesTest

using CandlesTest
main()

Now the using CandlesTest and main() at the end are new, from desperately trying to make it work attempting things I have read here.

Again thanks for helping solve this.

Jimmie

jlhouchin · June 27, 2023, 10:44pm

And for a semblance of completeness.

module TimeFrames

using Dates

export TimeFrame

Thanks.

Jimmie

algunion · June 27, 2023, 10:55pm

Thanks for sharing more context.

Here are a few things:

Not sure what your workflow is, but usually, there is no need to create a dedicated module for tests.
Usually, you don’t need to include all your source files in the test file (this depends on how your project/package is configured - but for regular workflows, there is no need to include the files from src.
In your initial post, you referenced the error you got - the type Main.CandlesTest.Candles.Candle might be an indicator that there is actually an issue with the way you constructed your test file. I suggest removing the include statements from the test file and using the normal using statements (no need to prepend Candles with . (dot). And try just to run the tests again.

I hope this makes sense.

In summary: try to drop the module definition in tests and call your module in a standard way (e.g., using Candles).

The above assumes you followed a standard workflow when initiating your project/package.

Benny · June 27, 2023, 11:00pm

It’s a few lines down.

Just going on a limb here, when you say “Candles is using TimeFrames”, is it includeing TimeFrames.jl right before? That would cause two independent modules both named TimeFrames, one in Candles and one in CandlesTest. A Candles.candles method defined for Candles.TimeFrames.TimeFrame does not work on CandlesTest.TimeFrames.TimeFrame.

It would help if you post the using and export lines in Prices.jl, Candles.jl, and TimeFrames.jl

nsajko · June 27, 2023, 11:04pm

Just a thought: are you sure that all of your module are well formed syntactically? In particular could there be any missing end keywords that close expressions?

You can check by including the module file from the REPL, when successful the REPL should print the name of the module.

jlhouchin · June 28, 2023, 1:38pm

Thanks for the reply.

From TimeFrames.jl

module TimeFrames

using Dates

export TimeFrame

From Prices.jl

module Prices

export AbstractPrice, Price, PriceFull

using Dates

From Candles.jl

module Candles

using Base: String, Float64, Vector
using Printf
import Base: show, open, close, print, string
include("core_dates.jl")
include("TimeFrames.jl")
include("Prices.jl")
using .TimeFrames
using .Prices

export Candle, CandleFull, CandleJpy, CandleFullJpy, 
    hcavg, hlavg, lcavg, movement, 
    ohlcavg, hlcavg, show, candles, csv

From candles_test.jl

using Test, Dates, Printf

include("../src/core/Prices.jl")
using .Prices
include("../src/core/Candles.jl")
using .Candles
include("../src/core/TimeFrames.jl")
using .TimeFrames

That is exactly the problem I am having. I do not know how to resolve it. I am not used to a language that behaves this way. I don’t believe I would have this type of problem in Python. (It has been a while since use.) I have primarily been a Smalltalk user (Pharo) but am interested in Julia due to performance and also it be a dynamic, interactive language.

@nsajko requested that I run this in the REPL to check for syntactic errors.
This what I got.

I have previously been simply doing julia ./tests/candles_test.jl.

include("/home/jimmie/Dev/Julia/RhythmTrader/test/candles_test.jl")
WARNING: replacing module Prices.
WARNING: replacing module Candles.
WARNING: both Prices and Prices export "PriceFull"; uses of it in module Main must be qualified
WARNING: both Prices and Prices export "Price"; uses of it in module Main must be qualified
WARNING: both Candles and Candles export "Candle"; uses of it in module Main must be qualified
WARNING: both Candles and Candles export "csv"; uses of it in module Main must be qualified
WARNING: both Candles and Candles export "hlcavg"; uses of it in module Main must be qualified
WARNING: both Candles and Candles export "ohlcavg"; uses of it in module Main must be qualified
WARNING: both Candles and Candles export "movement"; uses of it in module Main must be qualified
WARNING: both Candles and Candles export "lcavg"; uses of it in module Main must be qualified
WARNING: both Candles and Candles export "hlavg"; uses of it in module Main must be qualified
WARNING: both Candles and Candles export "hcavg"; uses of it in module Main must be qualified
WARNING: both Candles and Candles export "CandleJpy"; uses of it in module Main must be qualified
WARNING: both Candles and Candles export "CandleFull"; uses of it in module Main must be qualified
WARNING: both Candles and Candles export "CandleFullJpy"; uses of it in module Main must be qualified
Testing Candle: Error During Test at /home/jimmie/Dev/Julia/RhythmTrader/test/candles_test.jl:37
  Got exception outside of a @test
  UndefVarError: `Candle` not defined

Any help in understanding how to structure my Julia code in a manner that resolves this problem and avoids it in the future is greatly appreciated.

Jimmie

nsajko · June 28, 2023, 2:44pm

One issue that is apparent from the given warnings is that you have multiple conflicting copies of identical modules. You could probably fix this by adopting the practice of not includeing files from within your modules. If you’re deveping a package, then only use include in the top-level file/module.

Keep in mind that include literally includes the contents of a file within another file.

Try includeing each of your modules individually, from a fresh REPL session.

EDIT: in case an example would be helpful, see my package here. The top-level file, with the top-level module, is the only place with any includes:

The sub-modules then use other modules like so:

jmair · June 28, 2023, 2:45pm

You shouldn’t include the modules in the test file. Just write:

using Test, Dates, Printf

using Prices
using Candles
using TimeFrames

Since these look like three custom projects, you may want to create a Project.toml in your test directory and dev all packages, e.g.

cd tests
julia --project

Then in REPL press ] for pkg mode:

pkg> dev path/to/Prices
pkg> dev path/to/Candles
pkg> dev path/to/TimeFrames

Along with adding other deps that you need. Then you should be able to run the tests as a script.

This isn’t really the standard way of testing, but it should (hopefully) work. See PkgTemplates.jl for creating a standard directory structure.

nsajko · June 28, 2023, 2:55pm

I don’t see why would it be good to create separate packages, that would complicate things with little benefit.

nsajko · June 28, 2023, 3:01pm

Another thing, perhaps unrelated, but it’s a recommendation for newbies: I recommend using Julia with the option --warn-overwrite=yes, at least ocassionally while developing. It makes Julia complain when a method gets overwritten, which can help in catching some programmer mistakes.

algunion · June 28, 2023, 5:00pm

I created a very simple example that might help here.

Please bear with me.

Let’s create A.jl with the following content:

module A

struct TA
    x::Int
end

export TA

end

Now, let’s create B.jl with the following content:

module B
    include("A.jl")
    using .A
    export makeTA, ta_use

    makeTA() = TA(1)

    ta_use(ta::TA) = ta.x
end

Let’s create a test.jl file with the following content:

using Test
include("A.jl")
include("B.jl")

using .A
using .B

@test ta_use(B.makeTA()) == 1 # passes
@test typeof(B.makeTA()) == typeof(B.TA(1)) # passes

# The following test that ends in error (not fail) is similar 
# to the one you present in your original post. 
# Also - see the corresponding test that passed above (first one).
# error outside of test: no method matching ta_use(::TA)
@test ta_use(A.TA(1)) == 1 # error - see comment above

@test typeof(B.makeTA()) == typeof(A.TA(1)) # fails
@test typeof(B.TA(1)) == typeof(A.TA(1)) # fails

In this context, you can see that B.TA differs from A.TA.

Also - defining a type instance from B and calling a function from A that uses (supposedly) the same type will end up in no method matching error (see @test ta_use(A.TA(1)) == 1).

Now look at your original post: it seems that you created instances of Main.CandlesTest.Candles.Candle using the types resulted after the include in the test file (in my example that is similar to include("A.jl")).

They are not the same type (see failing test: @test typeof(B.TA(1)) == typeof(A.TA(1))).

I think exactly this is what happens in your scenario - caused by the double include issue.

I suggest playing with this very simple example on your end. It might help you figure out and improve your understanding of how include/modules work.

P.S. Read my example as what not to do. To fix the issue, I should remove include("A.jl") from the tests altogether and use only instances of TA via the B.jl. (take a look at the passing tests).

Benny · June 28, 2023, 9:03pm

import/using in Julia actually behaves similarly to how Python’s import reuses code, it’s the include that’s tripping you up. If you’re used to IPython, its runfile does the same thing of evaluating code in a file. With Base Python, you can do the same with with open("TimeFrames.py") as f: exec(f.read()). Of course, these are not used as regularly as include is in Julia because Python turns every file into a module and the file directory into a module tree, so you’re forced to make a module the same size as a file. On the other hand, Julia makes a module tree with nested module statements, and include lets you write a global scope (such as a module) across multiple files. The concepts are orthogonal; a module block can include multiple files, and an included file can have multiple module statements in it.

The problem with multiple evaluations is that duplicated code is separate. For example, if you run the code x = 3 in two different .py files, those x’s will both exist independently, even if you import both of those files as modules. Let’s see a stripped down structure of your CandlesTest code just to see modules and imports, with marked import fixes and without includes for now:

Click to see modules structure

module CandlesTest
  using Test, Dates, Printf
  module Prices
    using Dates
  end
  using .Prices
  module TimeFrames
    using Dates
  end
  using .TimeFrames
  module Candles  ### move here where TimeFrames and Prices exist
    using Base: String, Float64, Vector
    using Printf
    import Base: show, open, close, print, string
    using ..TimeFrames  ### reuse CandlesTest.TimeFrames
    using ..Prices  ### reuse CandlesTest.Prices
  end
  using .Candles
end

The modules imported without the dots . are fine, they are packages searched from an independent place, and the code is not reevaluated. To reuse modules you evaluated only once, you just needed to adjust the order of evaluating your modules and the number of dots, which is like the number of modules levels you traverse. 1 dot using .Prices stays in the module CandlesTest, and 2 dots using ..Prices goes along Candles to CandlesTest.

Now let’s introduce some includes, the code itself may run as one nested unit but you certainly don’t want to lump all the actual code into one massive file.

Click to see files structure

## prices.jl
module Prices
  using Dates
end

## timeframes.jl
module TimeFrames
  using Dates
end

## candles.jl
module Candles
    using Base: String, Float64, Vector
    using Printf
    import Base: show, open, close, print, string
    using ..TimeFrames  ### reuse CandlesTest.TimeFrames
    using ..Prices  ### reuse CandlesTest.Prices
end

## candlestest.jl, this is the one you run ##
module CandlesTest
  using Test, Dates, Printf
  include("prices.jl")
  using .Prices
  include("timeframes.jl")
  using .TimeFrames
  include("candles.jl")   ### move here where TimeFrames and Prices exist
  using .Candles
end

That’ll take care of the repeated evaluation issue, but you probably noticed that CandlesTest is surrounding everything, and you might have intended the test module to be independent of the rest. It’s ok, you just have to refactor the CandlesTest module as a module on the same level as Prices, TimeFrames, and Candles, and adjust your using statements’ accordingly.

Click to see adjusted CandlesTest

## candlestest.jl ##
module CandlesTest
  using Test, Dates, Printf
  using ..Prices
  using ..TimeFrames
  using ..Candles
end

## main file you run ##
include("prices.jl")
using .Prices
include("timeframes.jl")
using .TimeFrames
include("candles.jl")   ### move here where TimeFrames and Prices exist
using .Candles
include("candlestest.jl")
using .CandlesTest

As you can see, all the includes of independent modules show up in the same file, which is good practice to keep track of them if you ever refactor your module structure. includes elsewhere are fine if those are guaranteed to run in unchanged locations, like if I’m splitting the code specific inside a module across smaller files (note that those files would not have a top-level module statement because I want every file’s code to reside in and share the one module’s scope).

jlhouchin · June 28, 2023, 10:04pm

I looked at your code and decided my package wasn’t a proper package. So I have read up on that (still reading) and have made proper package with PkgTemplates.

I have in my top level file, RhythmTrader.jl done the

include("core/Prices.jl")
include("core/Candles.jl")
include("core/TimeFrames.jl")

And have now have a runtests.jl with:

using RhythmTrader
using Test

@time @testset "RhythmTrader.jl" begin
    # Write your tests here.
    include("./core_dates_test.jl")
    include("./timeframes_test.jl")
    include("./candles_test.jl")
    include("./instruments_test.jl")
end

Now when I attempt to run the tests I get:

Package RhythmTrader does not have Prices in its dependencies:
- You may have a partially installed environment. Try `Pkg.instantiate()`
  to ensure all packages in the environment are installed.
- Or, if you have RhythmTrader checked out for development and have
  added Prices as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.

My package is where Prices, Candles and TimeFrames are. I am not sure how to add the modules as dependencies for the project that is providing them.

Thanks for all your help.

Jimmie

mkitti · June 28, 2023, 10:13pm

I am unclear if you want Prices.jl to be an independent package or just a module within your package.

If you want Prices.jl to be an independent package, you need to modify your Project.toml. The easiest way to do that is to activate the RythmTrader.jl as the activate environment, and then add the other packages.

# cd("~/.julia/dev/RythmTrader")
cd("src/RythmTrader")
using Pkg
Pkg.activate(".")
Pkg.dev("/path/to/Prices.jl") # or Pkg.add

However, I suspect that you still want Prices.jl just to be a module. In this case, you probably have a using Prices somewhere in your code where you should probably have using .Prices.

Here’s a quick demonstation of reproducing your error.

PS C:\Users\mkitti> tree /F Demo
Folder PATH listing for volume Windows
C:\USERS\MKITTI\DEMO
│   Manifest.toml
│   Project.toml
│
└───src
    │   Demo.jl
    │
    └───core
            Prices.jl

PS C:\Users\mkitti> cat Demo\Project.toml
name = "Demo"
uuid = "378f13b5-161c-49b5-995a-c14b5ffcafb0"

PS C:\Users\mkitti> cat Demo\src\Demo.jl
module Demo
    include("core/Prices.jl")
    using Prices
end

PS C:\Users\mkitti> cat Demo\src\core\Prices.jl
module Prices
end

PS C:\Users\mkitti> julia --project=Demo -e "using Demo"
ERROR: LoadError: ArgumentError: Package Demo does not have Prices in its dependencies:
- You may have a partially installed environment. Try `Pkg.instantiate()`
  to ensure all packages in the environment are installed.
- Or, if you have Demo checked out for development and have
  added Prices as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with Demo

If I modify Demo.jl so that I use using .Prices then the issue goes away.

PS C:\Users\mkitti> cat Demo\src\Demo.jl
module Demo
    include("core/Prices.jl")
    using .Prices
end

PS C:\Users\mkitti> julia --project=Demo -e "using Demo"

PS C:\Users\mkitti>

mkitti · June 28, 2023, 10:40pm

If you did want Prices.jl to be an independent package, here the minimal example for that.

PS C:\Users\mkitti> tree /F Demo
Folder PATH listing for volume Windows
C:\USERS\MKITTI\DEMO
│   Manifest.toml
│
├───core
│   └───Prices
│       │
│       └───src
│               Prices.jl
│
└───src
        Demo.jl

PS C:\Users\mkitti> cat Demo\Project.toml
uuid = "378f13b5-161c-49b5-995a-c14b5ffcafb0"

Prices = "c1468d2f-5eac-4b3c-b570-69b1183839d0"
PS C:\Users\mkitti> cat Demo\Manifest.toml
# This file is machine-generated - editing it directly is not advised

julia_version = "1.9.1"
manifest_format = "2.0"
project_hash = "4297016e1d9f93efd317dfebe8f154301320d250"

[[deps.Prices]]
path = "core/Prices"
uuid = "c1468d2f-5eac-4b3c-b570-69b1183839d0"
version = "0.0.0"

PS C:\Users\mkitti> cat Demo\src\Demo.jl
module Demo
    using Prices
end

PS C:\Users\mkitti> cat Demo\core\Prices\Project.toml
name = "Prices"
uuid = "c1468d2f-5eac-4b3c-b570-69b1183839d0"

PS C:\Users\mkitti> cat Demo\core\Prices\src\Prices.jl
module Prices
end

Demo\Project.toml and Demo\Manifest.toml were modified by doing the following.

PS C:\Users\mkitti> julia --project=Demo -e 'using Pkg; Pkg.develop(path=""Demo/core/Prices"")'

Please excuse the weird PowerShell quoting. On a Linux system, that would be julia --project=Demo -e "Pkg.develop(path=\"Demo/core/Prices\")"

nsajko · June 29, 2023, 5:26am

As @mkitti explained, you probably do things like using Prices when you should do using .Prices or using ..Prices.

jlhouchin · July 6, 2023, 10:07pm

I want to thank everyone who helped inform my understanding of how Julia works and problems or possible problems in my code. After a busy week with life. I finally made the time to work on the code and below are the changes I have made. All currently written tests are running and passing.

I come from a pure object oriented background, Smalltalk/Pharo. And this is the beginning of a port of a large project. I wanted to understand Julia enough to get past this barrier before I contemplated porting the rest of the code. I know Julia’s performance is a big reason to make the move. I also expect there will be other benefits that I will encounter over time.

Now I have successfully crossed the barrier in understanding and now know how to properly structure my Julia code, at least better than before and it works. I am now encouraged and willing to invest time to properly learn Julia and port the rest of the code.

Now to provide the Solution to my problem. A lot of answers came from @nsajko. I spent time studying the repo you provided until I understood what you were doing different from what I was doing. Here goes my current code.

First and foremost I had to make my code a proper Package. I did PkgTemplates minimum. I fixed anything required to get pkg precompile to work.

I made my top level RhythmTrader.jl to have all the necessary includes.

Candles

import ..TimeFrames
using .TimeFrames
import ..Prices
using .Prices
import ..CoreDates
using .CoreDates

runtests.jl

import RhythmTrader
using Test

const RT = RhythmTrader

@time @testset "RhythmTrader.jl" begin
    # Write your tests here.
    include("./core_dates_test.jl")
    include("./instruments_test.jl")
    include("./timeframes_test.jl")
    include("./candles_test.jl")
end

And from there

#In all tests...

RT.FullyQualified.Names

And from there

julia --project=. test/runtests.jl 
skipping test of  sleeptonextminute.  Please run if implementation changes
Test Summary:   | Pass  Total   Time
RhythmTrader.jl | 3553   3553  10.1s
 10.196584 seconds (5.96 M allocations: 417.791 MiB, 1.62% gc time, 93.59% compilation time)

What I haven’t learned yet is why when I am in Julia and I am running like this… why I get this error.

julia 
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.9.1 (2023-06-07)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.9) pkg> activate RhythmTrader/
  Activating project at `~/Dev/Julia/RhythmTrader`

(RhythmTrader) pkg> test
┌ Warning: The active manifest file is an older format with no julia version entry. Dependencies may have been resolved with a different julia version.
└ @ ~/Dev/Julia/RhythmTrader/Manifest.toml:0
     Testing RhythmTrader
ERROR: expected the file `src/RhythmTrader.jl` to exist for package `RhythmTrader` at `/tmp/jl_dNnqF4`

But as I go throught the documentation and other learning material to learn how to properly program in Julia as opposed to programming in Julia from my previous experience…
I expect to understand Environments and everything else much better.

Thanks again to all that helped.

Jimmie