Multiple Julia processes while using Modules

roh_codeur
Sun, 23 Jan 2022 14:52:03 +0000
General Usage

Hi

Following my discussion from the below thread, I have been trying out 2 approaches to evaluate Julia project organisation approaches.

Best practise: organising code in Julia New to Julia

Hi A newbie here, from OOP background. I am looking to put together a project which will: Download data from Internet Do some manipulations on the downloaded data Do some analysis on the downloaded data Please note the downloader, analysis would be reused across other projects as well. I have been reading up Julia docs and looking at how other Julia projects have been implemented. In Java, I would have organised these into separate packages and class files. Options: Create Modules: Juli…

1. Creating files as modules and “using” them in other files

2. Creating them as individual files and “include” them in other files

module MyModule

using Dates
using DataFrames

function MyModuleMethod()
    println("some method")
end

export MyModuleMethod
end

using MyModule2

using MyModule

function MyModuleFunc2()
       println("MyModuleFunc2 called")
end

export MyModuleFunc2

end

module MyRunner

using MyModule
using MyModule2

function __init__()
    println("in init")
end

function main2()
    MyModuleMethod()
end

export main2

end

runner.jl

baseDir = @__DIR__
@info "Starting in $baseDir"
cd(baseDir)

push!(LOAD_PATH, pwd())

using MyRunner
println("starting")

main2()

println("done")

This approach seems considerably slower than approach 2. While looking into the reason, I noticed that it was spinning up multiple Julia processes, using up a lot of memory on my machine. the second approach remarkably well. Its faster and doesn’t spin up multiple processes.

My machine:
Apple MacBook Pro M1 Max
Julia: 1.7.1

I am going with approach (2) for now.

1. I would like to know where I went wrong

2. It seems that there is a memory overhead(along with compilation overhead with Modules). that’s a bit of shame because I am trying to port a massive project to Julia. While I do this, I am also learning Julia. Revise.jl is excellent because it recompiles changes to Modules, so, separate small modules seemed like a great idea. Now, with approach (2), I have a single module, which takes a long time to compile. I must point out that subsequent compiles are pretty fast

thanks

cheers
Roh

7 posts - 4 participants