Hello, I have been porting a web service I have to Genie. I have found the default structure very helpful to organize my project (auto-loading of libs, controller default structure, etc).
I have the following problem:
a lib/LanguageUtils.jl module that uses PyCall to import a bunch of libraries (__init__). This adds a ~30s cost per instantiation of the module (using LanguageUtils) and gets autoloaded by Genie.
Multiple controllers using this module.
using seems to trigger another call to __init__, with the consequent delay. I can sidestep this by setting an ENV["LANGUAGE_INIT"]=true flag, but that will cause problems (objects not being loaded). So it seems like every time using is called for a local module, itās quite expensive.
I always feel like separating code in local modules in Julia is sort of anti-pattern or makes things more difficult.
Am I doing something wrong or are there any patterns I should use/ know about?
In particular, if you define a function __init__() in a module, then Julia will call __init__() immediately after the module is loaded (e.g., by import , using , or require ) at runtime for the first time (i.e., __init__ is only called once, and only after all statements in the module have been executed).
There may be another issue with code you have that makes it look as though, at runtime, LanguageUtils.__init__ is called again after is first run.
"a lib/LanguageUtils.jl module that uses PyCall to import ā¦ libraries (__init__). "
Are you saying: LanguageUtils.jl is a module that defines an __init__() function and this __init__() function imports libraries using PyCall?
I always feel like separating code in local modules in Julia
is sort of anti-pattern or makes things more difficult.
Using local modules in Julia to separate specifically purposed code is the best way to introduce a namespace to qualify the purpose your code fulfills.
Separation-of-concerns in software design is a worthwhile perspective.
Using a local module to make your work better apportioned now and easier when revisiting the work (then when it may appear somewhat less fresh) does just that.
Local modules are infrequently needed, though. Nonetheless, where appears __init__() there is a module to be initialized, and it may be a local module.
Are you importing from the local module correctly? Does the local module access both enclosing modules and also other unnested modules properly? Is your local module an inner (wrapped) module or is it an āalongsideā module? Should it be? https://docs.julialang.org/en/v1/manual/modules/#Submodules-and-relative-paths
Are you saying: LanguageUtils.jl is a module that defines an __init__() function and this __init__() function imports libraries using PyCall?
Yes.
Using local modules in Julia to separate specifically purposed code is the best way to introduce a namespace to qualify the purpose your code fulfills
Absolutely I donāt doubt the idea of code separation in general, just that I was finding it somehow impractical in this case.
Are you importing from the local module correctly? Does the local module access both enclosing modules and also other unnested modules properly? Is your local module an inner (wrapped) module or is it an āalongsideā module? Should it be?
I am relying on Genieās auto-loading feature, which ārecursively adds all the folders to the LOAD_PATHā.
I could try doing using ..LanguageUtils and making sure the module hierarchy is correctly defined, but then I guess it kind of defeats the purpose of the auto-loading. But maybe thereās a good reason not to do it on the first place.
Does the local module access both enclosing modules and also other unnested modules properly?
I think all modules are self-contained, with the exception of LanguageUtils being used on several places. Thereās no deeper nesting. I agree that this behaviour does not seem normal.
Perhaps itās easily sidestepped, though. Thanks for the hints, Iāll see if I find any errors and will try to call it using a relative path in this case and come nack with the results.
Iāll share the code I use for lazy loading of Python. Feel free to borrow from it.
using PyCall
################################################################################
"""
LazyModule is PyNULL until one of its fields is accessed. The PyObject will always be loaded after MyPython module has been completely initialized. So it's safe to construct them outside of module's __init__()
"""
struct LazyModule
mod::PyObject
modpath::String
LazyModule(modpath::String) = new(PyNULL(), modpath)
end
Base.getproperty(x::LazyModule, f::Symbol) = begin
if f in fieldnames(LazyModule)
return getfield(x, f)
else
if ispynull(x.mod)
copy!(x.mod, pyimport(x.modpath))
@assert !ispynull(x.mod)
end
return PyCall.getproperty(x.mod, f)
end
end
################################################################################
"""
LazyFunction's PyObject is PyNULL until the first time it's called. The function object will always be loaded after MyPython module has been completely initialized. So it's safe to construct them outside of module's __init__()
"""
struct LazyFunction
func::PyObject
func_factory::Function
LazyFunction(func_factory::Function) = new(PyNULL(), func_factory)
end
(x::LazyFunction)(args...) = begin
if ispynull(x.func)
copy!(x.func, x.func_factory())
end
x.func(args...)
end
################################################################################
const fwgmod = LazyModule("deeplearn.tensorgraph2.fixedwidthgraph")
const wfiltermod = LazyModule("deeplearn.tensorgraph2.wfilter")
const flagfiltermod = LazyModule("deeplearn.tensorgraph2.flagfilter")
const fbamod = LazyModule("deeplearn.tensorgraph2.fixedbararray")
const np = LazyModule("numpy")
const tf = LazyModule("tensorflow")
const isinstance = LazyFunction() do
pybuiltin("isinstance")
end
const convert_jl_array = LazyFunction() do
py"""
import numpy as np
def convert_jl_array(jlarr):
a = np.array(jlarr[:,:,:], copy=False, order='F')
b = np.ascontiguousarray(a)
return b.transpose([1,2,0])
"""
py"convert_jl_array"o
end
function __init__()
py"""
import sys
PROJECT_ROOT = $(ENV["PYTHON_PROJECT"])
if PROJECT_ROOT not in sys.path:
sys.path.insert(0, PROJECT_ROOT)
print(sys.path)
"""
end
I donāt doubt the idea of code separation in general, just that I was finding it somehow impractical in this case.