Avoiding module __init__ being called multiple times

Hello, I have been porting a web service I have to Genie. I have found the default structure very helpful to organize my project (auto-loading of libs, controller default structure, etc).

I have the following problem:

  • a lib/LanguageUtils.jl module that uses PyCall to import a bunch of libraries (__init__). This adds a ~30s cost per instantiation of the module (using LanguageUtils) and gets autoloaded by Genie.
  • Multiple controllers using this module.
  • using seems to trigger another call to __init__, with the consequent delay. I can sidestep this by setting an ENV["LANGUAGE_INIT"]=true flag, but that will cause problems (objects not being loaded). So it seems like every time using is called for a local module, itā€™s quite expensive.

I always feel like separating code in local modules in Julia is sort of anti-pattern or makes things more difficult.

Am I doing something wrong or are there any patterns I should use/ know about?

Thanks in advance.

That sounds wrong.

In particular, if you define a function __init__() in a module, then Julia will call __init__() immediately after the module is loaded (e.g., by import , using , or require ) at runtime for the first time (i.e., __init__ is only called once, and only after all statements in the module have been executed).

There may be another issue with code you have that makes it look as though, at runtime, LanguageUtils.__init__ is called again after is first run.

"a lib/LanguageUtils.jl module that uses PyCall to import ā€¦ libraries (__init__). "
Are you saying: LanguageUtils.jl is a module that defines an __init__() function and this __init__() function imports libraries using PyCall?

I always feel like separating code in local modules in Julia
is sort of anti-pattern or makes things more difficult.

Using local modules in Julia to separate specifically purposed code is the best way to introduce a namespace to qualify the purpose your code fulfills.
Separation-of-concerns in software design is a worthwhile perspective.

Using a local module to make your work better apportioned now and easier when revisiting the work (then when it may appear somewhat less fresh) does just that.
Local modules are infrequently needed, though. Nonetheless, where appears __init__() there is a module to be initialized, and it may be a local module.

Are you importing from the local module correctly? Does the local module access both enclosing modules and also other unnested modules properly? Is your local module an inner (wrapped) module or is it an ā€œalongsideā€ module? Should it be?
https://docs.julialang.org/en/v1/manual/modules/#Submodules-and-relative-paths

Are you saying: LanguageUtils.jl is a module that defines an __init__() function and this __init__() function imports libraries using PyCall?

Yes.

Using local modules in Julia to separate specifically purposed code is the best way to introduce a namespace to qualify the purpose your code fulfills

Absolutely :slight_smile: I donā€™t doubt the idea of code separation in general, just that I was finding it somehow impractical in this case.

Are you importing from the local module correctly? Does the local module access both enclosing modules and also other unnested modules properly? Is your local module an inner (wrapped) module or is it an ā€œalongsideā€ module? Should it be?

I am relying on Genieā€™s auto-loading feature, which ā€œrecursively adds all the folders to the LOAD_PATHā€.

I could try doing using ..LanguageUtils and making sure the module hierarchy is correctly defined, but then I guess it kind of defeats the purpose of the auto-loading. But maybe thereā€™s a good reason not to do it on the first place.

Does the local module access both enclosing modules and also other unnested modules properly?
I think all modules are self-contained, with the exception of LanguageUtils being used on several places. Thereā€™s no deeper nesting. I agree that this behaviour does not seem normal.

Perhaps itā€™s easily sidestepped, though. Thanks for the hints, Iā€™ll see if I find any errors and will try to call it using a relative path in this case and come nack with the results.

Iā€™ll share the code I use for lazy loading of Python. Feel free to borrow from it.

using PyCall
################################################################################
"""
LazyModule is PyNULL until one of its fields is accessed. The PyObject will always be loaded after MyPython module has been completely initialized. So it's safe to construct them outside of module's __init__()
"""
struct LazyModule
	mod::PyObject
	modpath::String
	LazyModule(modpath::String) = new(PyNULL(), modpath)
end
Base.getproperty(x::LazyModule, f::Symbol) = begin
	if f in fieldnames(LazyModule)
		return getfield(x, f)
	else    
		if ispynull(x.mod)
			copy!(x.mod, pyimport(x.modpath))
			@assert !ispynull(x.mod)
		end
		return PyCall.getproperty(x.mod, f)
	end
end
################################################################################
"""
LazyFunction's PyObject is PyNULL until the first time it's called. The function object will always be loaded after MyPython module has been completely initialized. So it's safe to construct them outside of module's __init__()
"""
struct LazyFunction
	func::PyObject
	func_factory::Function
	LazyFunction(func_factory::Function) = new(PyNULL(), func_factory)
end
(x::LazyFunction)(args...) = begin
	if ispynull(x.func)
		copy!(x.func, x.func_factory())
	end
	x.func(args...)
end
################################################################################
const fwgmod = LazyModule("deeplearn.tensorgraph2.fixedwidthgraph")
const wfiltermod = LazyModule("deeplearn.tensorgraph2.wfilter")
const flagfiltermod = LazyModule("deeplearn.tensorgraph2.flagfilter")
const fbamod = LazyModule("deeplearn.tensorgraph2.fixedbararray")
const np = LazyModule("numpy")
const tf = LazyModule("tensorflow")
const isinstance = LazyFunction() do
	pybuiltin("isinstance")
end
const convert_jl_array = LazyFunction() do
	py"""
	import numpy as np
	def convert_jl_array(jlarr):
		a = np.array(jlarr[:,:,:], copy=False, order='F')
		b = np.ascontiguousarray(a)
		return b.transpose([1,2,0])
	"""
	py"convert_jl_array"o
end

function __init__()
	py"""
	import sys
	PROJECT_ROOT = $(ENV["PYTHON_PROJECT"])
	if PROJECT_ROOT not in sys.path:
		sys.path.insert(0, PROJECT_ROOT)
	print(sys.path)    
	"""
end

1 Like