How to import functions defined in a Julia script file without running the script itslef

lucatrv · October 21, 2017, 9:52pm

I would like to reopen this topic already discussed years ago at the following links:

https://stackoverflow.com/questions/14462557/within-a-julia-script-can-you-tell-whether-the-script-has-been-imported-or-exec

https://groups.google.com/forum/#!topic/julia-users/ufpi8tV7sk8/discussion

I am used to write Python code, and I find the if __name__ == '__main__' functionality very convenient in the Python open source environment, because it allows to reuse functions written by others (who maybe did not think they would have been useful outside their project) without modifying their code.

I am now dealing with a concrete case, which I submit here for advice. I am contributing to the static-julia project, whose main juliac.jl script calls the julia-config.jl script to define the proper gcc compilation flags. To do this, since julia-config.jl is thought to be directly executed (see function main()) and not to be included by other scripts, juliac.jl has to use the following awkward code to get the flags:

command = `$(Base.julia_cmd()) $(joinpath(dirname(JULIA_HOME), "share", "julia", "julia-config.jl"))`
cflags = Base.shell_split(readstring(`$command --cflags`))
ldflags = Base.shell_split(readstring(`$command --ldflags`))
ldlibs = Base.shell_split(readstring(`$command --ldlibs`))

This of course is very inefficient.

In my mind Python’s if __name__ == '__main__' functionality would be handy in this and similar cases. I would like to ask your opinion and advice on this.

Thanks

Tamas_Papp · October 22, 2017, 7:07am

Split the code into two files, main.jl:

include("reusable_code.jl")
do_stuff_based_on(ARGS)

and reusable_code.jl, which contains the part you want to reuse. Even better, you should wrap it in a module.

lucatrv · October 22, 2017, 1:53pm

The issue is that I am not the owner of the julia-config.jl script. To do that I would need to ask the maintainer of the project whom julia-config.jl belongs to (in this case Julia itself) to change his code for me. Apart from this specific case (which I do not know what the authors of the Julia would think about it), in general the owner of the project could refuse to change his code for me, at least not in time for my needs, or the project could even be unmaintained.

I think a function similar to include but which only includes functions and similar definitions without actually executing the script would be very useful (it could be named for instalnce includef).

yuyichao · October 22, 2017, 2:17pm

A function like that is impossible. There’s no difference between function definitions and other code. We don’t have a way similar to python for doing this because files are not tied to a module and so executing a script is nothing more than just include it. You don’t have to split the file to make part of it reusable though, you can just have a check for a certain global variable. In either case (python included) the script must be written to support this.

Tamas_Papp · October 22, 2017, 2:33pm

Are we talking about this file? If you find functionality there that is useful in general, you should submit a PR that factors it out.

I fail to understand the issue. If you have the source, you do what you like with it, as quickly as you prefer.

This is not necessarily well-defined in general. Consider

const HASH_CUSTOMTYPE = 0xfd53ac3dfed72cdf
struct CustomType{T}
    value::T 
end
Base.hash(A::CustomType, h::UInt) = hash(A.value, hash(HASH_CUSTOMTYPE, h))

Is the first line part of the “script” or the “functions”? It is not a function, but necessary for the function to work. You will find similar examples with @eval etc all over the Julia codebase.

lucatrv · October 22, 2017, 3:38pm

Yes, but this would work only if the author of julia-config.jl accepts my PR to make my code work, but maybe he would not like to make his code more complex just to make mine work or simply does not care (of course I am taking julia-config.jl just as a specific example but for a more general talk).

The drawback of doing this is that I would duplicate existing code in my project (every time a new version of julia-config.jl is released I would need to copy and paste the relative code into my project).

I agree to some extent, that is why I wrote “functions and similar definitions” (thinking in general about structures, constants, etc). I opened this topic to brainstorm on this, but I am still not convinced that there cannot be better solutions.

For the sake of discussion, what about an includeonly function which allows to specify what exactly to include among available definitions?
For example:

includeonly("script.jl", "struct CustomType")

or

includeonly("script.jl", "CustomType::struct")

yuyichao · October 22, 2017, 4:19pm

That’s even more impossible to do…

Per · October 22, 2017, 5:18pm

Impossible in the general case, yes. Things might be defined through macros, etc, in ways that are impossible to detect without actually running the code. But it should not be to hard to parse a file and search for expression.head == :type && expression.args[2] == :CustomType, or something like that.

yuyichao · October 22, 2017, 5:38pm

If you really want that, you might as well copy the code since any small change in the original code can break it.

lucatrv · October 23, 2017, 10:10pm

My proposal for the includeonly function was to consider only certain kinds of definitions (e.g. functions, structures, macros, constants, etc), and only those that are at the first level of the script (same level as the main function). So if a function is defined within a macro, the macro itslef should be included. Same if a function is defined within another function, only the external function could be included.
I am not an expert so maybe I cannot see all possible difficulties and consequences, but I think such a functionality would be very handy.

lucatrv · October 26, 2017, 9:23pm

Ok, as I am probably the only one here thinking that an includeonly function would be useful, and I would like to better understand and conform to Julia’s best coding practices, I would like to ask some more information about the following suggestion:

Could you please provide an example or a documentation reference?
Thanks

yuyichao · October 26, 2017, 9:32pm

I’m pretty sure someone has mentioned using it in a script somewhere but I don’t remember what/where it was. But the idea is trivial

You just have

if !@isdefined(Do_not_run_the_script_please)
    run_the_script()
end

And the caller can just do

Do_not_run_the_script_please = nothing
include("the_script.jl")

It’s the same as what you do in python, just not a standard way due to not having file bound to module.

Tamas_Papp · October 27, 2017, 7:12am

Or something like

if haskey(ENV, "RUNSCRIPT")
    run_the_script()
end

which requires no action for not running the script, and just something like
ENV["RUNSCRIPT"] = true for running it.

lucatrv · October 27, 2017, 9:10pm

I think that the julia-config.jl script behaviour should not change when executed, but only when included by another script. Therefore I think this is the best suggestion:

So then what if an environment variable was defined (for instance INCLUDED) which is usually false, and is temporarily set true by the include function itself?
The standard way of doing this would then become:

if !INCLUDED
    run_the_script()
end

Even better maybe would be to have function isincluded().

lucatrv · October 29, 2017, 10:34pm

After lots of searching and reading, I think now I have a better understanding on how Julia code should be organized in modules to make it reusable.
I have to say that the “Modules” section of the official Julia documentation (which I had already read before starting this thread) did not help me very much on this, while I found more useful the “Modules and packages” section of the “Introducing Julia” Wikibook.
What however finally opened my eyes are the following two GitHub threads (I add the links here thinking they may be useful for others in future hitting this discussion):
https://github.com/JuliaLang/julia/issues/4600
https://github.com/JuliaLang/julia/issues/23619
IMHO the Julia documentation should eventually include more insight and examples on this topic, but I understand that some important details are still under discussion, so I guess we should wait for them to be completely defined and implemented.

Now I feel I am starting to think more Julia than Python, so if you agree in order to make the julia-config.jl functions reusable in juliac.jl I am going to submit a PR with the following edits:

Take all reusable functions out from the julia-config.jl script, and put them in a JuliaConfig.jl script.
In the JuliaConfig.jl script, wrap all the code within a module named JuliaConfig, and export all reusable functions (cflags, ldflags, ldlibs, allflags).
In the julia-config.jl script add the following lines:

include("JuliaConfig.jl")
using .JuliaConfig

Similarly, in the juliac.jl script add the following lines:

include(joinpath("path_to_JuliaConfig", "JuliaConfig.jl"))
using .JuliaConfig

If I understood correctly, this is the current syntax as of Julia v0.6 (please let me know otherwise).

In future, from the two GitHub threads linked above maybe the include statement will not be needed anymore, however in this case it is not clear to me how I would set the "path_to_JuliaConfig". I am going to ask this question there.

fredrikekre · October 30, 2017, 7:37am

Please submit a PR with your findings so that we can improve the manual!

lucatrv · October 30, 2017, 8:36pm

I’ll do when I have some spare time, but for now I am putting all the time I can spare into static-julia.