Include many files and call the same function from each file in a loop but got wrong result

General Usage
1

Hi guys!

I have three .jl files, which contains a test() function like this:

1.jl:

function test()
Dict([("A", 1), ("B", 2)])
end

2.jl:

function test()
    Dict([("A", 1), ("B", 2), ("C", 3)])
end

3.jl:

function test()
Dict([("A", 1), ("B", 2)，("C", 3)，("D", 4)])
end

and I write a loop to read and call the test() function:

filelist = readdir(filepath)  
for idx in eachindex(filelist)  
@show include(joinpath(filepath,filelist[idx]))
 @show dic = test()
end

and the result is:

include(joinpath("readtest", filelist[idx])) = test
dic = test() = Dict("B" => 2, "A" => 1, "C" => 3, "D" => 4)
include(joinpath("readtest", filelist[idx])) = test
dic = test() = Dict("B" => 2, "A" => 1, "C" => 3, "D" => 4)
include(joinpath("readtest", filelist[idx])) = test
dic = test() = Dict("B" => 2, "A" => 1, "C" => 3, "D" => 4)

how can I solve this?

2

invokelast()

but why on the earth would you need this to begin with

1 Like
3

to create a pipeline for running codes…… all files in the same format but varied size of dictions.

Problem solved! Thanks!

filelist = readdir(filepath)  
    include_func = () -> include(joinpath("readtest", filelist[idx]))
    test_func = () -> test()
    @show Base.invokelatest(include_func)
    @show Base.invokelatest(test_func)
end
4

Is there a better way to pipeline test code？ I am really new to this… using Jenkins is too much to me

5

Plain scripts should be okay for the work, no?

# 1.jl
d = Dict([("A", 1), ("B", 2)])

# 2.jl
d = Dict([("A", 1), ("B", 2), ("C", 3)])

# 3.jl
d = Dict([("A", 1), ("B", 2)，("C", 3)，("D", 4)])

and if you want to do something with them:

# main.jl
for datafile in files
    include(datafile)
    # do something with d
end
6

The key point is calling the same function test() in each .jl file. The dict is just one simple case.

7

You should put your files inside separate modules, to sandbox them properly. That way, there is no contamiation between them, and the test function is not overwritten. You can either put the module declaration explicitly in your test files, or in the load code, e.g.:

sources = filter(endswith(".jl"), readdir("tests"; join = true))
for file in sources
    module_name = gensym("mod_$file")
    @eval module $module_name
        function test end # prevent defining test something else as a function
        include($file)
    end
    println(getfield(@__MODULE__, module_name).test())
end
4 Likes
8

You are right! The best choice is to sandbox them. But I didn’t know how to :frowning:

Still have a question, will this significantly increase memory usage?

9

Thanks again for your help. I think it’s time to ask a new question: How can I pipeline my test code without contamination between different code files?

10

I doubt there is significant difference in memory usage. Modules are really lightweight structures. Here’s a measurement, though, using memusage:

  1. Without modules:
Memory usage summary: heap total: 45728473, heap peak: 19364334, stack peak: 50144
         total calls   total memory   failed calls
 malloc|      73180       41433864              0
realloc|       2612         739666              0  (nomove:1326, dec:1, free:0)
 calloc|       1106        3554943              0
   free|      81653       26646704
  1. With modules:
Memory usage summary: heap total: 48196844, heap peak: 19427256, stack peak: 41488
         total calls   total memory   failed calls
 malloc|      76011       43900414              0
realloc|       2489         713492              0  (nomove:1286, dec:1, free:0)
 calloc|       1171        3582938              0
   free|      86405       29067198
11

I’m not sure that I understand the question. The code I wrote above puts each file in its own module, effectively preventing contamination. What exactly do you mean by pipelining your test code?

12

I should have explained the details. The questions I’m asking here are simplified. My actual code is much more complex and involves multiple module and function calls, some of which are even multi-threaded. Unfortunately, I am not familiar with sandbox and not sure which should be written inside the separate module.

I think I should raise a new question about it. or ask if there are any documents that can help me.

your code is really helpful but I need to understand and modify it further.