Why is this dependency not precompiling?

New to Julia
1

Frequently I get messages saying that a dependency did not precompile. This has been going on for months. I did not mind because my code runs. Still, it would be good to understand what is wrong and fix the problem. Maybe somebody can help.

This is my Project.toml:

name = "BW"
uuid = "23dbc89a-8e17-4d23-a4a2-f1aee6f5fec8"
authors = ["Fernando Saldanha <fsaldanha@alum.mit.edu>"]
version = "0.1.0"

[deps]
BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
Debugger = "31a5f54b-26ea-5ae9-a837-f05ce5417438"
Distributions = "31c24e10-a181-5473-b8eb-7969acd0382f"
FillArrays = "1a297f60-69ca-5386-bcde-b61e274b549b"
Indicators = "70c4c096-89a6-5ec6-8236-da8aa3bd86fd"
Infiltrator = "5903a43b-9cc3-4c30-8d17-598619ec4e9b"
IterTools = "c8e1da08-722c-5040-9ed9-7db0dc04731e"
JuliaFormatter = "98e50ef6-434e-11e9-1051-2b60c6c9e899"
MicroCollections = "128add7d-3638-4c79-886c-908ea0c25c34"
OhMyREPL = "5fb14364-9ced-5910-84b2-373655c76a03"
PlotThemes = "ccf2f8ad-2431-5c83-bf29-c5338b663b6a"
PlotlyBase = "a03496cd-edff-5a9b-9e67-9cda94a718b5"
Plots = "91a5bcdd-55d7-5caf-9e0b-520d859cae80"
PrettyTables = "08abe8d2-0d0c-5749-adfa-8a2ac140af0d"
ProgressMeter = "92933f4c-e287-5a05-a399-4b506db050ca"
PyPlot = "d330b81b-6aea-500a-939a-2ce795aea3ee"
Revise = "295af30f-e4ad-537b-8983-00126c2a3abe"
StatsBase = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"
StatsPlots = "f3b207a7-027a-5e70-b257-86293d7955fd"
ThreadsX = "ac1d9e8a-700a-412c-b207-f0111f4b6c0d"

I get this message:

ERROR: The following 1 direct dependency 
failed to precompile:

BW [23dbc89a-8e17-4d23-a4a2-f1aee6f5fec8]
Failed to precompile BW [23dbc89a-8e17-4d23-a4a2-f1aee6f5fec8] to C:\Users\fsald\.julia\compiled\v1.7\BW\jl_1F73.tmp.      
┌ Warning: Package Base does not have Dates in its dependencies:
│ - If you have Base checked out for development and have
│   added Dates as a dependency but haven't updated your primary
│   environment's manifest file, try `Pkg.resolve()`.
│ - Otherwise you may need to report an issue with Base
â”” Loading Dates into Base from project dependency, future warnings for Base are suppressed.
ERROR: LoadError: UndefVarError: include 
not defined
Stacktrace:
 [1] top-level scope
   @ c:\Dropbox\Code\Julia\bootstrapwalkforward\BW\src\BW.jl:33
 [2] include
   @ .\Base.jl:418 [inlined]
 [3] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
   @ Base .\loading.jl:1318
 [4] top-level scope
   @ none:1
 [5] eval
   @ .\boot.jl:373 [inlined]
 [6] eval(x::Expr)
   @ Base.MainInclude .\client.jl:453    
 [7] top-level scope
   @ none:1
in expression starting at c:\Dropbox\Code\Julia\bootstrapwalkforward\BW\src\BW.jl:33
Stacktrace:
 [1] pkgerror(msg::String)
   @ Pkg.Types C:\Users\fsald\AppData\Local\Programs\Julia-1.7.0\share\julia\stdlib\v1.7\Pkg\src\Types.jl:68
 [2] precompile(ctx::Pkg.Types.Context; internal_call::Bool, strict::Bool, warn_loaded::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})   @ Pkg.API C:\Users\fsald\AppData\Local\Programs\Julia-1.7.0\share\julia\stdlib\v1.7\Pkg\src\API.jl:1362
 [3] precompile
   @ C:\Users\fsald\AppData\Local\Programs\Julia-1.7.0\share\julia\stdlib\v1.7\Pkg\src\API.jl:1013 [inlined]
 [4] #precompile#220
   @ C:\Users\fsald\AppData\Local\Programs\Julia-1.7.0\share\julia\stdlib\v1.7\Pkg\src\API.jl:1011 [inlined]
 [5] precompile()
   @ Pkg.API C:\Users\fsald\AppData\Local\Programs\Julia-1.7.0\share\julia\stdlib\v1.7\Pkg\src\API.jl:1011
 [6] top-level scope
   @ REPL[4]:1
2

What happens when you run PKG.resolve()? Your problem seems slightly more complicated but if this doesn’t fix it you could try deleting the Manifest and the [deps] part of the Project.toml and adding the packages you need again through Pkg.

3

Pkg.resolve() did not help. I deleted Manifest.toml and the deps section of Project.toml. Then I added DataFrames.jl and got thevsame type of message ai posted. BW.jl did not precompile.

4

What does your global environment look like? What if you try in an empty environment?

5

What is line 33 in BW.jl?

6

Line 33 in BW.jl:

include(“./Trading.jl”);

The file Trading.jl is very short:

$$$$$$$$$$$$$$$$$$$$$$ Trading $$$$$$$$$$$$$$$$$$$$$$$

println(“Loading Trading”)

function tr_sign(x::Vector{Float64})::Vector{Float64}

[sign(t) for t in x]

end

function tr_positive_sign(x::Vector{Float64})::Vector{Float64}

[max(sign(t), 0) for t in x]

end

function tr_negative_sign(x::Vector{Float64})::Vector{Float64}

[min(sign(t), 0) for t in x]

end

7

That’s very odd, as the error suggests your BW module is failing to do include("./Trading.jl") because include is not defined, but include should always be available from Base.

What happens if you change BW.jl by just copying the three function definitions directly into the file?

(Incidentally, those function definitions seem to be redundant, as all you’re doing is applying scalar sign and max/min functions to a vector of Floats, for which you should just use broadcasting - i.e. tr_sign(x) is the same as just doing sign.(x), and the other two functions are just max.(sign.(t), 0) and min.(sign.(t), 0), respectively.)

8

Beginner’s question: How do I find out how my global environment looks like?

In an empty environment the file BW.jl would probably not run, since it needs packages like DataFrames.jl, Distributions.jl and others.

Exploring a bit, I clicked on Julia env:BW at the bottom of the VSCode screen. A window with Select Environment popped open. I selected v1.7 C:\Users\fsald\.julia\environments\v1.7. Another window popped up asking if I wanted to change the environment. I clicked on Change Julia environment. Now at the bottom I see Julia env v1.7 instead of Julia env:BW. However, clicking on ] in the Terminal I get (BW) Pkg which leaves it unclear in which environment I am. Typing st it becomes clear I am still in the BW environment. So I am not sure how do I try it in an empty environment.

I exited VSCode and restarted it and came back to exactly the same status: ] shows Julia env:BW but at the bottom of the screen I see Julia env:v1.7

9

You can do ]activate without any arguments to activate the default environment.

10

I replaced the include with the code, as you suggested. The first error message was exactly as before:

pkg> add Distributions.jl
    Updating registry at `C:\Users\fsald\.julia\registries\General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
  No Changes to `C:\Dropbox\Code\Julia\bootstrapwalkforward\BW\Project.toml`
  No Changes to `C:\Dropbox\Code\Julia\bootstrapwalkforward\BW\Manifest.toml`
Precompiling project...
  âś— BW
  0 dependencies successfully precompiled in 22 seconds (210 already precompiled)       
  1 dependency errored. To see a full report either run `import Pkg; Pkg.precompile()` or load the package

Then I ran

import Pkg
Pkg.precompile()

and VSCode froze. I restarted VSCode and got this message:


 Session contents restored from 12/13/2021 at 7:36:07 AM 

ERROR: LoadError: IOError: connect: no such file or directory (ENOENT)
Stacktrace:
 [1] wait_connected(x::Base.PipeEndpoint)
   @ Sockets C:\Users\fsald\AppData\Local\Programs\Julia-1.7.0\share\julia\stdlib\v1.7\Sockets\src\Sockets.jl:532
 [2] connect
   @ C:\Users\fsald\AppData\Local\Programs\Julia-1.7.0\share\julia\stdlib\v1.7\Sockets\src\Sockets.jl:567 [inlined]
 [3] connect
   @ C:\Users\fsald\AppData\Local\Programs\Julia-1.7.0\share\julia\stdlib\v1.7\Sockets\src\PipeServer.jl:97 [inlined]
 [4] serve(args::String; is_dev::Bool, crashreporting_pipename::String)
   @ VSCodeServer c:\Users\fsald\.vscode\extensions\julialang.language-julia-1.5.6\scripts\packages\VSCodeServer\src\VSCodeServer.jl:100
 [5] top-level scope
   @ c:\Users\fsald\.vscode\extensions\julialang.language-julia-1.5.6\scripts\terminalserver\terminalserver.jl:45
in expression starting at c:\Users\fsald\.vscode\extensions\julialang.language-julia-1.5.6\scripts\terminalserver\terminalserver.jl:24
julia>

I will rewrite the tr_sign functions as you suggested. One of them is passed to a struct that is an argument of a function, so I like having nice names for them. Could pass a lambda as an argument instead but that would obscure the code.

In any case, I tried completely eliminating all three functions from the code and passing

x -> max.(sign.(t), 0)

to the struct. Restarted VSCode, tried again add Distributions.jl and got the same message.

Giving up on import Pkg I used the ] key and tried precompile. Got this message:

(BW) pkg> precompile
Precompiling project...
  âś— BW
  0 dependencies successfully precompiled in 18 seconds (210 already precompiled)

ERROR: The following 1 direct dependency failed to precompile:

BW [23dbc89a-8e17-4d23-a4a2-f1aee6f5fec8]

Failed to precompile BW [23dbc89a-8e17-4d23-a4a2-f1aee6f5fec8] to C:\Users\fsald\.julia\compiled\v1.7\BW\jl_3392.tmp.
┌ Warning: Package Base does not have Dates in its dependencies:
│ - If you have Base checked out for development and have
│   added Dates as a dependency but haven't updated your primary
│   environment's manifest file, try `Pkg.resolve()`.
│ - Otherwise you may need to report an issue with Base
â”” Loading Dates into Base from project dependency, future warnings for Base are suppressed.
ERROR: LoadError: UndefVarError: include not defined
Stacktrace:
 [1] top-level scope
   @ c:\Dropbox\Code\Julia\bootstrapwalkforward\BW\src\BW.jl:57
 [2] include
   @ .\Base.jl:418 [inlined]
 [3] include_package_for_output(pkg::Base.PkgId, input::String, depot_path::Vector{String}, dl_load_path::Vector{String}, load_path::Vector{String}, concrete_deps::Vector{Pair{Base.PkgId, UInt64}}, source::Nothing)
   @ Base .\loading.jl:1318
 [4] top-level scope
   @ none:1
 [5] eval
   @ .\boot.jl:373 [inlined]
 [6] eval(x::Expr)
   @ Base.MainInclude .\client.jl:453
 [7] top-level scope
   @ none:1
in expression starting at c:\Dropbox\Code\Julia\bootstrapwalkforward\BW\src\BW.jl:57

Now the problem is in line 57 of BW.jl, which is another include statement.

So it looks like the problem is really with include statements. I guess if I inlined all the included code and eliminated all include statements I could get BW.jl to precompile.

The warning message about package Base not having Dates among its dependencies is also mysterious.

Oddly, the code runs despite all of the above.

11

I just did that. Saw that there were two packages in the deps section of Project.toml. Deleted those two lines. Restarted VSCode. Did ]activate and then st. The project was empty. Then did add Distributions.jl. Precompilation was successful. But BW.jl, which was the problem, was not involved in any way, so maybe this is not a surprise.

12

This is really odd - it seems to me like you’re doing something to break base Julia, i.e. Base, as somehow include isn’t working and there’s also the warning about Base not having Dates in its dependencies.

What happens when you replace include with Base.MainInclude.include?

Also are you using the name Base for something in your code?

13

I would put all your code in a git repo and upload it to e.g. GitHub so that people can check it out properly.

1 Like
14

Base.include is working. If it were not my code would not run. Most of my code is in included files. BW.jl basically calls functions that are in included files. The program runs for several minutes (lots of bootstrapping), generates several charts and DataFrames, prints what I ask it to print on the screen, etc… The only problem is it tells me it does not precompile BW.jl. But that does not seem to have any effect on computations.

I replaced all include statements with Base.MainInclude.include and that seems to have solved the problem. After that I added packages that were already in the project and also added a new one (Weave.jl) that was not. In both cases there were no messages saying that precompilation failed.

Besides the new statements with Base.MainInclude.include I have one line that has Base in it:

param_tuples = collect(Base.product(values(parameter_lists)…))

Until recently that line had Iterators where Base is now. I discovered that Iterators.jl was deprecated and replaced the package. The product command does the same thing in both cases. I believe the precompilation problem existed before I replaced Iterators with Base.

In any case, many thanks for helping me solve the problem.

BTW, what is the problem with using just include instead of Base.MainInclude.include?

15

It shouldn’t be a problem, that’s why I asked whether you are doing anything unusual with Base - as Kristoffer said it’s hard to tell without seeing your whole code what is going wrong here.

(As an aside, Iterators was replaced by IterTools, so you might want to consider using that)

16

IterTools.product() did not work. Maybe it had a different syntax. After investigating a bit on Google I found Base.product() does the same thing as Iterations.product(), actually maybe better.