Could this be related to an internet connection with very high latency caused by an imposed indirection through the professional network associated to complex and costly security procedures ?
(I have witnessed very slow update on my wife’s professional laptop connected to internet through a professional network.)
I don’t know how I could determine this.
I have tried the timings on my home computer, though. This is a v old homebrew based on an i7-2600k SandyBridge, but it sits on the same desk next to my work computer. They both connect wirelessly to the same home network and domestic broadband connection. These timings were after a restart:
DataFrames : 2.408634 seconds (618.85 k allocations: 46.273 MiB, 4.50% gc time, 0.66% compilation time)
CSV : 0.538690 seconds (130.93 k allocations: 7.862 MiB)
Dates : 0.001983 seconds (418 allocations: 28.969 KiB)
ZipArchives : 0.131648 seconds (29.80 k allocations: 1.937 MiB)
GeoStats : 6.741997 seconds (1.45 M allocations: 78.924 MiB, 1.50% gc time, 2.41% compilation time: 92% of which was recompilation)
GeoIO : 7.316927 seconds (2.36 M allocations: 145.175 MiB, 1.83% gc time, 3.07% compilation time: 97% of which was recompilation)
Proj : 0.118895 seconds (23.58 k allocations: 1.279 MiB)
Makie : 7.924092 seconds (1.71 M allocations: 105.704 MiB, 7.24% gc time, 0.28% compilation time)
CSV.read : 3.276746 seconds (869.42 k allocations: 56.316 MiB, 99.25% compilation time: 74% of which was recompilation)
julia> versioninfo()
Julia Version 1.10.2
Commit bd47eca2c8 (2024-03-01 10:14 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: 8 × Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-15.0.7 (ORCJIT, sandybridge)
Threads: 1 default, 0 interactive, 1 GC (on 8 virtual cores)
Environment:
JULIA_EDITOR = code
JULIA_NUM_THREADS = 0
I guess these figures put my home set-up in the clear and do point to the corporate configuration of my laptop.
I’m am not totally sure to understand your last sentence (my English is not very good).
If I read the figures correctly, you have to wait around 25 s at the beginning of a new Julia session. It is better than the several minutes on your professional computer. right ?
You may try to launch internet bandwidth and latency benchmark from both computers to assess my hypothesis.
Task Manager is a good place to start but “Resource Monitor” (as Admin) can sometimes provide more useful details.
If something is hogging the disk during the first run it should be easy to spot.
I don’t think it is the network because he is describing slow loading time in the same environment after rebooting his computer. Meaning it appears to just be local loading of packages, not downloading from the network.
Could you try julia --pkgimages=no? My guess there is that might speed up the loading of a few packages when loading for the first time, but it might make some packages slower to load the second time.
If you want to pursue my DLL hypothesis, I would try this procedure.
julia> using SuiteSparse_jll
julia> SuiteSparse_jll.libcamd_path
"~/.julia/juliaup/julia-1.10.3+0.x64.linux.gnu/bin/../lib/julia/libcamd.so.3"
julia> exit()
julia> using Libdl
julia> @time Libdl.dlopen("~/.julia/juliaup/julia-1.10.3+0.x64.linux.gnu/bin/../lib/julia/libcamd.so.3")
julia> exit()
julia> using Libdl
julia> @time Libdl.dlopen("~/.julia/juliaup/julia-1.10.3+0.x64.linux.gnu/bin/../lib/julia/libcamd.so.3")
julia> exit()
If the above works, then we should try to remove Julia from the equation and write a short C or Zig executable to test dynamic loading.
The end game is that you go back to security, saying that you are a software developer who creates native DLLs for a living.
After restart at cmd prompt:
(project folder)>julia --project -t auto
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.10.3 (2024-04-30)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> using Libdl
julia> @time Libdl.dlopen("C:\\Users\\TGebbels\\.julia\\juliaup\\julia-1.10.3+0.x64.w64.mingw32\\bin\\libcamd.dll")
0.433800 seconds
Ptr{Nothing} @0x00000000658c0000
julia>
In new cmd window but without restart:
(project folder)>julia --project -t auto
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.10.3 (2024-04-30)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> using Libdl
julia> @time Libdl.dlopen("C:\\Users\\TGebbels\\.julia\\juliaup\\julia-1.10.3+0.x64.w64.mingw32\\bin\\libcamd.dll")
0.001135 seconds
Ptr{Nothing} @0x00000000658c0000
julia>
Also, after a restart and in cmd window
(I’m not sure if this is what you meant…):
(Project folder)>julia --project --pkgimages=no -t auto
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.10.3 (2024-04-30)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> print("DataFrames : ")
DataFrames :
julia> @time using DataFrames
4.104258 seconds (611.29 k allocations: 45.355 MiB, 0.58% gc time, 4.38% compilation time)
julia> print("CSV : ")
CSV :
julia> @time using CSV
0.352755 seconds (133.12 k allocations: 8.167 MiB, 1.03% compilation time)
julia> print("Dates : ")
Dates :
julia> @time using Dates
0.001532 seconds (350 allocations: 26.516 KiB)
julia> print("ZipArchives : ")
ZipArchives :
julia> @time using ZipArchives
Precompiling ZipArchives
1 dependency successfully precompiled in 4 seconds. 8 already precompiled.
9.496244 seconds (4.95 M allocations: 363.651 MiB, 1.65% gc time, 56.69% compilation time: 6% of which was recompilation)
julia> print("GeoStats : ")
GeoStats :
julia> @time using GeoStats
Precompiling GeoStats
24 dependencies successfully precompiled in 50 seconds. 167 already precompiled.
1 dependency had output during precompilation:
┌ TableTransforms
│ [pid 9012] waiting for IO to finish:
│ Handle type uv_handle_t->data
│ fs_event 000002094e6e7650->000002094c39db70
│ timer 000002094ecc6940->000002094c39dba0
│ This means that a package has started a background task or event source that has not finished running. For precompilation to complete successfully, the event source needs to be closed explicitly. See the developer documentation on fixing precompilation hangs for more help.
└
56.022109 seconds (4.73 M allocations: 303.766 MiB, 0.10% gc time, 0.43% compilation time: 12% of which was recompilation)
julia> print("GeoIO : ")
GeoIO :
julia> @time using GeoIO
[ Info: Precompiling GeoIO [f5a160d5-e41d-4189-8b61-d57781c419e3]
149.169383 seconds (5.71 M allocations: 371.417 MiB, 0.11% gc time, 2.29% compilation time: 77% of which was recompilation)
julia>
julia> print("Proj : ")
Proj :
julia> @time import Proj
Precompiling Proj
2 dependencies successfully precompiled in 3 seconds. 21 already precompiled.
7.301920 seconds (2.63 M allocations: 181.161 MiB, 3.98% gc time, 60.37% compilation time: 96% of which was recompilation)
julia> print("Makie : ")
Makie :
julia> @time import CairoMakie as Mke
[ Info: Precompiling CairoMakie [13f3f980-e62b-5c42-98c6-ff1f3baf88f0]
[ Info: Precompiling MeshesMakieExt [fd73dc56-13e7-52f5-a3bb-9f1608bf6b57]
[ Info: Precompiling GeoTablesMakieExt [a9b04ea9-04f7-5d13-8c2e-9a13f647e8cf]
[ Info: Precompiling GeoStatsBaseMakieExt [cd7822ce-9704-5052-ac77-ec670066593c]
[ Info: Precompiling GeoStatsFunctionsMakieExt [e88b219a-01c6-5919-9285-8c4e188332f1]
[ Info: Precompiling ShapefileMakieExt [ce22dd63-8704-5a0b-b9df-feab1218054b]
179.774295 seconds (4.81 M allocations: 406.355 MiB, 0.13% gc time, 1.91% compilation time: 85% of which was recompilation)
julia>
julia> print("CSV.read : ")
CSV.read :
julia> @time postcodes = CSV.read("TerminatedPostcodes.csv", DataFrame)
15.203897 seconds (3.19 M allocations: 204.070 MiB, 0.17% gc time, 654.96% compilation time: 9% of which was recompilation)
14660×3 DataFrame
No Admin me, I’m afraid.
Yes, that matches my expectation. We can isolate the effect to DLL loading.
We can build a simple DLL using zig as follows.
julia> using zig_jll
julia> write("add.zig",
"""
export fn add(a: i32, b: i32) i32 {
return a + b;
}
""")
56
julia> zig = zig_jll.zig();
julia> run(`$zig build-lib -dynamic -lc add.zig`);
julia> using Libdl
julia> @time dlopen("./libadd.so") # probably add.dll for you
Ptr{Nothing} @0x0000000000c39e10
julia> libadd = :libadd
:libadd
julia> @ccall libadd.add(5::Int32, 5::Int32)::Int32
10
Later I will expand this to build an executable that loads the DLL with timings. Then we’ll know if this is Julia specific or not.
julia> write("testdll.zig",
"""
const std = @import("std");
pub fn main() !void {
const start = std.time.microTimestamp();
var lib = try std.DynLib.open("libadd.dll");
const stop = std.time.microTimestamp();
const elapsed = stop - start;
const add = lib.lookup(*fn (i32,i32) i32, "add").?;
std.debug.print("add {} in {}\\n", .{add(5,6), elapsed});
}
""")
332
julia> run(`$zig build-exe testdll.zig`);
julia> run(`.\testdll.exe`)
All apparently going without hitch until this point…
julia> zig = zig_jll.zig();
julia> run(`$zig build-lib -dynamic -lc add.zig`);
error: unable to find zig installation directory: FileNotFound
ERROR: failed process: Process(setenv(`'C:\Users\TGebbels\.julia\artifacts\27b9f55e9c432e24e799adc83d883a706684fe98\bin\zig\zig.exe' build-lib -dynamic -lc add.zig`,["WINDIR=C:\\WINDOWS", "PATH=C:\\Users\\TGebbels\\.julia\\juliaup\\julia-1.10.3+0.x64.w64.mingw32\\bin\\..\\lib\\julia;C:\\Users\\TGebbels\\.julia\\juliaup\\julia-1.10.3+0.x64.w64.mingw32\\bin\\..\\lib;C:\\Users\\TGebbels\\.julia\\juliaup\\julia-1.10.3+0.x64.w64.mingw32\\bin;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Users\\TGebbels\\AppData\\Local\\Microsoft\\WindowsApps", "USERDOMAIN_ROAMINGPROFILE=TNLCF", "LOCALAPPDATA=C:\\Users\\TGebbels\\AppData\\Local", "HOMEPATH=\\Users\\TGebbels", "PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 140 Stepping 1, GenuineIntel", "NUMBER_OF_PROCESSORS=8", "PATHEXT=.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC", "SESSIONNAME=Console", "SYSTEMROOT=C:\\WINDOWS" … "PROCESSOR_LEVEL=6", "SYSTEMDRIVE=C:", "USERDNSDOMAIN=tnlcommunityfund.org.uk", "FPS_BROWSER_APP_PROFILE_STRING=Internet Explorer", "PROGRAMW6432=C:\\Program Files", "TEMP=C:\\Users\\TGebbels\\AppData\\Local\\Temp", "HOMEDRIVE=C:", "OPENBLAS_MAIN_FREE=1", "PROCESSOR_ARCHITECTURE=AMD64", "COLORTERM=truecolor"]), ProcessExited(1)) [1]
Stacktrace:
[1] pipeline_error
@ .\process.jl:565 [inlined]
[2] run(::Cmd; wait::Bool)
@ Base .\process.jl:480
[3] run(::Cmd)
@ Base .\process.jl:477
[4] top-level scope
@ REPL[3]:1
julia>
That just tells you you don’t have zig installed so can’t build a zig binary.
We might have to debug on Windows.
You could try downloading Zig directly:
https://ziglang.org/download/
The idea is just to build a simple program that loads dynamic libraries to see if we see the same effect.
I don’t think so. To do so would violate my IT user agreement. To install non-approved software I need explicit IT agreement, which can take weeks and needs a supporting case. (ref: GMT wasn’t approved).
There is no Do run option!
Maybe I’m a lost cause, but tightly controlled IT environments aren’t uncommon.
Non-admin might still show something but it’s a long shot. Sorry.
If these restrictions are making it hard for you to do your job, I highly recommend speaking with your manager. See this thread for more thoughts on the matter:
The zig thing may be a curiosity at this point, but you could try the following to install zig.
winget install -e --id zig.zig
You can get winget from here:
Otherwise, do you have any other compilers available?
Thanks @mbauman. I’ve seen this thread. For my money, they key drivers of reluctance for IT are security concerns and the additional support requirements. The former is probably about adding another risk to a list of risks to manage rather than simply about the absolute level of risk associated with julia.
For context, as I said above, this issue costs me perhaps 5 minutes a day. This isn’t huge compared with the productivity I get from using julia. Since I’ve never used Python, R, Matlab, etc, this cost is unlikely to drive me to abandon julia any time soon. 
Actually, I’m only a lowly data analyst, so this line isn’t an option!
Not that I’m aware of (“I don’t know” rather than “no”).
I’ll put my tinfoil hat on and try winget.
julia --project -t auto
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.10.3 (2024-04-30)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> using zig_jll
julia> write("testdll.zig",
"""
const std = @import("std");
pub fn main() !void {
var start = std.time.microTimestamp();
var lib = try std.DynLib.open("libadd.so");
var stop = std.time.microTimestamp();
var elapsed = stop - start;
var add = lib.lookup(*fn (i32,i32) i32, "add").?;
std.debug.print("add {} in {}\\n", .{add(5,6), elapsed});
}
""")
332
julia> run(`$zig build-exe testdll.zig`);
testdll.zig:7:9: error: local variable is never mutated
var add = lib.lookup(*fn (i32,i32) i32, "add").?;
^~~
testdll.zig:7:9: note: consider using 'const'
testdll.zig:6:9: error: local variable is never mutated
var elapsed = stop - start;
^~~~~~~
testdll.zig:6:9: note: consider using 'const'
testdll.zig:5:9: error: local variable is never mutated
var stop = std.time.microTimestamp();
^~~~
testdll.zig:5:9: note: consider using 'const'
testdll.zig:3:9: error: local variable is never mutated
var start = std.time.microTimestamp();
^~~~~
testdll.zig:3:9: note: consider using 'const'
ERROR: failed process: Process(`zig build-exe testdll.zig`, ProcessExited(1)) [1]
Stacktrace:
[1] pipeline_error
@ .\process.jl:565 [inlined]
[2] run(::Cmd; wait::Bool)
@ Base .\process.jl:480
[3] run(::Cmd)
@ Base .\process.jl:477
[4] top-level scope
@ REPL[3]:1
julia>