Is there a way to save the profiling data gathered by the Profile module? I don’t see a user-facing API for it, although some methods come close, like fetch(). I’d like to capture the profiling statistics of a large batch job on a compute node and then analyze the results on my workstation.
See the save function here:
https://timholy.github.io/FlameGraphs.jl/stable/reference/#I/O-1
2 Likes
Hmm, interesting, seem to have missed that. But then again, it appears the version of FlameGraphs in my Julia 1.5.3 installation is only 0.1.1 and save seems to have been added later. But updating the package doesn’t seem to help?
(@v1.5) pkg> up FlameGraphs
Updating registry at `~/.julia/registries/General`
Updating git-repo `https://github.com/JuliaRegistries/General.git`
No Changes to `~/.julia/environments/v1.5/Project.toml`
No Changes to `~/.julia/environments/v1.5/Manifest.toml`
(@v1.5) pkg> st FlameGraphs
Status `~/.julia/environments/v1.5/Project.toml`
[08572546] FlameGraphs v0.1.1
Use pkg> add FlameGraphs@0.2 and then fix any conflicts:
https://julialang.github.io/Pkg.jl/v1/managing-packages/#conflicts
See also this wonderful package:
https://github.com/KristofferC/RegistryCompatTools.jl
1 Like
Thanks, managed to get that to work! The package management in Julia is something I haven’t really dived into yet. People always complain about Python’s package management, but at least that one is quite straightforward once you understand how Python searches for a particular package/module, with pip doing mostly no management at all. With Julia it seems there’s a bit more going on behind the scenes with Pkg doing lots of work for you.
Yes, the infrastructure to make compatibility guarantees actually work pushes the resolution of version conflicts onto users. Fortunately it’s really easy to fix once you understand the error messages…but it takes sitting down and carefully working through a couple of examples before that starts seeming easy.
I think I made a first step in resolving the confict with ProfileSVG’s version that was installed 
By the way, the FlameGraphs docs mention 3 save methods, one with signature save(filename::AbstractString, data, ldict), but that particular one doesn’t seem to be available. I only see 2 that both take a FileIO as first argument. Or is there an implicit conversion that should work?
julia> @profile f()
833334500000
julia> data, lidict = Profile.retrieve()
...
julia> FlameGraphs.save("doh.jlprof", data, lidict)
ERROR: MethodError: no method matching save(::String, ::Array{UInt64,1}, ::Dict{UInt64,Array{Base.StackTraces.StackFrame,1}})
Closest candidates are:
save(::FileIO.File{FileIO.DataFormat{:JLPROF}}, ::AbstractArray{var"#s23",1} where var"#s23"<:Unsigned, ::Dict{UInt64,Array{Base.StackTraces.StackFrame,1}}) at /home/paulm/.julia/packages/FlameGraphs/KTpPA/src/io.jl:20
Stacktrace:
[1] top-level scope at REPL[13]:1
[2] run_repl(::REPL.AbstractREPL, ::Any) at /build/julia/src/julia-1.5.2/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:288
1 Like
Load the FileIO package. If you can come up with a way to clarify that in the docstring, please contribute! You can enter your suggestions directly in the webbrowser (despite appearances, this link will take you directly to the file that needs to be edited):
https://github.com/timholy/FlameGraphs.jl/edit/master/src/io.jl
It took me quite some time just now to figure out how FileIO is supposed to work. The Julia manual has no mention of the package at all and the FileIO readme doesn’t show how to create a FileIO of a specific type (it only mentions load and save). I needed to dig into the FlameGraphs test code to see that f = File(format"JLPROF", "test.jlprof") is apparently the way. But that still doesn’t explain if passing only the output file name, e.g. "test.jlprof", to FlameGraph.save() is supposed to work?
By the way, shouldn’t FileIO be a dependency of FlameGraphs? I did not have it installed, while the signatures for FlameGraphs.save obviously refer to FileIO types. I don’t know enough of Julia yet to understand how that can work.
All you have to do is say using FileIO and it all works.
By the way, shouldn’t
FileIObe a dependency ofFlameGraphs
It is: https://github.com/timholy/FlameGraphs.jl/blob/master/Project.toml But that’s not the same thing as it being in your namespace. You’re in charge of that.
I went ahead and made a PR, see Add an example to `save` by timholy · Pull Request #34 · timholy/FlameGraphs.jl · GitHub
Well, I’m happy to say that the example you added indeed works as expected in saving profiling data.
But now I’m completely confused as to how using FlameGraphs can work when I have neither FileIO or any of its other dependencies like LeftChildRightSiblingTrees installed. I guess I have a fundamental misunderstanding of how module loading/execution works and I should go back to the “First steps” subforum. Shouldn’t using FlameGraphs be more or less the same as executing https://github.com/timholy/FlameGraphs.jl/blob/master/src/FlameGraphs.jl? Any particular sections of the manual I should go through to understand this? I looked at the section on Modules but that doesn’t really enlighten on the above.
The packages you have installed are different from the package names in your namespace. pkg> add SomePackage puts SomePackage in your namespace; any dependency gets installed regardless of whether it’s in your namespace. Check out the difference between pkg> st and what’s in your ~/.julia/packages directory.
This is part of the mechanism Julia uses to handle conflicts when two different packages have the same name: you can have two Foo packages installed simultaneously, but PkgA/Project.toml depends on one of them and PkgB/Project.toml depends on the other. That Just Works without any intervention from you. However, if you have both of them in different registries, then using Foo without any qualifier would be ambiguous. That’s why add fundamentally is about maintaining your namespace, not about installing software.
3 Likes
You can check that FileIO is already loaded as a dependency by using st -m showing it is present in the Manifest (but possibly not in the Project)
1 Like
Thank you both, that enlightens the fundamentals a lot. I now see that I should read the docs on Pkg more closely. I had only skimmed Pkg · The Julia Language so far, assuming the general concepts of adding/updating/removing packages would work mostly the same as other programming environments. But apparently the crucial concepts of projects and the manifest are only really discussed in the Pkg docs (the mentions of Project.toml and Manifest.toml to me suggested too much detail for daily use).
For completeness sake, here’s the two snippets for saving and loading profile data, including visualizing them in ProfileView:
using Profile, FileIO
@profile test()
save("test.jlprof", Profile.retrieve()...)
using ProfileView, FileIO
data = load("test.jlprof")
ProfileView.view(data[1], lidict=data[2])
3 Likes