# Saving Profile data

**URL:** <https://discourse.julialang.org/t/saving-profile-data/51173>\
**Category:** Performance\
**Created:** [December 3, 2020, 7:54am UTC](https://discourse.julialang.org/t/saving-profile-data/51173 "2020-12-03T07:54:25Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![paulmelis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paulmelis/32/35063_2.png) [@paulmelis](https://discourse.julialang.org/u/paulmelis)\
**Post date:** [December 3, 2020, 7:54am UTC](https://discourse.julialang.org/t/saving-profile-data/51173/1 "2020-12-03T07:54:25Z")

</div>

Is there a way to save the profiling data gathered by the `Profile` module? I don’t see a user-facing API for it, although some methods come close, like `fetch()`. I’d like to capture the profiling statistics of a large batch job on a compute node and then analyze the results on my workstation.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [December 3, 2020, 8:10am UTC](https://discourse.julialang.org/t/saving-profile-data/51173/2 "2020-12-03T08:10:55Z")

</div>

See the `save` function here:  
[https://timholy.github.io/FlameGraphs.jl/stable/reference/#I/O-1](https://timholy.github.io/FlameGraphs.jl/stable/reference/#I/O-1)

---

<div class="post-metadata">

**Author:** ![paulmelis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paulmelis/32/35063_2.png) [@paulmelis](https://discourse.julialang.org/u/paulmelis)\
**Post date:** [December 3, 2020, 8:25am UTC](https://discourse.julialang.org/t/saving-profile-data/51173/3 "2020-12-03T08:25:53Z")

</div>

Hmm, interesting, seem to have missed that. But then again, it appears the version of FlameGraphs in my Julia 1.5.3 installation is only 0.1.1 and `save` seems to have been added later. But updating the package doesn’t seem to help?

```julia
(@v1.5) pkg> up FlameGraphs
   Updating registry at `~/.julia/registries/General`
   Updating git-repo `https://github.com/JuliaRegistries/General.git`
No Changes to `~/.julia/environments/v1.5/Project.toml`
No Changes to `~/.julia/environments/v1.5/Manifest.toml`

(@v1.5) pkg> st FlameGraphs
Status `~/.julia/environments/v1.5/Project.toml`
  [08572546] FlameGraphs v0.1.1

```

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [December 3, 2020, 8:29am UTC](https://discourse.julialang.org/t/saving-profile-data/51173/4 "2020-12-03T08:29:44Z")

</div>

Use `pkg> add FlameGraphs@0.2` and then fix any conflicts:  
[https://julialang.github.io/Pkg.jl/v1/managing-packages/#conflicts](https://julialang.github.io/Pkg.jl/v1/managing-packages/#conflicts)

See also this wonderful package:  
[https://github.com/KristofferC/RegistryCompatTools.jl](https://github.com/KristofferC/RegistryCompatTools.jl)

---

<div class="post-metadata">

**Author:** ![paulmelis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paulmelis/32/35063_2.png) [@paulmelis](https://discourse.julialang.org/u/paulmelis)\
**Post date:** [December 3, 2020, 8:35am UTC](https://discourse.julialang.org/t/saving-profile-data/51173/5 "2020-12-03T08:35:54Z")

</div>

> [@tim.holy](#):
>
> Use `pkg> add FlameGraphs@0.2` and then fix any conflicts:

Thanks, managed to get that to work! The package management in Julia is something I haven’t really dived into yet. People always complain about Python’s package management, but at least that one is quite straightforward once you understand how Python searches for a particular package/module, with `pip` doing mostly no management at all. With Julia it seems there’s a bit more going on behind the scenes with `Pkg` doing lots of work for you.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [December 3, 2020, 8:38am UTC](https://discourse.julialang.org/t/saving-profile-data/51173/6 "2020-12-03T08:38:52Z")

</div>

Yes, the infrastructure to make compatibility guarantees actually work pushes the resolution of version conflicts onto users. Fortunately it’s really easy to fix once you understand the error messages…but it takes sitting down and carefully working through a couple of examples before that starts seeming easy.

---

<div class="post-metadata">

**Author:** ![paulmelis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paulmelis/32/35063_2.png) [@paulmelis](https://discourse.julialang.org/u/paulmelis)\
**Post date:** [December 3, 2020, 8:42am UTC](https://discourse.julialang.org/t/saving-profile-data/51173/7 "2020-12-03T08:42:42Z")

</div>

> [@tim.holy](#):
>
> but it takes sitting down and carefully working through a couple of examples before that starts seeming easy.

I think I made a first step in resolving the confict with `ProfileSVG`’s version that was installed 😉

By the way, the FlameGraphs docs mention 3 `save` methods, one with signature `save(filename::AbstractString, data, ldict)`, but that particular one doesn’t seem to be available. I only see 2 that both take a `FileIO` as first argument. Or is there an implicit conversion that should work?

```julia
julia> @profile f()
833334500000

julia> data, lidict = Profile.retrieve()
...

julia> FlameGraphs.save("doh.jlprof", data, lidict)
ERROR: MethodError: no method matching save(::String, ::Array{UInt64,1}, ::Dict{UInt64,Array{Base.StackTraces.StackFrame,1}})
Closest candidates are:
  save(::FileIO.File{FileIO.DataFormat{:JLPROF}}, ::AbstractArray{var"#s23",1} where var"#s23"<:Unsigned, ::Dict{UInt64,Array{Base.StackTraces.StackFrame,1}}) at /home/paulm/.julia/packages/FlameGraphs/KTpPA/src/io.jl:20
Stacktrace:
 [1] top-level scope at REPL[13]:1
 [2] run_repl(::REPL.AbstractREPL, ::Any) at /build/julia/src/julia-1.5.2/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:288

```

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [December 3, 2020, 8:49am UTC](https://discourse.julialang.org/t/saving-profile-data/51173/8 "2020-12-03T08:49:16Z")

</div>

Load the FileIO package. If you can come up with a way to clarify that in the docstring, please contribute! You can enter your suggestions directly in the webbrowser (despite appearances, this link will take you directly to the file that needs to be edited):

[https://github.com/timholy/FlameGraphs.jl/edit/master/src/io.jl](https://github.com/timholy/FlameGraphs.jl/edit/master/src/io.jl)

---

<div class="post-metadata">

**Author:** ![paulmelis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paulmelis/32/35063_2.png) [@paulmelis](https://discourse.julialang.org/u/paulmelis)\
**Post date:** [December 3, 2020, 9:09am UTC](https://discourse.julialang.org/t/saving-profile-data/51173/9 "2020-12-03T09:09:05Z")

</div>

> [@tim.holy](#):
>
> Load the FileIO package. If you can come up with a way to clarify that in the docstring, please contribute!

It took me quite some time just now to figure out how FileIO is supposed to work. The Julia manual has no mention of the package at all and the FileIO readme doesn’t show how to create a `FileIO` of a specific type (it only mentions `load` and `save`). I needed to dig into the FlameGraphs test code to see that `f = File(format"JLPROF", "test.jlprof")` is apparently the way. But that still doesn’t explain if passing only the output file name, e.g. `"test.jlprof"`, to `FlameGraph.save()` is supposed to work?

By the way, shouldn’t `FileIO` be a dependency of `FlameGraphs`? I did not have it installed, while the signatures for `FlameGraphs.save` obviously refer to `FileIO` types. I don’t know enough of Julia yet to understand how that can work.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [December 3, 2020, 9:17am UTC](https://discourse.julialang.org/t/saving-profile-data/51173/10 "2020-12-03T09:17:13Z")

</div>

All you have to do is say `using FileIO` and it all works.

> By the way, shouldn’t `FileIO` be a dependency of `FlameGraphs`

It is: [https://github.com/timholy/FlameGraphs.jl/blob/master/Project.toml](https://github.com/timholy/FlameGraphs.jl/blob/master/Project.toml) But that’s not the same thing as it being in your namespace. You’re in charge of that.

I went ahead and made a PR, see [Add an example to `save` by timholy · Pull Request #34 · timholy/FlameGraphs.jl · GitHub](https://github.com/timholy/FlameGraphs.jl/pull/34)

---

<div class="post-metadata">

**Author:** ![paulmelis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paulmelis/32/35063_2.png) [@paulmelis](https://discourse.julialang.org/u/paulmelis)\
**Post date:** [December 3, 2020, 9:43am UTC](https://discourse.julialang.org/t/saving-profile-data/51173/11 "2020-12-03T09:43:59Z")

</div>

Well, I’m happy to say that the example you added indeed works as expected in saving profiling data.

But now I’m completely confused as to how `using FlameGraphs` can work when I have neither `FileIO` or any of its other dependencies like `LeftChildRightSiblingTrees` installed. I guess I have a fundamental misunderstanding of how module loading/execution works and I should go back to the “First steps” subforum. Shouldn’t `using FlameGraphs` be more or less the same as executing [https://github.com/timholy/FlameGraphs.jl/blob/master/src/FlameGraphs.jl](https://github.com/timholy/FlameGraphs.jl/blob/master/src/FlameGraphs.jl)? Any particular sections of the manual I should go through to understand this? I looked at the section on Modules but that doesn’t really enlighten on the above.

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [December 3, 2020, 9:50am UTC](https://discourse.julialang.org/t/saving-profile-data/51173/12 "2020-12-03T09:50:02Z")

</div>

The packages you have installed are different from the package names in your namespace. `pkg> add SomePackage` puts `SomePackage` in your namespace; any dependency gets installed regardless of whether it’s in your namespace. Check out the difference between `pkg> st` and what’s in your `~/.julia/packages` directory.

This is part of the mechanism Julia uses to handle conflicts when two different packages have the same name: you can have two `Foo` packages installed simultaneously, but `PkgA/Project.toml` depends on one of them and `PkgB/Project.toml` depends on the other. That Just Works without any intervention from you. However, if you have both of them in different registries, then `using Foo` without any qualifier would be ambiguous. That’s why `add` fundamentally is about maintaining your namespace, not about installing software.

---

<div class="post-metadata">

**Author:** ![jdad](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jdad/32/4739_2.png) [@jdad](https://discourse.julialang.org/u/jdad)\
**Post date:** [December 3, 2020, 9:59am UTC](https://discourse.julialang.org/t/saving-profile-data/51173/13 "2020-12-03T09:59:43Z")

</div>

You can check that FileIO is already loaded as a dependency by using `st -m` showing it is present in the Manifest (but possibly not in the Project)

---

<div class="post-metadata">

**Author:** ![paulmelis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paulmelis/32/35063_2.png) [@paulmelis](https://discourse.julialang.org/u/paulmelis)\
**Post date:** [December 3, 2020, 10:17am UTC](https://discourse.julialang.org/t/saving-profile-data/51173/14 "2020-12-03T10:17:48Z")

</div>

Thank you both, that enlightens the fundamentals a lot. I now see that I should read the docs on `Pkg` more closely. I had only skimmed [Pkg · The Julia Language](https://docs.julialang.org/en/v1/stdlib/Pkg/) so far, assuming the general concepts of adding/updating/removing packages would work mostly the same as other programming environments. But apparently the crucial concepts of projects and the manifest are only really discussed in the `Pkg` docs (the mentions of `Project.toml` and `Manifest.toml` to me suggested too much detail for daily use).

For completeness sake, here’s the two snippets for saving and loading profile data, including visualizing them in ProfileView:

```julia
using Profile, FileIO
@profile test()
save("test.jlprof", Profile.retrieve()...)

```

```julia
using ProfileView, FileIO
data = load("test.jlprof")
ProfileView.view(data[1], lidict=data[2])

```