Which files or subdirectories in `~/.julia/` can I delete?

General Usage
1

I am running Julia 1.5.4 on a HPC cluster, which has a strict 200,000 file count per user.

~/.julia consumes >75% of this quota:

$ du -a | cut -d/ -f2 | sort | uniq -c | sort -nr
 153692 .julia
  19159 .conda
  10007 NFBC_chr21_sim
   1559 .local
    187 .cache
...

Insider ~/.julia, the breakdown looks like

$ du -a | cut -d/ -f2 | sort | uniq -c | sort -nr
  51947 conda
  45479 packages
  28661 registries
  23839 artifacts
   1945 clones
   1570 compiled
    217 dev
     16 environments
      9 lib
      4 prefs
      4 logs

This thread suggested to delete clones which doesn’t help much.

Anything else I can do? Can I delete conda or registries? What about ~/.conda?

1 Like
2

I have ~/.julia/conda but it’s nearly empty and I don’t know how it even got there. Might move it out of the way and see if anything breaks.

I think /registries is needed for installing julia packages, though you could delete it and re-add it later if you are really stuck. /clones will accumulate over time but probably isn’t necessary.

Anything labeled “cache” here can be deleted and rebuilt as needed, it just might take some time to rebuild them. Store settings and packages in XDG directories instead of ~/.julia · Issue #4630 · JuliaLang/julia · GitHub

1 Like
3

The other thing that may help a little is to use the package manager’s gc command to clear out some files/packages no longer needed.

E.g.
(@v1.6) pkg> gc

2 Likes
4

I have a related question about artifacts. My artifacts folder is 20GB. I have no doubt that some of those are probably from packages I installed to try out once and haven’t used since. Is it possible to do some sort of purge there without repercussions? I did run ] gc and it deleted only a trivial amount.

1 Like
5

Stefan’s answer in the linked thread is that they can be deleted and rebuilt.

2 Likes
6

Ah ok I misread your post! I thought “cache” was referring to a folder. I will give that a shot.

7

If your cluster has a scratch filesystem with a larger quota, you can also put ~/.julia there instead by setting JULIA_DEPOT_PATH appropriately.

3 Likes
8

If you want a more thorough garbage collection, you can do ] gc --all

2 Likes
9

Yea… not exactly a real killer is it?

(UCARMAModels) pkg> gc --all
      Active manifest files: 44 found
      Active artifact files: 168 found
      Active scratchspaces: 7 found
     Deleted 39 package installations (40.117 MiB)
     Deleted 1 artifact installation (6.348 MiB)
1 Like
10

Oof. That is disappointing. I guess the artifacts that didn’t get GC’d must correspond to packages that you actually still have installed in some environment. If you uninstall those packages and then do gc --all, it should remove the artifacts.

I agree that the results you got are kind of underwhelming.

11

To be fair, I had already run a regular gc not all that long ago. Maybe 2 weeks? But it didn’t do much better than that. I agree that the packages are likely hidden in Project.toml files, but I wonder if some sort of expiration date could be set?

1 Like
12

For a start it would be great if there were commands for listing all the “tracked” environments and for “untracking” (some of) them. These features have been discussed ~2 years ago but have never been implemented (please correct me if I’m wrong).

(@tbeason Note that you have 44 Manifests and I bet that you don’t care about some / many of them anymore.)

6 Likes
13

You can pass --verbose to gc to see the manifests that are being “tracked”.

2 Likes
14

I once quickly wrote a small package to make it easier to clean up environments: PkgCleanup.jl. Not very thoroughly developed though

5 Likes
15

Last time I wanted to purge my filesystem from unused packages I had the same problem: many environments tracked, which I didn’t really want. Manually editing .julia/logs/manifest_usage.toml helped ]gc to be more efficient.

16

Which is what PkgCleanup.jl helps doing

1 Like
17

The artifacts is full of unneeded files. In almost all cases only the shared lib file is needed but it comes with include, .exe, logs, *.a or *.lib, ,gz etc.
They should only come with the needed files.

18

I take you’re volunteering to help?

2 Likes
19

Thanks! This looks really helpful but…isn’t working for me? I have tried multiple times now but it never seems to remove anything from the active manifests list. In fact even if I select n=none it doesn’t remove any. I will edit manually.

20

Mind opening an issue with more details?

I didn’t claim that package is battle-tested, that’s like the result of a couple of hours of coding out of frustration one night, never touched it again since then.

1 Like