Contents of `.julia/registries` deleted upon Pkg operations or Pluto notebook use

General Usage
,
1

So after a few to many edits and updates, here is a cleaner and updated second rewrite of the post.

The problem is that during what I have identified as “trigger events”, the contents of my .julia/registries folder is deleted. This can occur when I

  1. update or install packages (specifically during the “Updating registry at C:\Users\username.julia\registries\General” step)
  2. Run a pluto notebook. The notebook can be empty or contain a single cell with 1+1 and the problem can occur. When the error occurs, the notebook is not responsive, and a result is never evaluated, nor does the cell timer count.

The deletion of the folder contents happen during the Updating registry step, or the notebook has appear to run for about a minute. The behaviour is not consistently reproduced, but it is reoccuring. Package updating and installing has been a trigger event at least 3 times, and the notebook process has been a trigger event about 10 times. The problem can be consistent, then be fixed for time time, but then reoccur again soon.

Below is the error thrown upon stopping a not-responding pluto-notebook:

┌ Error: mktempdir cleanup
│   exception =
│    InterruptException:
│    Stacktrace:
│     [1] unlink(::String) at .\file.jl:917
│     [2] rm(::String; force::Bool, recursive::Bool) at .\file.jl:268
│     [3] rm(::String; force::Bool, recursive::Bool) at .\file.jl:278 (repeats 3 times)
│     [4] mktempdir(::Pkg.Types.var"#107#110"{String}, ::String; prefix::String) at .\file.jl:714
│     [5] mktempdir(::Function, ::String) at .\file.jl:707 (repeats 2 times)
│     [6] update_registries(::Pkg.Types.Context, ::Array{Pkg.Types.RegistrySpec,1}; force::Bool) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\Types.jl:1119
│     [7] up(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; level::Pkg.Types.UpgradeLevel, mode::Pkg.Types.PackageMode, update_registry::Bool, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:227
│     [8] #up#38 at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:68 [inlined]
│     [9] instantiate(::Pkg.Types.Context; manifest::Nothing, update_registry::Bool, verbose::Bool, platform::Pkg.BinaryPlatforms.Windows, kwargs::Base.Iterators.Pairs{Symbol,Base.DevNull,Tuple{Symbol},NamedTuple{(:io,),Tuple{Base.DevNull}}}) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Pkg\src\API.jl:810
│     [10] top-level scope at C:\Users\densb\.julia\packages\Pluto\luIaC\src\runner\Loader.jl:19
│     [11] include(::String) at .\client.jl:457
│     [12] top-level scope at none:1
│     [13] eval(::Module, ::Any) at .\boot.jl:331
│     [14] (::Distributed.var"#103#104"{Distributed.CallMsg{:call}})() at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\process_messages.jl:290
│     [15] run_work_thunk(::Distributed.var"#103#104"{Distributed.CallMsg{:call}}, ::Bool) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\process_messages.jl:79
│     [16] run_work_thunk(::Distributed.RemoteValue, ::Function) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\process_messages.jl:88
│     [17] (::Distributed.var"#96#98"{Distributed.RemoteValue,Distributed.var"#103#104"{Distributed.CallMsg{:call}}})() at .\task.jl:356
└ @ Base.Filesystem file.jl:717
TaskFailedException:
InterruptException:
Stacktrace:
 [1] sync_end(::Channel{Any}) at .\task.jl:314
 [2] macro expansion at .\task.jl:333 [inlined]
 [3] remotecall_eval(::Module, ::Array{Int64,1}, ::Expr) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\macros.jl:218
 [4] create_workspaceprocess(; compiler_options::Pluto.Configuration.CompilerOptions) at C:\Users\densb\.julia\packages\Pluto\luIaC\src\evaluation\WorkspaceManager.jl:170
 [5] make_workspace(::Tuple{Pluto.ServerSession,Pluto.Notebook}) at C:\Users\densb\.julia\packages\Pluto\luIaC\src\evaluation\WorkspaceManager.jl:43
 [6] #7 at C:\Users\densb\.julia\packages\Pluto\luIaC\src\evaluation\WorkspaceManager.jl:188 [inlined]
 [7] macro expansion at C:\Users\densb\.julia\packages\Pluto\luIaC\src\evaluation\Tokens.jl:76 [inlined]
 [8] (::Pluto.var"#1#2"{Pluto.WorkspaceManager.var"#7#9"{Tuple{Pluto.ServerSession,Pluto.Notebook}},Pluto.Promise{Pluto.WorkspaceManager.Workspace}})() at .\task.jl:356┌ Warning: Response function to message of type complete failed
└ @ Pluto C:\Users\densb\.julia\packages\Pluto\luIaC\src\webserver\WebServer.jl:359

Stacktrace:┌ Warning: Reading WebSocket client stream failed for unknown reason:
│   parentbody =
│    Dict{Any,Any} with 5 entries:
│      "client_id"   => "1wwwjfk"
│      "body"        => Dict{Any,Any}("query"=>"sq")
│      "request_id"  => "bkboqy"
│      "notebook_id" => "03f35370-5341-11eb-0379-49c494b1da65"
│      "type"        => "complete"
│   exception =
│    TaskFailedException:
│    InterruptException:
│    Stacktrace:
│     [1] sync_end(::Channel{Any}) at .\task.jl:314
│     [2] macro expansion at .\task.jl:333 [inlined]
│     [3] remotecall_eval(::Module, ::Array{Int64,1}, ::Expr) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.5\Distributed\src\macros.jl:218
│     [4] create_workspaceprocess(; compiler_options::Pluto.Configuration.CompilerOptions) at C:\Users\densb\.julia\packages\Pluto\luIaC\src\evaluation\WorkspaceManager.jl:170
│     [5] make_workspace(::Tuple{Pluto.ServerSession,Pluto.Notebook}) at C:\Users\densb\.julia\packages\Pluto\luIaC\src\evaluation\WorkspaceManager.jl:43
│     [6] #7 at C:\Users\densb\.julia\packages\Pluto\luIaC\src\evaluation\WorkspaceManager.jl:188 [inlined]
│     [7] macro expansion at C:\Users\densb\.julia\packages\Pluto\luIaC\src\evaluation\Tokens.jl:76 [inlined]
│     [8] (::Pluto.var"#1#2"{Pluto.WorkspaceManager.var"#7#9"{Tuple{Pluto.ServerSession,Pluto.Notebook}},Pluto.Promise{Pluto.WorkspaceManager.Workspace}})() at .\task.jl:356
│    Stacktrace:
│     [1] wait at .\task.jl:267 [inlined]
│     [2] wait(::Pluto.Promise{Pluto.WorkspaceManager.Workspace}) at C:\Users\densb\.julia\packages\Pluto\luIaC\src\evaluation\Tokens.jl:83
│     [3] get_workspace at C:\Users\densb\.julia\packages\Pluto\luIaC\src\evaluation\WorkspaceManager.jl:190 [inlined]
│     [4] (::Pluto.var"#201#203")(::Pluto.ServerSession, ::Dict{Any,Any}, ::Pluto.Notebook; initiator::Pluto.Initiator) at C:\Users\densb\.julia\packages\Pluto\luIaC\src\webserver\REPLTools.jl:60
│     [5] process_ws_message(::Pluto.ServerSession, ::Dict{Any,Any}, ::HTTP.WebSockets.WebSocket{HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}}) at C:\Users\densb\.julia\packages\Pluto\luIaC\src\webserver\WebServer.jl:357
│     [6] (::Pluto.var"#215#225"{Pluto.ServerSession,Base.RefValue{Function}})(::HTTP.WebSockets.WebSocket{HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}}) at C:\Users\densb\.julia\packages\Pluto\luIaC\src\webserver\WebServer.jl:160
│     [7] upgrade(::Pluto.var"#215#225"{Pluto.ServerSession,Base.RefValue{Function}}, ::HTTP.Streams.Stream{HTTP.Messages.Request,HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}}; binary::Bool) at C:\Users\densb\.julia\packages\HTTP\IAI92\src\WebSockets.jl:160
│     [8] upgrade at C:\Users\densb\.julia\packages\HTTP\IAI92\src\WebSockets.jl:142 [inlined]
│     [9] (::Pluto.var"#214#224"{Pluto.ServerSession,HTTP.Handlers.Router{Symbol("##253")},Base.RefValue{Function}})(::HTTP.Streams.Stream{HTTP.Messages.Request,HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}}) at C:\Users\densb\.julia\packages\Pluto\luIaC\src\webserver\WebServer.jl:147
│     [10] handle at C:\Users\densb\.julia\packages\HTTP\IAI92\src\Handlers.jl:269 [inlined]
│     [11] #4 at C:\Users\densb\.julia\packages\HTTP\IAI92\src\Handlers.jl:345 [inlined]
│     [12] macro expansion at C:\Users\densb\.julia\packages\HTTP\IAI92\src\Servers.jl:367 [inlined]
│     [13] (::HTTP.Servers.var"#13#14"{HTTP.Handlers.var"#4#5"{HTTP.Handlers.StreamHandlerFunction{Pluto.var"#214#224"{Pluto.ServerSession,HTTP.Handlers.Router{Symbol("##253")},Base.RefValue{Function}}}},HTTP.ConnectionPool.Transaction{Sockets.TCPSocket},HTTP.Streams.Stream{HTTP.Messages.Request,HTTP.ConnectionPool.Transaction{Sockets.TCPSocket}}})() at .\task.jl:356
└ @ Pluto C:\Users\densb\.julia\packages\Pluto\luIaC\src\webserver\WebServer.jl:168

 [1] wait at .\task.jl:267 [inlined]
 [2] wait(::Pluto.Promise{Pluto.WorkspaceManager.Workspace}) at C:\Users\densb\.julia\packages\Pluto\luIaC\src\evaluation\Tokens.jl:83
 [3] get_workspace at C:\Users\densb\.julia\packages\Pluto\luIaC\src\evaluation\WorkspaceManager.jl:190 [inlined]
 [4] delete_vars(::Tuple{Pluto.ServerSession,Pluto.Notebook}, ::Set{Symbol}, ::Set{Tuple{Base.UUID,Array{Symbol,1}}}, ::Set{Expr}; kwargs::Base.Iterators.Pairs{Symbol,Array{Pluto.Cell,1},Tuple{Symbol},NamedTuple{(:to_run,),Tuple{Array{Pluto.Cell,1}}}}) at C:\Users\densb\.julia\packages\Pluto\luIaC\src\evaluation\WorkspaceManager.jl:273
 [5] run_reactive!(::Pluto.ServerSession, ::Pluto.Notebook, ::Pluto.NotebookTopology, ::Pluto.NotebookTopology, ::Array{Pluto.Cell,1}; deletion_hook::Function, persist_js_state::Bool) at C:\Users\densb\.julia\packages\Pluto\luIaC\src\evaluation\Run.jl:65
 [6] run_reactive! at C:\Users\densb\.julia\packages\Pluto\luIaC\src\evaluation\Run.jl:26 [inlined]
 [7] macro expansion at C:\Users\densb\.julia\packages\Pluto\luIaC\src\evaluation\Run.jl:13 [inlined]
 [8] (::Pluto.var"#79#81"{Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}},Pluto.ServerSession,Pluto.Notebook,Pluto.NotebookTopology,Pluto.NotebookTopology,Array{Pluto.Cell,1}})() at .\task.jl:356

At some point yesterday, the notebook started working consistently, but gave the following error message every time I stopped the process:

┌ Warning: temp cleanup
│   exception =
│    schedule: Task not runnable
│    Stacktrace:
│     [1] error(::String) at .\error.jl:33
│     [2] enq_work(::Task) at .\task.jl:526
│     [3] yield at .\task.jl:626 [inlined]
│     [4] yield at .\task.jl:625 [inlined]
│     [5] Channel{Any}(::Base.Filesystem.var"#_it#24"{Bool,Bool,typeof(throw),String,Array{String,1},Array{String,1}}, ::Int64; taskref::Nothing, spawn::Bool) at .\channels.jl:135
│     [6] #Channel#488 at .\channels.jl:157 [inlined]
│     [7] Channel at .\channels.jl:150 [inlined]
│     [8] walkdir(::String; topdown::Bool, follow_symlinks::Bool, onerror::typeof(throw)) at .\file.jl:913
│     [9] walkdir at .\file.jl:872 [inlined]
│     [10] prepare_for_deletion(::String) at .\file.jl:478
│     [11] temp_cleanup_purge(::Bool) at .\file.jl:513
│     [12] temp_cleanup_purge() at .\file.jl:507
│     [13] _atexit() at .\initdefs.jl:316
└ @ Base.Filesystem file.jl:518
      From worker 2:    InterruptException()
2

Is this on Windows?

3

Julia Version 1.5.3
Commit 788b2c77c1 (2020-11-09 13:37 UTC)
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: Intel(R) Core™ i5-8265U CPU @ 1.60GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-9.0.1 (ORCJIT, skylake)

4

Is there any chance the folders are deleted by a too zealous antivirus? Is running a pluto notebook a reproducible recipe to make them disappear?

5

Those slow appearance of registry subdirectories is IMHO a sure sign of an impending registry update of General by tar extract operation.

On Windows, using Julia 1.5.x it switched the Pkg update to Pkg server protocol just after the release. Using the Pkg protocol make registry update extremely slow on windows because of the tar extraction (slow because I/O on NTFS is slow), and can be aggravated by the antivirus checking for each file.

So

  1. Are you doing Pkg operations ( eg, Pkg.update() ) inside you Pluto notebook ? Ordinary notebook operation should not trigger registry update IMHO …

  2. you could use setting Pkg server to “” , to force Pkg operation to revert to old GitHub protocol. For that, before calling Pluto/using the notebook, do
    ENV["PKG_SERVER"]=""
    (Or may be ENV["JULIA_PKG_SERVER"]="" )
    before those Pkg operations

6

For all I know, that could be. But I don’t think so - I don’t have any more than the build in virus protection in windows 10, and a full scan just not came up clean.

It is reproducable when the bug occurs. It seems like updating the package and getting freshly installed contents back, and then launching a notebook from the same process can temporarily fix the issue.




.

The speed is not really the issue - it is more about the folder getting deleted. But a faster reinstall is definitely something to make the problem more endurable. Thanks.

7

This is the second time that the contents of the registries folder have been deleted when I try to add a package, terminating in an error. The deletion occurs during the registry updating at line 2:

(@v1.5) pkg> add NLSolve
   Updating registry at `C:\Users\username\.julia\registries\General`
ERROR: IOError: stat: permission denied (EACCES) for file "C:\\Users\\densb\\.julia\\registries\\General"

Running exacly the same command afterwards reinstalls the contents, and then the package. Then everything is fine for a little while, but according to experience, it will occur again soon when triggered by a package update, installation, or notebook.

8

A registry update happens in three steps:

  1. Download new registry to tmp folder,
  2. Delete the old registry in .julia/registries,
  3. Copy the new registry from tmp to .julia/registries,

so it is expected that content will be deleted (step 2) but Windows seems to have pretty big problems with step 3. See Pkg.jl#1668 and Pkg.jl#2014 for example.

9

But should the registry be updated (And therefore re-downloaded) every time I install a package?

And I don’t understand why a pluto-notebook that defines three variables and computes their sum should cause a botched registry update…

10

It updates once per Julia session (or if you ask for it implicitly such as with Pkg.update).

I don’t know, I guess you have to ask Pluto devs about that.