Julia 0.7 package development workflow - where are Git repositories of packages?

General Usage
1

Hello,

I’m trying today Julia 0.7 but a package which was written for Julia 0.6 is not compatible.

To install it I did

using Pkg
Pkg.add("ThisBrokenPackage")

I tried to use it using

using ThisBrokenPackage

but encountered an error

I noticed that now with Julia 0.7 and Pkg3, this package code is available at /Users/scls/.julia/packages/ThisBrokenPackage/random_sub_dir

but this is no longer a git repository

How can I fix code of ThisBrokenPackage?
what is the correct workflow now since packages are no longer in Git repositories?

Kind regards

2

With the new package manager, you should use Pkg.dev if you want to clone a full repository. It may be worth to read up a bit on the new package mangager. I have also found this thread useful.

2 Likes
3

It’s little off-topic but does anyone know what is the .julia\clones directory for?

It seems to me that it contains all the “.git” directories of all the downloaded packages. I’ve thought the new Pkg3 uses tarbals and you only clone the git repo after explicit ] dev mypackage command. With only 94 packages it requires thousends of files summing to ~360 MB for every user on the machine.

Is it possible to make this only optional (of course requiring a connection to github in order to do dev command to clone on demand)?

4

That should be happening unless the tarball-download fails, and then it falls back to a git clone.

.julia/clones is a cache for repos that have been added from urls so that the next time they are added, or updated, they do not need to be cloned from scratch.

1 Like
5

Thank you Kristoffer for your heroic work on getting PKG3 go-live, I really do appreciate this.

I’m only using packages from the general registry, installed only by adding Images and Makie (that is where all the dependent ~90 packages come from).

This means there is something wrong with my (corporate) network settings and my Julia 0.7-beta2 needs to use some fallback download method? Is there a log that would help me find out what’s wrong in my case?

6

Please try the following in the REPL:

import Pkg
Pkg.BinaryProvider.probe_platform_engines!(;verbose=true)
cmd = Pkg.BinaryProvider.gen_download_cmd("https://github.com/KristofferC/Crayons.jl/archive/v0.5.1.tar.gz", "Crayons.tar.gz")
run(cmd)
unpack = Pkg.BinaryProvider.gen_unpack_cmd("Crayons.tar.gz", "Crayons")
mkdir("Crayons")
run(unpack)

It should simulate the whole process how Pkg3 downloads a package and we can see if something fails.

7

This is result of my experiment:
Deleted whole .julia dir to have clean start.

julia> import Pkg
julia> Pkg.BinaryProvider.probe_platform_engines!(;verbose=true)
[ Info: Probing for download engine...
[ Info: Probing powershell as a possibility...
[ Info:   Probe successful for powershell
[ Info: Found download engine powershell
[ Info: Probing for compression engine...
[ Info: Probing C:\Julia-0.7.0-beta2\bin\7z.exe as a possibility...
[ Info:   Probe successful for C:\Julia-0.7.0-beta2\bin\7z.exe
[ Info: Found compression engine C:\Julia-0.7.0-beta2\bin\7z.exe
true

julia> cmd = Pkg.BinaryProvider.gen_download_cmd("https://github.com/KristofferC/Crayons.jl/archive/v0.5.1.tar.gz", "Crayons.tar.gz")
`powershell -NoProfile -Command '[System.Net.ServicePointManager]::SecurityProtocol =
    [System.Net.SecurityProtocolType]::Tls12;
$webclient = (New-Object System.Net.Webclient);
$webclient.DownloadFile("https://github.com/KristofferC/Crayons.jl/archive/v0.5.1.tar.gz", "Crayons.tar.gz")'`
julia> run(cmd)
Process(`powershell -NoProfile -Command '[System.Net.ServicePointManager]::SecurityProtocol =
    [System.Net.SecurityProtocolType]::Tls12;
$webclient = (New-Object System.Net.Webclient);
$webclient.DownloadFile("https://github.com/KristofferC/Crayons.jl/archive/v0.5.1.tar.gz", "Crayons.tar.gz")
'`, ProcessExited(0))
julia> unpack = Pkg.BinaryProvider.gen_unpack_cmd("Crayons.tar.gz", "Crayons")
pipeline(`'C:\Julia-0.7.0-beta2\bin\7z.exe' x Crayons.tar.gz -y -so`, stdout=`'C
:\Julia-0.7.0-beta2\bin\7z.exe' x -si -y -ttar -oCrayons`)
julia> mkdir("Crayons")
"Crayons"
julia> run(unpack)
7-Zip [64] 16.04 : Copyright (c) 1999-2016 Igor Pavlov : 2016-10-04
Extracting archive:
--
Path =
Type = tar
Code Page = UTF-8

Everything is Ok

Folders: 3
Files: 18
Size:       50364
Compressed: 15872
Base.ProcessChain(Base.Process[Process(`'C:\Julia-0.7.0-beta2\bin\7z.exe' x Crayons.tar.gz -y -so`, ProcessExited(0)), Process(`'C:\Julia-0.7.0-beta2\bin\7z.exe' x -si -y -ttar -oCrayons`, ProcessExited(0))], RawFD(0x00000000), RawFD(0x00000001), RawFD(0x00000002), #undef)

So far everything seems OK and nothing in infrastructure is missing. The tarball and extracted content exist on disk. .julia only contains logs dir.

Now I attempt to add a package Unitful.jl:

julia> Pkg.add("Unitful")
   Cloning default registries into C:\Users\phlavenk\.julia\registries
   Cloning registry General from "https://github.com/JuliaRegistries/General.git
"
  Updating registry at `C:\Users\phlavenk\.julia\registries\General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
 Resolving package versions...
   Cloning [1986cc42-f94f-5a68-af5c-568840ba703d] Unitful from https://github.co
m/ajkeller34/Unitful.jl.git
[ Info: Installed Unitful ─ v0.9.0
  Updating `C:\Users\phlavenk\.julia\environments\v0.7\Project.toml`
  [1986cc42] + Unitful v0.9.0
  Updating `C:\Users\phlavenk\.julia\environments\v0.7\Manifest.toml`
  [1986cc42] + Unitful v0.9.0
  [37e2e46d] + LinearAlgebra
  [9a3f8284] + Random
  [8dfed614] + Test
  [4ec0a83e] + Unicode

Here, Pkg does not use the previous machinery of powershell and 7z and instead clones the repo by git.
list of my .julia folder:
C:\USERS\PHLAVENK.JULIA\CLONES
** 1986cc42-f94f-5a68-af5c-568840ba703d
** ** hooks
** ** info
** ** objects
** ** refs
C:\USERS\PHLAVENK.JULIA\ENVIRONMENTS
** v0.7
Manifest.toml
Project.toml
C:\USERS\PHLAVENK.JULIA\LOGS
manifest_usage.toml
repl_history.jl
C:\USERS\PHLAVENK.JULIA\PACKAGES
** Unitful
** ** t403
** ** ** docs
** ** ** ** src
** ** ** src
** ** ** test
C:\USERS\PHLAVENK.JULIA\REGISTRIES
** General
** ** .git
** ** ** hooks
** ** **info
…
** ** A
** ** ** AbaqusReader
** ** ** AbbrvKW
…

Is there a way I could run the add command in verbose mode? Or enable somehow logging? As I have a binary installation, I expect the source code of Pkg (stdlib) is not present so it’s not easy to add debug output into the source code.

Could you give me some advice on what might help in my case?

8

I found the problem. It is when using powershell’s webclient AND when the user-agent request parameter is empty (which is the case when using the GitHub API, c# - The server committed a protocol violation. Section=ResponseStatusLine ERROR - Stack Overflow). It should be easy to fix now. Will take a little while for the fix to land in nightly.

Thanks for reporting!

1 Like
9

Thank you a lot for tackling the problem. I will try later with corresponding nightly build.

10

Would be also nice to give info or warning when fallback is used.