How to use Downloads.download() with SFTP?

Specific Domains Web Stack
1

Hi,
Is it possible to do SFTP with Julia ? I use filezilla to download files on a remote computer but I do not manage to access it with Julia. Is there something wrong in the URL format ?
Thanks for your reply !

using Downloads
WARNING: using Downloads.download in module Main conflicts with an existing identifier.

Downloads.download("sftp://user@xx.xx.xx.xx:22/home/user/file.csv")
ERROR: Error in the SSH layer while requesting user@xx.xx.xx.xx:22/home/user/file.csv
Stacktrace:
  [1] (::Downloads.var"#9#18"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Bool, String, Int64, Bool, Bool})(easy::Downloads.Curl.Easy)
    @ Downloads ~/Documents/logiciels/julia-1.7.1/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:369
  [2] with_handle(f::Downloads.var"#9#18"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Bool, String, Int64, Bool, Bool}, handle::Downloads.Curl.Easy)
    @ Downloads.Curl ~/Documents/logiciels/julia-1.7.1/share/julia/stdlib/v1.7/Downloads/src/Curl/Curl.jl:64
  [3] #8
2

libcurl is used and it’s sometimes not easy to find the problem.
Let us know what

Downloads.download("sftp://user@xx.xx.xx.xx:22/home/user/file.csv", verbose=true )

gives you back additionally.

3

Thank you ! there is a .netrc that I never modified…

julia> Downloads.download("user@xx.xx.xx.xx:22/home/fred/file.csv",verbose=true)
* Couldn't find host xx.xx.xx.xx in the .netrc file; using defaults
*   Trying xx.xx.xx.xx:22...
* Connected to xx.xx.xx.xx (xx.xx.xx.xx) port 22 (#0)
* Found host xx.xx.xx.xx in /home/user/.ssh/known_hosts
* Set "ssh-ed25519" as SSH hostkey type
* Closing connection 0
4

The .netrc is fine, but perhaps it is the hostkey type ssh-ed25519 which libcurl doesn’t like.
Check:

ssh-keyscan  xx.xx.xx.xx

It will give a list of keys available by the server.
If there is a ssh-rsa key you can force this one by removing the current entry of xx.xx.xx.xx in ~/.ssh/known_hosts and adding the ssh-rsa key.

I am only guessing, so no guaranty that we are even on the right path.

5

@oheil thank you for your advices !
It seems very complicated… I will continue with FileZilla :slight_smile:

if I remove the current entry of xx.xx.xx.xx in ~/.ssh/known_hosts

* Connected to xx.xx.xx.xx (xx.xx.xx.xx) port 22 (#0)
* Did not find host xx.xx.xx.xx in /home/user/.ssh/known_hosts
* Failure establishing ssh session: -5, Unable to exchange encryption keys
* Closing connection 0
ERROR: Failure establishing ssh session: -5, Unable to exchange encryption keys while requesting sftp://user@xx.xx.xx.xx:22/home/user/file.csv

if I add the key, the the previous error appears

* Couldn't find host xx.xx.xx.xx in the .netrc file; using defaults
*   Trying xx.xx.xx.xx:22...
* Connected to xx.xx.xx.xx (xx.xx.xx.xx) port 22 (#0)
* Found host xx.xx.xx.xx in /home/user/.ssh/known_hosts
* Set "ssh-ed25519" as SSH hostkey type
* Closing connection 0
ERROR: Error in the SSH layer while requesting sftp://user@xx.xx.xx.xx:22/home/user/file.csv
Stacktrace:
  [1] (::Downloads.var"#9#18"{IOStream, Base.DevNull, Nothing, Vector{Pair{String, String}}, Float64, Nothing, Bool, Bool, String, Int64, Bool, Bool})(easy::Downloads.Curl.Easy)
    @ Downloads ~/Documents/logiciels/julia-1.7.1/share/julia/stdlib/v1.7/Downloads/src/Downloads.jl:369

I think that the problem comes from my sshd_config set for RSA keys and not ed25519

PubkeyAuthentication yes
PubkeyAcceptedAlgorithms +ssh-rsa

it seems that Julia needs ed25519 keys…

6

Yes, it is complicated.
I have the same problem (at least exact same error) reproduced with one of my servers, but until now didn’t found a solution. Above was one of the steps I had to do until the “Error in the SSH layer…” error.
Something in libssh2 which comes with Julia, but not yet any hint what…

7

What I missed before, did you provide a password, like:

Downloads.download("sftp://user:password@xx.xx.xx.xx:22/home/user/file.csv")
8

@oheil, I tried with the password but still have Error in the SSH layer :slight_smile:

9

I think it’s a bug in ...\share\julia\stdlib\v1.7\Downloads\src\Curl\Easy.jl:

const PROTOCOL_STATUS = Dict{String,Function}(
    ...
    "rtsp"   => status_2xx_ok,
    "scp"    => status_zero_ok,
    "sftp"   => status_2xx_ok,
    "smtp"   => status_2xx_ok,
    ...
)

For "sftp" it should be status_zero_ok. (see libcurl example - sftpget.c) I will create an issue and perhaps a PR.

But there is more:

My first attempt with the host key of type ssh-ed25519 was right. According to libssh2 docs https://www.libssh2.org/ only

  • Hostkey Types : ssh-rsa, ssh-dss

are supported. So you have to force ssh-rsa or ssh-dss host keys using your ~/.ssh/known_hosts file. ssh-rsa is clearly preferred over dss.
Add an entry to ~/.ssh/known_hosts using

ssh xx.xx.xx.xx

If it’s not asking to add it it’s already there.
Use

ssh-keyscan  xx.xx.xx.xx

to receive an ssh-rsa key and edit ~/.ssh/known_hosts by removing the existing non-rsa key with the ssh-rsa key.

For me this was not enough.

My user name contains a @ because it’s of the form user@domain. And the password has another problematic character which was #:
User: xxx@domain.com
Pass: fsfsdf#
With this, I have to encode the problematic characters using html entities:
@ = %40
# = %23
With that the following URL works:

Downloads.download(
    "sftp://xxx%40domain.com:fsfsdf%23@xx.xx.xx.xx/path/file.csv",
    "./file.csv"
)

but the local file ./file.csv is deleted after successful download, because the return value of 0 is interpreted as an error state, which is the sftp specific bug.

You can overcome this until the bug is fixed with:

julia> using Downloads

julia> Downloads.Curl.PROTOCOL_STATUS["sftp"]=Downloads.Curl.status_zero_ok
status_zero_ok (generic function with 1 method)

julia> Downloads.download("sftp://user:pass@xx.xx.xx.xx/path/file.csv","./file.csv")
3 Likes
10

https://github.com/JuliaLang/Downloads.jl/pull/180

11

Here is some additional information on how to get debug feedback from the libssh2 library.

Building Julia from source is straight forward and works well (here I used a debian WSL2 in Windows 10)
Getting source via git from https://github.com/JuliaLang/julia#master which creates a new folder julia:

cd julia
export USE_BINARYBUILDER_LIBSSH2=0
export USE_BINARYBUILDER_CURL=0
make

Edit file ./deps/srccache/curl-7.73.0/lib/vssh/libssh2.c (the path may change over time with version change of curl) and uncomment the following line at the beginning so that it is

#define CURL_LIBSSH2_DEBUG

Edit file deps/libssh2.mk and change Release to Debug, so that it is:

LIBSSH2_OPTS := $(CMAKE_COMMON) -DBUILD_SHARED_LIBS=ON -DBUILD_EXAMPLES=OFF \
                -DCMAKE_BUILD_TYPE=Debug

Now build again:

rm -rf deps/scratch/curl*
rm -rf deps/scratch/libssh2*
make

The resulting julia binary will give plenty information output from libssh2 via the libssh2_trace() API.

12

wow ! very impressive debugging ! :wink:
I almost suceded after removing non-RSA keys :

Downloads.Curl.PROTOCOL_STATUS["sftp"]=Downloads.Curl.status_zero_ok
status_zero_ok (generic function with 1 method)

* Connected to xx.xx.xx.xx (xx.xx.xx.xx) port 22 (#0)
* Found host xx.xx.xx.xx in /home/user/.ssh/known_hosts
* Set "ssh-rsa" as SSH hostkey type
* Failure establishing ssh session: -5, Unable to exchange encryption keys
* Closing connection 0
13

If you have access to the server or to server admin you can ask or look, if the server supports one of the following:

* **Key Exchange Methods** : diffie-hellman-group1-sha1, diffie-hellman-group14-sha1, 
         diffie-hellman-group-exchange-sha1, 
         diffie-hellman-group-exchange-sha256
* **Ciphers** : aes256-ctr, aes192-ctr, aes128-ctr, 
         aes256-cbc (rijndael-cbc@lysator.liu.se), aes192-cbc, 
         aes128-cbc, 3des-cbc, blowfish-cbc, cast128-cbc, 
         arcfour, arcfour128, none

again from https://www.libssh2.org/
I am not sure if the error is about Key exchange method or the Ciphers.

14

Or you can try with a sftp client and option -v :

sftp -v sftp://user@xx.xx.xx.xx/

It prints also some usefull information about key exchage algorithms (kex) and more.
You can also force a cipher with option -c:

sftp -v -c aes256-ctr sftp://user@xx.xx.xx.xx/

gives for example:

...
debug1: kex: server->client cipher: aes256-ctr MAC: umac-64-etm@openssh.com compression: none
debug1: kex: client->server cipher: aes256-ctr MAC: umac-64-etm@openssh.com compression: none
d
...

Well, I admit, it’s tedious.

15

Thank you @oheil !
SFTP (and fileZila) password authentification works very well. Indeed key exchange is tried first and fail, this is why password authentification is finally selected. It seems that Julia download() function is much more picky…

debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: /home/user/.ssh/id_rsa RSA SHA256:XE...
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /home/user/.ssh/id_dsa
debug1: Trying private key: /home/user/.ssh/id_ecdsa
debug1: Trying private key: /home/user/.ssh/id_ecdsa_sk
debug1: Trying private key: /home/user/.ssh/id_ed25519
debug1: Trying private key: /home/user/.ssh/id_ed25519_sk
debug1: Trying private key: /home/user/.ssh/id_xmss
debug1: Next authentication method: password
user@xx.xx.xx.xx's password: 
debug1: Enabling compression at level 6.
Authenticated to xx.xx.xx.xx ([xx.xx.xx.xx]:22) using "password".
16

The authentication methods are tried until one succeeds or none is left.
It’s the same for Julias download as it is done by libssh2.

Your snippet is not the right part for your issue.
You need to look for host keys, kex and ciphers. Those tell you, which your client (or filezille) is able to choose for your server. Your client and the server are negotiating them.
If it is something, which libssh2 doesn’t support, it would be a hint on whats going on.

17

Fixed (most) of the issues you were having: Julia SFTP Client

2 Likes