Download with password (digression)

Internals & Design
,
1

I took a look at the docs and (passwords and) .netrc is undocumented (maybe for good reasons… is it clearly supported, I would want its support removed, i.e. curl, is it claimed to be supported, or just an implementation detail?).

I’m a bit conflicted, what to recommend doing here, about the pros and cons of a) and b), both seem to have very bad cons:

a) If you put the password in the URL yourself, as you can (supported by the standard and all major web browsers, but more importantly all web servers I guess, and I would think also problematic there, would web servers log my password?), then you have two problems; you need still to get the password from somewhere, and hardcoding it in your code is bad practice, and even accessing it from some (standard) string could be a security issue, I think, plus now I think it’s your responsibility to put https in the URL, and you cuold put http in, but that’s a huge security risk.

b) Use .netrc, which avoids putting the password in your code (hardcoded or some other way); or even into the address space? Seems like a good thing, except it’s undocumented, and likely should stay that way and even get unsupported eventually.

The (current) top rated answer there (and the title of the question) uses http, which is a security risk, and even using https is problematic, from the other answer:

https://user:password@domain.com/

Note that you must urlencode special characters in the user or password fields (I frequently use ‘@’ in my passwords, so those must be written as ‘%40’).

I’m NOT suggesting Julia would do that for you, rather some (proposed name for module) SecureDownloads package.

Why not simply the missing functionality Downloads.jl? Because I want it away in Julia 2.0, and having the security in a different package can be an incentive to use it, and ween people of the standard package. I actively DO NOT want Julia to have security/crypto stuff in it. It means Julia must be updated when crypto needs to be updated, which is often, and I think a better way needs to be implemented (that new package will need to be somehow self-updating and/or depend on the OS crypto). That new package could I guess though be very light, adding just what’s needed, depending on Downloads (for now).

You ideally want to store NO passwords anywhere on your machine, so the package should by default work for some modern replacement, ssh keys (and/or 2FA? and maybe passwords too, not sure, maybe as non-default method clearly pointing to the better alternatives).

Hypothetically we could document .netrc, since it works; or does it? I’m not sure about for Windows, and even for Linux, some claim it’s a security risk (only if someone can read your file system, then you have a bigger issue, but I’m not sure we should encourage it if it’s a bad practice):

.netrc only works since we ship with curl (which I would like to trim from Julia), that supports Linux, and Windows I see, but differently for this (and I really would want some simple instructions that work identically everywhere):

The .netrc file is typically stored in a user’s home directory. (On Windows, curl will look for it with the name _netrc ).

There is code that makes .netrc in a test, I’m not sure it’s fully secure, but could be a base for someone making a new package (and generalized for Windows):

Googling Julia docs for password, gets you false alarms (but should point to new package), for netrc, no results, and for “.netrc” gets you a long list however, seemingly all false alarms.

2

The OP wanted a direct replacement for the Matlab download function, which also embeds the password directly in the code as plaintext. Putting it into the https url is the direct equivalent of this.

By all means, use a better authentication method such as ssh keys if possible. But let’s also answer the question asked.

2 Likes
3

Also, putting the password in the URL isn’t particularly insecure as long as the request goes over HTTPS. The HTTP protocol doesn’t really support any better methods of authentication natively, so that’s about as good it gets for HTTP auth.

1 Like
4

The best I’ve come up with, almost drop-in replacement:

using Downloads: download
using HTTP

function urlread(url, username, password="")
    password = HTTP.escapeuri(password == "" : get(ENV, "PASSWORD", readline()))
    download("https://$username:$password@" * replace(url, r"^.*://"=>""))
end

Still consider doing things differently (see my other comment, and NOT put this in Julia’s standard library), this is at least strictly better than what’s currently being suggested, a) and b).

I can’t control where the file lands, it’s in /tmp at least on Linux (likely different on Windows), it could be moved to the current directory, but not something I wanted to think about or conform to MATLAB convention.

If you only need this interactively then I read the password in (currently without a prompt) as a last resort. Better is the default from env variable, so you could start your Julia program without hard-coding the password (or rely on the password question):

$ PASSWORD=my_password julia my_program.jl

Note, this will just rather putt the password in your shell bash history, rather than in Julia’s REPL history… but you can also put the password elsewhere e.g. in .bashrc or whatever on Windows. It’s not at least worse then .netrc, since this is cross-platform.

I intentionally avoided HTTP, since that’s always a security risk, and almost always HTTPS is set up on web servers (at least should be for all password-needing setup).

I believe an even better option, is HTTP GET (or POST?) where I believe the password doesn’t have to be in the URL, but I didn’t see how it’s done. At least the same API could be used (urlread isn’t the best name though).

In the past I bypassed using Julia for this, rather using wget, just before running my Julia program, both run from the same shell script, also because of recursive traversal (for sFTP).

https://www.rfc-editor.org/rfc/rfc3986

Use of the format “user:password” in the userinfo field is
deprecated. Applications should not render as clear text any data
after the first colon (“:”) character

5

HTTPS requests also use the HTTP protocol, which is the issue when it comes to authenticating.

When the entire transaction is encrypted it doesn’t matter if the password is in the URL or the body, so this is not an improvement. I would recommend against trying to roll your own password protocol.

6

The code I sent (or any adding password to the URL) will likely work, for now, but there’s seemingly no guarantee (seems the updated HTTP/1.1 standard can ignore it), nor something you want to do, even for HTTPS (see below): RFC 7230 - Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing

Userinfo (i.e., username and password) are now disallowed in HTTP and HTTPS URIs, because of security issues related to their transmission on the wire.

And: RFC 3986 - Uniform Resource Identifier (URI): Generic Syntax

URI producers should not provide a URI that contains a username or password that is intended to be secret. URIs are frequently […] and logged by […] intermediary applications (proxies). A password appearing within the userinfo component is deprecated and should be considered an error (or simply ignored)

Yes of course, my suggestion for HTTP GET wasn’t to invent something new, rather what I (mistakenly?) though was an already existing protocol (this one, Basic Auth, seems you want to support, at a minimum, digest auth is also cool, but most effort should go into supporting non-password based auth): Authorization - HTTP | MDN

FYI: You have a Python package handling this, and more auth, e.g. (OAuth 1 and) “OAuth 2 and OpenID Connect Authentication” (likely what we want supported; plus it has Kerberos and NTLM)
https://requests.readthedocs.io/en/latest/user/authentication/#oauth-2-and-openid-connect-authentication

The netrc file overrides raw HTTP authentication headers set with headers=

that you can use with PythonCall.jl (or PyCall.jl), or to learn from, there’s also httr for R, you could actually also use. It also supports uploading files, and I guess with (or without) a password, the code is almost symmetric, reuses most of the download code, so hopefully a Julia package could support that soon (it also has nice features, like a progress bar, likely for both, that we would also want to support).