Designing a Paths Julep

[kkretschmer]

Recently there was an LWN news item pointing to an article on the Go blog about Traversal-resistant file APIs.

[savq]

That’s a nice API. I’d also be nicer in Julia since we could use a do block instead of defer.

[sairus7]

My two cents here: if Paths API has some kind of abstract interface, then maybe it can be used consistently across packages like Tar.jl, ZipArchives.jl, ZipFile.jl (maybe with some constraints)?

[tecosaur]

@kkretschmer

Recently there was an LWN news item pointing to an article on the Go blog about Traversal-resistant file APIs.

Thanks for the link! Incidentally this touches on one of the tensions in the current design: my desire to break with select conventions in the name of consistency/safety.

For instance, currently p"/some/absolute/path" * p"../../../../../../../../etc/passwd" will raise an error once we try to get the parent directory of p"/", however the “standard” behavior is for the parent of / to be / .

I particularly enjoyed reading this section:

If the attacker controls part of the local filesystem, they may be able to use symbolic links to cause a program to access the wrong file:

// Attacker links /home/user/.config to /home/otheruser/.config:
err := os.WriteFile("/home/user/.config/foo", config, 0o666)

If the program defends against symlink traversal by first verifying that the intended file does not contain any symlinks, it may still be vulnerable to time-of-check/time-of-use (TOCTOU) races, where the attacker creates a symlink after the program’s check:

Since this is exactly the issue that is systematically eliminated by requiring the use of a FD-as-Path type, as I am now experimenting with (and based on encouraging results, advocating for).

I don’t think it would be too hard to extend the proposal slightly and have rooted-ness be an extra flag that the FD-as-Path type can have, which is then automatically propagated to all path joining operations.

From reading that article, with this proposal will give us a better story with this class of vulnerabilities than Go

@savq

That’s a nice API. I’d also be nicer in Julia since we could use a do block instead of defer.

RAII would be nice here, but in the current experiment I’m just using a finaliser. I’m not sure that do is really what we want, since it’s good if the same handle is passed around more instead of being dropped and re-acquired.

[tecosaur]

@sairus7

My two cents here: if Paths API has some kind of abstract interface, then maybe it can be used consistently across packages like Tar.jl, ZipArchives.jl, ZipFile.jl (maybe with some constraints)?

Sure, why not How does julia-basic-paths/abstractpaths.jl at main - tec/julia-basic-paths - Code by TEC look to you?

[tecosaur]

@tecosaur

It’s incomplete, and among other things the naming has to be worked out (do we really want to call a handle to a resource a Path? I’m not sure but FilesystemEntityHandle is way too awkward and I haven’t had any naming brainwaves).

It’s slightly imprecise, but I’m thinking that maybe PathHandle is the name to go with here. I think having “handle” in the name is important, and while FSHandle etc. are most accurate, by putting Path in the name we get an explicit connection to the Path type and an indication of the relationship between the two. Appearing as a completion with Path<tab> is a nice bonus. I think together this more than makes up for the loss in accuracy.

So, with this, there are three kinds of system-native paths:

• Path as a “pure” path for the current system (an alias of PosixPath or WindowsPath)
• PathHandle for a file descriptor/filesystem handle produced from a Path
• DirEntry is a parent directory PathHandle combined with a relative Path and (optional) metadata.

All of these types can be converted between (e.g. PathHandle to a Path, DirEntry to a PathHandle, etc.) using the platform filesystem APIs. We can do this automatically where convenient, and require a particular type when we want to prod the user into behaving more safely (e.g. requiring them to acquire a handle, to encourage them to re-use it).

[goerz]

Since @tecosaur explicitly requested further commenting in

The strangeness (or not) of * as string concatenation

If I could direct you over to that thread, I’d love to hear more. At this stage, I think I can say that thread will eventually become a something, and I’d like that something to be of high enough quality to be worth serious consideration for inclusion in Julia proper (which is somewhat needed for the value of a Path type to be realised, given open, read, etc.). There are small pile of design compromises that need to be made. More shared thoughts on what the right priorities, trade-offs, and ot…

I already expressed most of my views during the original discussion, in Designing a Paths Julep - #58 by goerz, but in the context of the point I was trying to make in the context of the “The strangeness (or not) of * as string concatenation” thread, the worry I took away from the discussion here is pretty much summed up with

@adienes

the counterpoint being I realllllly think it would be a shame if Julia chose once again to be a special flower and reject overwhelmingly consistent & near universal precedent for the sake of pseudo-mathematical ideological “purity”

One aspect of such “purity” is @jar1’s (and others’) stance

@jar1

Most simply: Base./ is defined to be division. I don’t want to use it for something unrelated.

It’s not that I don’t understand this at some level, and it’s hard to argue about it completely objectively. But the experience of every other programming language that has Path objects has shown that there is no practical problem with using / for something completely unrelated to division, in the context of paths. These functionalities do not clash, in practice.

But beyond the purity of “division”:

@goerz

If the argument is “* is the only appropriate operator for string concatenation, because of abstract string algebra in theoretical computer science”, then I’m starting to have serious problems. Because then we’re in the territory of “I need to understand abstract algebra concepts to work with Julia / Julia is only for PhDs”, which is not a good place to be in.

And if you then go further and say "because we have *, abstract algebra demands that we also have / (despite there not being a multiplicative inverse for strings), then I think we’ve totally lost the plot, and we end up with "Oh, then we must also have the same algebra with * and / for Path objects, and then p"folder/sub" / p"sub" ends up meaning the exact opposite of what it means in literally every other programming language that has Path objects.

I’d also want to strongly re-emphasize

@DNF

But since a path is a kind of string (or closely related), it’s very unfortunate to use the string concatenation operator for something that’s very different from concatenation. Especially since concatenation is something you probably want to do.

IMO, * is the worst possible operator for path join, better to use something completely different.

But enough about the joinpath operator

The other point @tecosaur was bringing up was

@tecosaur

@goerz

In Python’s pathlib, Path("user_content/" + untrusted_file) is simply PosixPath('user_content/../../../../../../etc/passwd'), which seems perfectly alright.

I’d describe this as perfectly dangerous not alright. There’s a pile of CVEs across all sorts of software and libraries from exactly this behavior.

It’s dangerous because there’s untrusted input being used. People have to be aware of untrusted input, and sanitize it before letting it flow through the rest of their program. But that’s not your responsibility. Or rather, you’re in no position to “fix” this in a path library, and I’d be concerned that any well-meaning attempt to fix it is only going to mess up perfectly legitimate use cases. Stick to the solutions that have been found to work in other ecosystems, such as Python’s pathlib.

A Path('user_content/../../../../../../etc/passwd') is absolutely something I might want to do, and yes, it should resolve to /etc/passwd. The way to sanitize Path("user_content/" + untrusted_file) is to resolve it and check that the result is “secure”. Maybe “secure” means it’s in a subfolder of user_content, or maybe that it’s in a subfolder of the current user’s home directory. The point is, you can’t know, so how could you possibly “fix” this? The CVEs are for libraries improperly using untrusted input, not for the Path library. the Path library isn’t the problem here, and it’s not the place to fix anything.

I would be strongly advise against changing the behavior of existing solutions for things like relative_path / absolute_path -> absolute_path. Even the existing string-based joinpath in Julia implements that behavior. Other pathlib implementation adopt this for good reason. Yes, it’s potentially dangerous with unsanitized input, but it’s important for common practical use cases, such as join_path(cwd(), location) not having to distinguish whether location is relative or absolute.

Basically,

@davidanthoff

I think I would try to keep this very simple

As for some of the “ideas” for design goals floating around,

@jules

• rejecting invalid path segments at creation
• disallowing a root segment within a path’s segments
• disallowing joining a path with an absolute one
• clear handling of special . and .. segments

No! Don’t do any of these things! Don’t fix it if it ain’t broken. All we need is a Path object that understands the “segments” that make up a file path, and functions like parent, name, suffix, stem, relative_to, etc. to manipulate them.

Don’t get in the way of constructing “weird” paths, like “rejecting invalid path segments”, or applying any kind of normalization prematurely.

Then on top of that, you can normalize paths to deal with “handling of special . and .. segments”. At that point, you still don’t want to access the actual file system: it should be possible to generate normalized Unix paths on a Windows system and vice versa.

Then lastly, you resolve paths on the actual file system, and that’s where things like symlinks come into play. (I’m not sure whether the name resolve is appropriate at the normalization stage or at the resolve-on-filesystem stage; again: look at the terminology used in existing implementations). At that level, maybe you also want to add some functionality to verify that certain paths are “secure”, but I’d be pretty careful with that.