# Designing a Paths Julep

**URL:** <https://discourse.julialang.org/t/designing-a-paths-julep/124335>\
**Category:** Internals & Design\
**Tags:** proposal, filesystem, path, rfc\
**Created:** [January 1, 2025, 12:37pm UTC](https://discourse.julialang.org/t/designing-a-paths-julep/124335 "2025-01-01T12:37:25Z")\
**Posts on this page:** 20\
**Page:** 7

<div class="post-metadata">

**Author:** ![tecosaur](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tecosaur/32/23206_2.png) [@tecosaur](https://discourse.julialang.org/u/tecosaur)\
**Post date:** [January 15, 2025, 12:10pm UTC](https://discourse.julialang.org/t/designing-a-paths-julep/124335/122 "2025-01-15T12:10:30Z")

</div>

> [@nsajko](#):
>
> What do you mean? Can you propose an alternative?

At this point it’s probably worth pointing out that FilePaths2.jl was implemented as an `AbstractTree`.

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [January 15, 2025, 1:45pm UTC](https://discourse.julialang.org/t/designing-a-paths-julep/124335/123 "2025-01-15T13:45:14Z")

</div>

> [@tecosaur](#):
>
> Since there’s a `Path(::Vector{String})` constructor, and `AbstractPath`s are iterable, for whole-path replacements, you could also do:
> 
> ```julia
> 
> ```

Is paths iterating segments a good idea? I don’t really see how that would be useful and it seems to open up bugs like scalars being iterable does.

Even in your example, if the users homedir has foo in it, I don’t want to replace that. I don’t really see when I’d want to do something per segment like that.

---

<div class="post-metadata">

**Author:** ![tecosaur](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tecosaur/32/23206_2.png) [@tecosaur](https://discourse.julialang.org/u/tecosaur)\
**Post date:** [January 15, 2025, 3:18pm UTC](https://discourse.julialang.org/t/designing-a-paths-julep/124335/124 "2025-01-15T15:18:55Z")

</div>

> [@ericphanson](#):
>
> Is paths iterating segments a good idea? I don’t really see how that would be useful and it seems to open up bugs like scalars being iterable does.

I don’t expect that example to be that practically useful, it’s just to demonstrate that if you want to change things other than the `basename` as strings, there’s a mechanism to do so.

With iterating paths, I can’t see much of an issue myself. You want a way to get all the segments of a path, and to me `collect(::Path)` seems at leas as good as `splitpath(::String)`. Given that `Strings`-as-paths are already iterable, and I haven’t seen any issues as a result, it’s hard for me to see the issue, even hypothetically. Perhaps you could elaborate?

---

<div class="post-metadata">

**Author:** ![ericphanson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ericphanson/32/215186_2.png) [@ericphanson](https://discourse.julialang.org/u/ericphanson)\
**Post date:** [January 15, 2025, 3:27pm UTC](https://discourse.julialang.org/t/designing-a-paths-julep/124335/125 "2025-01-15T15:27:43Z")

</div>

If you iterate a string, you get a character, which will be more likely to throw an error, e.g. if you feed it to something typed `AbstractString` it will throw. If you iterate a path and get a string or something string-like, you might be able to continue to use it as a relative path or similar and not realize the error (because a lot of code will be supporting both strings-as-paths and the new paths).

---

<div class="post-metadata">

**Author:** ![tecosaur](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tecosaur/32/23206_2.png) [@tecosaur](https://discourse.julialang.org/u/tecosaur)\
**Post date:** [January 15, 2025, 4:17pm UTC](https://discourse.julialang.org/t/designing-a-paths-julep/124335/126 "2025-01-15T16:17:18Z")

</div>

> [@ericphanson](#):
>
> If you iterate a string, you get a character, which will be more likely to throw an error, e.g. if you feed it to something typed `AbstractString` it will throw.

Ah yep, thanks for spelling that out. I can’t quite say I’m sold, since it does still appear to me that “the segments of the path” is the clear sensible answer to “what is a path composed of?”, but it’s something to mull on as the proposal is tweaked.

---

<div class="post-metadata">

**Author:** ![adienes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/adienes/32/37459_2.png) [@adienes](https://discourse.julialang.org/u/adienes)\
**Post date:** [January 15, 2025, 4:55pm UTC](https://discourse.julialang.org/t/designing-a-paths-julep/124335/127 "2025-01-15T16:55:11Z")

</div>

I think Julia’s path object should be as unoriginal as possible and use all of the most standard conventions found in the most popular libraries.

and that being said, it seems like C++, Java, and Rust all iterate their paths via segments so copying that would be fine for Julia imo. Python’s `pathlib` does not, but it just errors so at least it’s not really a competing convention.

---

<div class="post-metadata">

**Author:** ![tecosaur](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tecosaur/32/23206_2.png) [@tecosaur](https://discourse.julialang.org/u/tecosaur)\
**Post date:** [January 17, 2025, 7:18pm UTC](https://discourse.julialang.org/t/designing-a-paths-julep/124335/128 "2025-01-17T19:18:37Z")

</div>

Update here: I’ve had a first pass at implementing this, and I’m really encouraged by it.

> **[Initial mapreducepath implementation · 5dc526d23f](https://code.tecosaur.net/tec/julia-basic-paths/commit/5dc526d23ff3f070fc80d7daea208dd7e5c3001d)**
>
> julia-basic-paths

With these changes, you can now:

- count all entries under a path with `mapreducepath(_ -> 1, +, path)`
- count directories with `mapreducepath(_ -> 1, _ -> 0, +, path)`
- find the largest file with `mapreducepath(filesize, max, path)`
- collect all paths with `mapreducepath(identity, identity, vcat, collect, path)`
- check if any non-empty files exist under a path with `mapreducepath(_ -> false, !iszero ∘ filesize, |, any, path)`

I still need to add a writeup to the Julep, but I’m excited by this addition. It’s like a really nice `walkdir` (and also ~10% faster in the current implementation).

---

<div class="post-metadata">

**Author:** ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Post date:** [January 17, 2025, 7:26pm UTC](https://discourse.julialang.org/t/designing-a-paths-julep/124335/129 "2025-01-17T19:26:35Z")

</div>

Just in terms of process: could you turn this into a regular package, people can try and play with it, and if after some real-world testing and use everyone likes it, then it could move to base?

---

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [January 17, 2025, 8:40pm UTC](https://discourse.julialang.org/t/designing-a-paths-julep/124335/130 "2025-01-17T20:40:25Z")

</div>

I don’t think `AbstractPath` should be iterable because

- it will make scalar-extension broadcasting very confusing: `with_ext.(path, extensions)` should be `[with_ext(path, e) for e in extensions]`, not `[with_ext(frag, ext) for (frag, ext) in zip(fragments(path), extensions)]`. You can get around this by special-casing `AbstractPath` for broadcasting like `AbstractString` does, but this already causes problems and imo is very much not worth it.
- having a function `components(path)` for that purpose is simple, easy, and clear.

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [January 18, 2025, 6:35am UTC](https://discourse.julialang.org/t/designing-a-paths-julep/124335/131 "2025-01-18T06:35:34Z")

</div>

Yeah the broadcasting argument convinces me. And the function `components` or `segments` or so can still return a lazy iterator of Substrings or something else that doesn’t need to allocate

---

<div class="post-metadata">

**Author:** ![tecosaur](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tecosaur/32/23206_2.png) [@tecosaur](https://discourse.julialang.org/u/tecosaur)\
**Post date:** [January 18, 2025, 8:20am UTC](https://discourse.julialang.org/t/designing-a-paths-julep/124335/132 "2025-01-18T08:20:34Z")

</div>

Yea, I find the broadcasting argument rather convincing.

One other thought: is it possible for to be iterable but broadcast as a scalar?

---

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [January 18, 2025, 9:28am UTC](https://discourse.julialang.org/t/designing-a-paths-julep/124335/133 "2025-01-18T09:28:17Z")

</div>

> [@tecosaur](#):
>
> is it possible for to be iterable but broadcast as a scalar?

Yes that’s what I meant by my allusion to `AbstractString`: `String`s are iterable but broadcast as scalar. However imo this behavior makes it harder to learn, increases complexity of usage, and creates opportunity for error.

For example, broadcasting over a vector of characters versus a string produces quite different results:

```julia
julia> ['a', 'b', 'c'] .^ 2 |> join
"aabbcc"

julia> "abc" .^ 2 |> join
"abcabc"

```

I’d rather use a little helper function like `segments` than have to worry about complexities every time.

---

<div class="post-metadata">

**Author:** ![cshen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cshen/32/217287_2.png) [@cshen](https://discourse.julialang.org/u/cshen)\
**Post date:** [January 18, 2025, 9:42am UTC](https://discourse.julialang.org/t/designing-a-paths-julep/124335/134 "2025-01-18T09:42:16Z")

</div>

To throw my 2 cents in, I think that iterating the components of a path is a rather niche operation. You rarely need to do operations on sequential levels of a path, instead it is much more common to act at the same level. i.e. rename all files in a folder, rename all folders with a new patterns, swap the root of the file system, etc…

So, at least to me, it’s is much more intuitive an “horizontal” iteration, rather than a “vertical” one. And by “horizontal” I mean something like a glob pattern broadcasting:

```julia
for file in p"/path/to/some/*.txt"
    #= ... do stuff =#
end

```

Not really proposing this kind of iteration, it is just as an example.

---

<div class="post-metadata">

**Author:** ![DNF](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dnf/32/10191_2.png) [@DNF](https://discourse.julialang.org/u/DNF)\
**Post date:** [January 18, 2025, 9:48am UTC](https://discourse.julialang.org/t/designing-a-paths-julep/124335/135 "2025-01-18T09:48:03Z")

</div>

> [@cshen](#):
>
> ```julia
> for file in p"/path/to/some/*.txt"
> #= ... do stuff =#
> end
> 
> ```

But that could also be

```julia
for file in eachfile(p"/path/to/some/*.txt") 
    #= ... do stuff =#
end

```

or something like that. Etc. etc.

---

<div class="post-metadata">

**Author:** ![cshen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cshen/32/217287_2.png) [@cshen](https://discourse.julialang.org/u/cshen)\
**Post date:** [January 18, 2025, 9:51am UTC](https://discourse.julialang.org/t/designing-a-paths-julep/124335/136 "2025-01-18T09:51:42Z")

</div>

yes absolutely, I’ve edited the post specifying I am not actually pushing that syntax.  
A specialized function would absolutely be the best approach there.

I was just pointing out why I think that iterating a path components as an implicit default is not a great idea.

---

<div class="post-metadata">

**Author:** ![tecosaur](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tecosaur/32/23206_2.png) [@tecosaur](https://discourse.julialang.org/u/tecosaur)\
**Post date:** [January 18, 2025, 12:48pm UTC](https://discourse.julialang.org/t/designing-a-paths-julep/124335/137 "2025-01-18T12:48:02Z")

</div>

I’ve been developing this as a package just so I could use Revise.jl, I’m happy to make it easier for others to load it as a package too. [Make loadable as a package · 2a15632626 - tec/julia-basic-paths - Code by TEC](https://code.tecosaur.net/tec/julia-basic-paths/commit/2a15632626972dafd2d75d4f4038cc6c15647ac0) might do the trick.

---

<div class="post-metadata">

**Author:** ![jlapeyre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jlapeyre/32/4514_2.png) [@jlapeyre](https://discourse.julialang.org/u/jlapeyre)\
**Post date:** [January 18, 2025, 5:50pm UTC](https://discourse.julialang.org/t/designing-a-paths-julep/124335/138 "2025-01-18T17:50:23Z")

</div>

> [@jakobnissen](#):
>
> Do we add lots of string operations to paths, or do we ensure that one can easily and efficiently construct one type from the other?

Do we want string operations because they are familiar to people? Or in order to support calls to code for ducktyping? (People tend to annotate function parameters that take strings as `String`. But requesting that the annotation be removed is not too much of an imposition.)

I wonder how adoption will play out. Most people will choose the most compatible option, the one that will generate the fewest issues and complaints. Even if the ecosystem is 90% compatible with Paths, one incompatibility in my dependency tree will cause headaches. Maybe there’s some way to deal with the problem. But I’ve got a ton of issues to deal with. So I always send strings when I make a call that wants a path. Only a small fraction of users use Paths, but I want to see Paths succeed. So I accept Paths; if it’s a high enough priority on my list of things to do.

The story is the same at 89% adoption, except a little worse. You want people to add Path support that no one will use until some indeterminate future date.

A counterexample is if Paths is powerful and convenient, and I do a lot of work with paths in a package. I accept Strings and Paths, work internally with Paths, and send Strings.

---

<div class="post-metadata">

**Author:** ![tecosaur](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tecosaur/32/23206_2.png) [@tecosaur](https://discourse.julialang.org/u/tecosaur)\
**Post date:** [February 14, 2025, 9:15am UTC](https://discourse.julialang.org/t/designing-a-paths-julep/124335/139 "2025-02-14T09:15:50Z")

</div>

I’ve (mentally) put this on the backburner for the past few weeks, but this isn’t forgotten about.

- I’m still trying to be bullish about path normalisation, but also practical: I think it’s worth doing to the maximal reasonable extent, working out what exactly that extent is is requiring more thinking.
- I’ve realised that there are even more headaches relating to symlinks. They truly are the gift that keeps on giving.
- @Sukera kindly made me aware of an [IBM research paper](https://dants.github.io/papers/PathRes08TR.pdf) on the Unix path syscalls encouraging [TOCTTOU](https://en.wikipedia.org/wiki/Time-of-check_to_time-of-use) bugs/vunerabilities. It seems like as of POSIX 2008 we can (on non-Windows at least) implement a File Descriptor -first filesystem API that encourages non-TOCTTOU code. Perhaps the introduction of a dedicated `Path` is a good opportunity to smuggle in this too?

---

<div class="post-metadata">

**Author:** ![tecosaur](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tecosaur/32/23206_2.png) [@tecosaur](https://discourse.julialang.org/u/tecosaur)\
**Post date:** [February 16, 2025, 3:57pm UTC](https://discourse.julialang.org/t/designing-a-paths-julep/124335/140 "2025-02-16T15:57:04Z")

</div>

> [@tecosaur](#):
>
> Sukera kindly made me aware of an IBM research paper on the Unix path syscalls encouraging TOCTTOU bugs/vunerabilities. It seems like as of POSIX 2008 we can (on non-Windows at least) implement a File Descriptor -first filesystem API that encourages non-TOCTTOU code. Perhaps the introduction of a dedicated Path is a good opportunity to smuggle in this too?

I’ve just pushed [Experiment with making Path FD-oriented](https://code.tecosaur.net/tec/julia-basic-paths/commit/83d2dbb88d65c4df2a14c1b6c175c1032364c0d8#diff-3a2b5c1abad1fdab6c23b5b75ceee9a45a9604cf), I’ll be mulling over this and updating the design doc with some motivating thoughts and extra considerations over the next week or two.

I have yet to investigate how cross-platform this can be.

---

<div class="post-metadata">

**Author:** ![tecosaur](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tecosaur/32/23206_2.png) [@tecosaur](https://discourse.julialang.org/u/tecosaur)\
**Post date:** [March 13, 2025, 2:59am UTC](https://discourse.julialang.org/t/designing-a-paths-julep/124335/141 "2025-03-13T02:59:45Z")

</div>

I’ve started writing about this here: [Charting a Path for Julia: Recognising the difference between locations and resources](https://hackmd.io/@tecosaur/SyMYFLhG1g#Recognising-the-difference-between-locations-and-resources).

It’s incomplete, and among other things the naming has to be worked out (do we really want to call a handle to a resource a `Path`? I’m not sure but `FilesystemEntityHandle` is way too awkward and I haven’t had any naming brainwaves).

At this stage, I feel good about how useful having such a construct is, as it systematically makes it much harder for subtle bugs/vulnerabilities to occur.

[Previous page](https://discourse.julialang.org/t/designing-a-paths-julep/124335.md?page=6)

[Next page](https://discourse.julialang.org/t/designing-a-paths-julep/124335.md?page=8)