# Best way to find the last saved file in a folder?

**URL:** https://discourse.julialang.org/t/best-way-to-find-the-last-saved-file-in-a-folder/99737
**Category:** New to Julia
**Tags:** question, filesystem
**Created:** [June 1, 2023, 5:39pm UTC](https://discourse.julialang.org/t/best-way-to-find-the-last-saved-file-in-a-folder/99737 "2023-06-01T17:39:39Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![phantom](https://avatars.discourse-cdn.com/v4/letter/p/e0b2c6/32.png) [@phantom](https://discourse.julialang.org/u/phantom)
#### Post date: [June 1, 2023, 5:39pm UTC](https://discourse.julialang.org/t/best-way-to-find-the-last-saved-file-in-a-folder/99737/1 "2023-06-01T17:39:40Z")

</div>

Hi!

I have a folder where new files are constantly being added and I would like to know the best way to find the name of the last file saved. Thus far I am doing something like

```julia
files = readdir(folder, join = true)
lastsvdfile = files[argmax(mtime.(files))]

```

But there are millions of files in the folder and it takes a bit of time each time it is called. Just wondering if there was a better way to go about it? I thought maybe if I set `sort =false`, `readdir` might return the files in a chronological list but the result is still alphabetical. Thanks so much!

---

<div class="post-metadata">

### Author: ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [June 1, 2023, 5:49pm UTC](https://discourse.julialang.org/t/best-way-to-find-the-last-saved-file-in-a-folder/99737/2 "2023-06-01T17:49:10Z")

</div>

You might find specific filesystems that support this kind of query more efficiently, but I doubt there is any portable way to do it besides querying the `mtime` of all the files individually as you are doing now.

You could avoid allocating so many temporary arrays. e.g. you could simply do

```julia
lastsvdfile = argmax(mtime, readdir(folder, join=true, sort=false))

```

which only allocates 1 array for the result of `readdir`. (See also [julia#27450](https://github.com/JuliaLang/julia/pull/27450) for discussiong of avoiding the allocation from `readdir`.) But I’m guessing that the time here is dominated by the filesystem accesses and not by the array allocations.

> [@phantom](#):
>
> But there are millions of files in the folder

If you have millions of files in a directory then maybe you shouldn’t be using so many individual files, rather than storing data in one big file in some format.

---

<div class="post-metadata">

### Author: ![phantom](https://avatars.discourse-cdn.com/v4/letter/p/e0b2c6/32.png) [@phantom](https://discourse.julialang.org/u/phantom)
#### Post date: [June 1, 2023, 6:02pm UTC](https://discourse.julialang.org/t/best-way-to-find-the-last-saved-file-in-a-folder/99737/3 "2023-06-01T18:02:47Z")

</div>

Thanks! I originally had a few larger files but I couldn’t figure out how to append incoming data to a partition of an existing arrow table without bringing the entire file into RAM. I’ll try rethinking the format a little better.

---

<div class="post-metadata">

### Author: ![stevengj](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stevengj/32/71_2.png) [@stevengj](https://discourse.julialang.org/u/stevengj)
#### Post date: [June 1, 2023, 6:05pm UTC](https://discourse.julialang.org/t/best-way-to-find-the-last-saved-file-in-a-folder/99737/4 "2023-06-01T18:05:33Z")

</div>

> [@phantom](#):
>
> I couldn’t figure out how to append incoming data to a partition of an existing arrow table without bringing the entire file into RAM.

HDF5 has more support for this kind of use-case. (In the low-level HDF5 API, you can make a dataset appendable. Presumably this is possible from the Julia HDF5.jl package too but I don’t see explicit mention of it in the documentation? You might have to do a little digging.)

---

<div class="post-metadata">

### Author: ![phantom](https://avatars.discourse-cdn.com/v4/letter/p/e0b2c6/32.png) [@phantom](https://discourse.julialang.org/u/phantom)
#### Post date: [June 1, 2023, 6:10pm UTC](https://discourse.julialang.org/t/best-way-to-find-the-last-saved-file-in-a-folder/99737/5 "2023-06-01T18:10:06Z")

</div>

awesome I’ll look into it. Thanks so much!
