# Lazy readdir in Julia with libuv

**URL:** https://discourse.julialang.org/t/lazy-readdir-in-julia-with-libuv/29522
**Category:** General Usage
**Created:** [October 5, 2019, 3:26pm UTC](https://discourse.julialang.org/t/lazy-readdir-in-julia-with-libuv/29522 "2019-10-05T15:26:06Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![xor0110](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xor0110/32/7926_2.png) [@xor0110](https://discourse.julialang.org/u/xor0110)
#### Post date: [October 5, 2019, 3:26pm UTC](https://discourse.julialang.org/t/lazy-readdir-in-julia-with-libuv/29522/1 "2019-10-05T15:26:06Z")

</div>

`readdir` relies on the libuv `scandir` function, meaning all the contents of a directory must be read to memory before we do anything. Sometimes it is useful to iterate over contents in a streaming way. Libuv allows this trough the `uv_fs_readdir` method. I’m trying to implement a lazy `readdir` using that, using `ccall`.

The following C code seems to do the job:

```julia
#include "uv.h"
#include "stdio.h"

uv_loop_t* loop;

uv_fs_t readdir_req;

const char* pth = "/home/user/Downloads";

int main() {

  uv_fs_opendir(NULL, &readdir_req, pth, NULL);

  uv_dirent_t dirent;
  uv_dir_t* rdir = readdir_req.ptr;
  rdir->dirents = &dirent;
  rdir->nentries = 1;

  while(uv_fs_readdir(NULL, &readdir_req, readdir_req.ptr, NULL))
    printf("%s\n", dirent.name);

  uv_fs_closedir(NULL, &readdir_req, readdir_req.ptr, NULL);

  return 0;
}

```

I basically need to find out how to reproduce it in Julia, if possible. What seems to be my problem right now is the assignment of the `dirents` pointer, where we need to allocate an array of `uv_dirent_t`.

The following Julia code is a draft, and doesn’t work yet. Where is the glaring mistake?

```julia
mutable struct uv_dirent_t
    name::Ptr{UInt8}
    typ::Cint
end

mutable struct uv_dir_t
    dirents::Ptr{uv_dirent_t}
    nentries::Cint
end

function lazyreaddir(dir::AbstractString)
    # Allocate space for uv_fs_t struct
    uv_readdir_req = zeros(UInt8, ccall(:jl_sizeof_uv_fs_t, Int32, ()))
    err = ccall(:uv_fs_opendir, Int32, (Ptr{Cvoid}, Ptr{UInt8}, Cstring, Ptr{Cvoid}),
                C_NULL, uv_readdir_req, dir, C_NULL)

    err < 0 && throw(SystemError("unable to read directory $dir", -err))

    thedirptr = ccall(:uv_fs_get_ptr, Ptr{uv_dir_t}, (Ptr{Cvoid},), uv_readdir_req)

    println("thedirptr ", thedirptr)

    thedir = unsafe_load(thedirptr)
    aa = Vector{uv_dirent_t}(undef, 10)
    # thedir.dirents = Libc.malloc(10240)
    thedir.dirents = pointer(aa)
    thedir.nentries = 1

    for a in 1:1
        read = ccall(:uv_fs_readdir, Cint,
                     (Ptr{Cvoid}, Ptr{Cvoid}, Ptr{uv_dir_t}, Ptr{Cvoid}),
                     C_NULL, uv_readdir_req, thedirptr, C_NULL)

        println("read $read")
        println(aa[1].name)
    end

    println("---over")
    ccall(:uv_fs_closedir, Int32, (Ptr{Cvoid}, Ptr{UInt8}, Cstring, Ptr{Cvoid}),
                C_NULL, uv_readdir_req, thedirptr, C_NULL)
end

lazyreaddir("/home/user/Downloads")

```

---

<div class="post-metadata">

### Author: ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [October 5, 2019, 8:13pm UTC](https://discourse.julialang.org/t/lazy-readdir-in-julia-with-libuv/29522/2 "2019-10-05T20:13:12Z")

</div>

Related: [https://github.com/JuliaLang/julia/pull/27450](https://github.com/JuliaLang/julia/pull/27450). The decision there basically came down to: when are you reading a directory with some many items that doing this lazily matters? Do you have such a situation?

---

<div class="post-metadata">

### Author: ![xor0110](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xor0110/32/7926_2.png) [@xor0110](https://discourse.julialang.org/u/xor0110)
#### Post date: [October 6, 2019, 9:45am UTC](https://discourse.julialang.org/t/lazy-readdir-in-julia-with-libuv/29522/3 "2019-10-06T09:45:19Z")

</div>

Thanks for pointing that out! I ended up making a PR before I saw this: [https://github.com/JuliaLang/julia/pull/33478](https://github.com/JuliaLang/julia/pull/33478)

Of course having 10 megafiles in a directory is probably not a good thing. It can happen as an accident to people sometimes, though, with me more than once. And when it happens things can get weird: you can’t `ls` anymore because it wants to sort everything, you need `ls -U`, etc… I think offering this makes Julia a more useful and trustworthy tool for dealing with the file system. A better justification is simply that the standard readdir implies a memory cost and a sorting overhead that this lazy approach avoids, so it should make operations such as `rm -r` generally leaner. I don’t have any numbers to show how much of a difference it can make, though, and I guess a good benchmark will probably always involve unreasonable things such as directories with many millions of files.