# Custom dataset

**URL:** https://discourse.julialang.org/t/custom-dataset/41329
**Category:** Data
**Tags:** question, images, flux
**Created:** [June 13, 2020, 3:37pm UTC](https://discourse.julialang.org/t/custom-dataset/41329 "2020-06-13T15:37:52Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![SubhadityaMukherjee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/subhadityamukherjee/32/15983_2.png) [@SubhadityaMukherjee](https://discourse.julialang.org/u/SubhadityaMukherjee)
#### Post date: [June 13, 2020, 3:37pm UTC](https://discourse.julialang.org/t/custom-dataset/41329/1 "2020-06-13T15:37:52Z")

</div>

Hello! I am new here.  
I have been trying to use Metalhead.jl/Flux.jl to train custom image classifiers but I cant seem to be able to load data from a folder directly. (Not part of the standard datasets).  
Say I have a folder with 10 sub folders → each of them with a 1000 images.  
How can I load this up to perform a simple classification task?

Thank you!  
Great to see such an active community 🙂

---

<div class="post-metadata">

### Author: ![dmolina](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dmolina/32/5246_2.png) [@dmolina](https://discourse.julialang.org/u/dmolina)
#### Post date: [June 13, 2020, 9:11pm UTC](https://discourse.julialang.org/t/custom-dataset/41329/2 "2020-06-13T21:11:15Z")

</div>

Welcome, @SubhadityaMukherjee!  
It is great that you want to use Flux.

There is true that there is not a DataLoader able to read from folder directly.  
I found the same problem, so I have to implement it myself, I am going to give you the code to help you:

```julia
using Metalhead
using FileIO
using Images
using Serialization
using Random

function absolute(dir::AbstractString)
    return expanduser(abspath(dir))
end
"""
    read_resnet(dir::AbstractString, output::AbstractString)

Read the images in the directory dir, apply the resnet and save the results
to an new directory diroutput.

- dir should have the structure partitionX/test/<category>.
- diroutput is a new directory that store the resnet structure, with
the structure partitionX/test, partitionX/train in which the results are serialized
as the pair matrix, category.
"""
function apply_model(apply_model::Function, dir::AbstractString, diroutput::AbstractString)
    partitions = readdir(dir)
    subdirs = ["test", "train"]
    categories = String[]

    if (!isdir(diroutput))
        mkdir(diroutput)
    end

    # Put in absolute path
    dir = absolute(dir)
    diroutput = absolute(diroutput)

    # Check categories
    for subdir in subdirs, partition in partitions
            cats = readdir(joinpath(dir, partition, subdir))

            if isempty(categories)
                categories = cats
            else
                @assert size(cats) == size(categories) && all(cats .== categories) "Error, categories '$categories' and '$cats' are not the same"
            end
    end

    for subdir in subdirs
        for partition in partitions
            for category in categories
                files = readdir(joinpath(dir, partition, subdir, category), join=true)
                outputdir = joinpath(diroutput, partition)

                if (!isdir(outputdir))
                    mkdir(outputdir)
                end

                outputfile = joinpath(outputdir, "$(subdir)_$(category).bin")

                if (isfile(outputfile))
                    println("Ignore '$outputfile'")
                    continue
                end

                output = reduce(hcat, [apply_model(file)::Array{Float32,2} for file in files])

                open(outputfile, "w") do file
                    serialize(file, output)
                end
                println("Written '$outputfile'")
            end
        end
    end
end

function main_apply_resnet()
    Random.seed!(42)
    model = ResNet()

    function apply_resnet(file)
        img = RGB.(Images.load(file))
        output = model.layers(Metalhead.preprocess(img))
        return output
    end
    apply_model(apply_resnet, "data/", "resnet_data/")
end

isinteractive() || main_apply_resnet()

```

Nowadays, there is effort in create a DataLoader that allow this type of things.

I hope it could help you.

---

<div class="post-metadata">

### Author: ![SubhadityaMukherjee](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/subhadityamukherjee/32/15983_2.png) [@SubhadityaMukherjee](https://discourse.julialang.org/u/SubhadityaMukherjee)
#### Post date: [June 14, 2020, 6:42am UTC](https://discourse.julialang.org/t/custom-dataset/41329/3 "2020-06-14T06:42:01Z")

</div>

Thank you so much! This really helps a lot. @dmolina

I was actually trying to make something like fastais(from the Pytorch world) data loader for Julia. I thought it would be really helpful as this bit of Flux isnt that developed yet and it is really awesome so why not contribute a bit haha. But I couldn’t figure out how to get the files in an array in usable time.  
Now hopefully I will be able to 🙂

Have a great day!