# Performance of creating a demo dataset

**URL:** <https://discourse.julialang.org/t/performance-of-creating-a-demo-dataset/85441>\
**Category:** General Usage\
**Tags:** question, performance, inmemorydatasets\
**Created:** [August 7, 2022, 5:15pm UTC](https://discourse.julialang.org/t/performance-of-creating-a-demo-dataset/85441 "2022-08-07T17:15:58Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [August 7, 2022, 5:15pm UTC](https://discourse.julialang.org/t/performance-of-creating-a-demo-dataset/85441/1 "2022-08-07T17:15:58Z")

</div>

I have the following code:

```julia
using InMemoryDatasets

function demo_data()
    ds = Dataset(time=0.0, d1=10, d2=20, d3=30)
    time = 0.1
    for i in 1:100000
        if i == 5
            d1 = missing
        else
            d1 = 10+rand(1:30)
        end
        ds2 = Dataset(time=time, d1=d1, d2=20+i, d3=30+rand(1:30))
        append!(ds, ds2)
        time += 0.1
    end
    ds
end

demo_data()
@time ds = demo_data()

```

The second call needs 4.8s on my machine. For realistic tests I need 1e6 rows in the Dataset. How can I improve the speed and reduce the memory usage?

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [August 7, 2022, 5:36pm UTC](https://discourse.julialang.org/t/performance-of-creating-a-demo-dataset/85441/2 "2022-08-07T17:36:30Z")

</div>

Typically the first step is to preallocate the needed space instead if incrementally append!-ing. Like:

```julia
n=100000
ds = Dataset(time=zeros(Float64,n), d1=zeros(Int,n), d2=zeros(Int,n), d3=zeros(Int,n))

```

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [August 7, 2022, 6:40pm UTC](https://discourse.julialang.org/t/performance-of-creating-a-demo-dataset/85441/3 "2022-08-07T18:40:53Z")

</div>

Thanks! And how can I create a vector of random numbers in the range of 1…30?

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [August 7, 2022, 6:46pm UTC](https://discourse.julialang.org/t/performance-of-creating-a-demo-dataset/85441/4 "2022-08-07T18:46:01Z")

</div>

Do you mean like

```julia
rand(1:30,n)

```

?  
Yes, this is faster than doing single rand’s in the loop, I guess.

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [August 7, 2022, 6:56pm UTC](https://discourse.julialang.org/t/performance-of-creating-a-demo-dataset/85441/5 "2022-08-07T18:56:07Z")

</div>

```julia
ds = Dataset(
    time= collect(range(start=0.0,step=0.1,length=n)), 
    d1=rand(11:40,n), 
    d2=collect(21:n+20), 
    d3=rand(31:60,n)
)

```

Now you need only inject the missings.

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [August 7, 2022, 7:12pm UTC](https://discourse.julialang.org/t/performance-of-creating-a-demo-dataset/85441/6 "2022-08-07T19:12:42Z")

</div>

```julia
using InMemoryDatasets

function demo_data()
    n = 100000
    time = 0.1:0.1:n*0.1
    d1 = rand(Int8(11):Int8(40), n)
    d2 = 21:n + 20
    d3 = rand(Int8(31):Int8(60), n)
    ds = Dataset(time=time, d1=d1, d2=d2, d3=d3)
    ds.d1[5] = missing
    ds
end

demo_data()
@time ds=demo_data()

```

This looks quite nice… Only 2 to 8 ms instead of 5s! 🙂

So a for loop is not always a good idea…

---

<div class="post-metadata">

**Author:** ![Henrique\_Becker](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/henrique_becker/32/15443_2.png) [@Henrique\_Becker](https://discourse.julialang.org/u/Henrique_Becker)\
**Post date:** [August 7, 2022, 11:33pm UTC](https://discourse.julialang.org/t/performance-of-creating-a-demo-dataset/85441/7 "2022-08-07T23:33:39Z")

</div>

The problem here is not the loop, but the multiple calls to `append!` instead of preallocating everything.

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [August 8, 2022, 2:56am UTC](https://discourse.julialang.org/t/performance-of-creating-a-demo-dataset/85441/8 "2022-08-08T02:56:38Z")

</div>

I do not think so. I think creating a new Dataset on each iteration was the bottleneck.