# \`varinfo()\` on a large SimpleGraph or DataFrame is prohibitively slow

**URL:** <https://discourse.julialang.org/t/varinfo-on-a-large-simplegraph-or-dataframe-is-prohibitively-slow/38851>\
**Category:** Performance\
**Created:** [May 6, 2020, 12:53am UTC](https://discourse.julialang.org/t/varinfo-on-a-large-simplegraph-or-dataframe-is-prohibitively-slow/38851 "2020-05-06T00:53:53Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [May 6, 2020, 12:53am UTC](https://discourse.julialang.org/t/varinfo-on-a-large-simplegraph-or-dataframe-is-prohibitively-slow/38851/1 "2020-05-06T00:53:53Z")

</div>

The only time I use `varinfo` is when I have a large object in memory and I want to know how much space it is taking. Unfortunately, in many scenarios `varinfo` runtime goes through the roof for large objects, so it’s impossible to know how large your objects are.

Here is some benchmarking. First for a `SimpleGraph` and then for a `DataFrame`.

## SimpleGraph

```julia
n = 10 .^ (1:6)

g1 = SimpleGraph(n[1])
g2 = SimpleGraph(n[2])
g3 = SimpleGraph(n[3])
g4 = SimpleGraph(n[4])
g5 = SimpleGraph(n[5])
g6 = SimpleGraph(n[6])

t_g = zeros(6)
for i in 1:6
    regex = Regex("g$i")
    t_g[i] = time(minimum(@benchmark(varinfo($regex)))) / 1e9
end

plot(
    log10.(n),
    log10.(t_g),
    legend=false,
    title="varinfo() runtime on SimpleGraph",
    xlab="log10(n)",
    ylab="log10(t_g)",
    seriestype=:line,
    markershape=:circle,
    aspectratio=1,
    ticks=-4:7,
    dpi=300
)

```

 ![df_varinfo_times](https://global.discourse-cdn.com/julialang/original/3X/e/4/e488520092de43d463d55857a922001159a92f29.png)

```julia
julia> DataFrame(n=n, t=t_g)
6×2 DataFrame
│ Row │ n │ t │
│ │ Int64 │ Float64 │
├─────┼─────────┼─────────────┤
│ 1 │ 10 │ 2.4053e-5 │
│ 2 │ 100 │ 5.9977e-5 │
│ 3 │ 1000 │ 0.000298084 │
│ 4 │ 10000 │ 0.00399715 │
│ 5 │ 100000 │ 0.0445316 │
│ 6 │ 1000000 │ 3.06422 │

```

## DataFrame

```julia
df1 = DataFrame(x = [randstring(8) for _ in 1:n[1]])
df2 = DataFrame(x = [randstring(8) for _ in 1:n[2]])
df3 = DataFrame(x = [randstring(8) for _ in 1:n[3]])
df4 = DataFrame(x = [randstring(8) for _ in 1:n[4]])
df5 = DataFrame(x = [randstring(8) for _ in 1:n[5]])
df6 = DataFrame(x = [randstring(8) for _ in 1:n[6]])

t_df = zeros(6)
for i in 1:6
    regex = Regex("df$i")
    t_df[i] = time(minimum(@benchmark(varinfo($regex)))) / 1e9
end

plot(
    log10.(n),
    log10.(t_df),
    legend=false,
    title="varinfo() runtime on DataFrame",
    xlab="log10(n)",
    ylab="log10(t_df)",
    seriestype=:line,
    markershape=:circle,
    aspectratio=1,
    ticks=-4:7,
    dpi=300
)

```

 ![graph_varinfo_times](https://global.discourse-cdn.com/julialang/original/3X/b/f/bfaf6674d3faa37025fc3dcff0311a9e1041771a.png)

```julia
julia> DataFrame(n=n, t=t_df)
6×2 DataFrame
│ Row │ n │ t │
│ │ Int64 │ Float64 │
├─────┼─────────┼─────────────┤
│ 1 │ 10 │ 1.9739e-5 │
│ 2 │ 100 │ 4.3522e-5 │
│ 3 │ 1000 │ 0.000242769 │
│ 4 │ 10000 │ 0.00209143 │
│ 5 │ 100000 │ 0.0252908 │
│ 6 │ 1000000 │ 0.987592 │

```

## Discussion

You can see from the graphs and the tables that the runtime grows faster than linearly when the objects get large. Running the benchmark for the `n = 1e7` case isn’t possible because it just takes too long.

Is there a better way to get size estimates for large objects in memory? Maybe `varinfo` could be modified to use some approximations for large objects so that it runs in a reasonable amount of time?

## Addendum

I noticed that if the DataFrame contains only numbers, e.g. `DataFrame(x = rand(1_000_000))`, then `varinfo` runs in a fraction of a second, even for very large data frames.