ERROR: OutOfMemoryError()

General Usage Performance
1

I got ERROR: OutOfMemoryError() while running this code. How can i mitigate it? Here 872 is number of mesh blocks of whose coordinates are x, y and z.

x = rand(16,872);
y = rand(4,872);
z = rand(18,872);

r = reshape(x, :, 1, 1)
θ = reshape(y, 1, :, 1)
ϕ = reshape(z, 1, 1, :)

X = r .* sin.(θ) .* cos.(ϕ)
Y = r .* sin.(θ) .* sin.(ϕ)
Z = r .* cos.(θ)

My laptop have 32Gb ram. My lscpu output looks like:

Summary

shell> lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 46 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 14
On-line CPU(s) list: 0-13
Vendor ID: GenuineIntel
Model name: Intel(R) Core™ Ultra 7 155U
CPU family: 6
Model: 170
Caches (sum of all):
L1d: 352 KiB (10 instances)
L1i: 640 KiB (10 instances)
L2: 10 MiB (5 instances)
L3: 12 MiB (1 instance)

2

I mean, you’re creating two arrays with 763839184896 8-byte elements in them. If I got my orders of magnitude right, that’s 12 terabytes right there, no?

1 Like
3

Which two arrays; Xt and Yt?

4

yes

5

Did you intend for the 872 dimension to be shared or reduced over?

6

I couldn’t understand what you mean by :down_arrow: shared or reduced over.

7

I’m trying to ask what you are intending this code to do.

8

I want to write this code below :backhand_index_pointing_down: using vectorisation @. so that i don’t iterate in for loops.

    for b in 1:872
        xs = x[:, b]
        ys = y[:, b]
        zs = z[:, b]

        for i in 1:length(xs), j in 1:length(ys), k in 1:length(zs)
            r, θ, ϕ = xs[i], ys[j], zs[k]
            X = r * sin(θ) * cos(ϕ)
            Y = r * sin(θ) * sin(ϕ)
            Z = r * cos(θ)

            push!(all_xs, X)
            push!(all_ys, Y)
            push!(all_zs, Z)
        end
    end

Is my x, y and z array reshaping code of first starting post wrong? :innocent:

9

Well, you got it wrong. You rather need something like this, and possibly permute things to get the orderings right.

x = rand(16,872);
y = rand(4,872);
z = rand(18,872);

r = reshape(x, :, 1, 1, 872)
θ = reshape(y, 1, :, 1, 872)
ϕ = reshape(z, 1, 1, :, 872)

X = vec(r .* sin.(θ) .* cos.(ϕ))
Y = vec(r .* sin.(θ) .* sin.(ϕ))
Z = vec(r .* cos.(θ))

Why do you want to? Vectorizing this kind of code most of the time makes it harder to reason about and there should be no fundamental differences in speed. If you want to run it on GPU I could see a point. But as you can already see, it’s easy to get things wrong.

2 Likes
10

Indeed, the reshaping did not work. Always good to check shapes:

julia> r = reshape(x, :, 1, 1);

julia> size(r)
(13952, 1, 1)

The closest analog to numpy would probably be

const newaxis = [CartesianIndex()]
r = x[:, newaxis, newaxis, :]
θ = y[newaxis, :, newaxis, :]
ϕ = z[newaxis, newaxis, :, :]

X = vec(r .* sin.(θ) .* cos.(ϕ))

Alternatively, you can use something like Tullio.jl or TensorCast.jl and write it as

using TensorCast

@cast X[i⊗ j⊗ k⊗ b] := x[i, b] * sin(y[j, b]) * cos(z[k, b])

which closely matches your intended loop structure.

11

Yes, You pointed it correctly X and Y are correct. But your Z = vec(r .* cos.(θ)) gives less elements than X and Y coordinate points. Thanks