Getting data from and to arrays

Ferran_Mazzanti · August 7, 2017, 4:54pm

Dear all,

a simple question that could however help me a lot with my Monte Carlo
codings I’m developping in Julia. The fact is that I need to be able
to modify values from an array in several ways. Iagine I have the
following array

z = [1,2,3,4,5]

then I copy it to zz

zz = z

if now I modify any element either in z or in zz, both z and zz see
the modification

zz[1] = 0
z

prints

5-element Array{Int64,1}:
0
2
3
4
5

z[1] = 1
zz

prints

5-element Array{Int64,1}:
1
2
3
4
5

as expected.

This works very well as I want
Now I need to do similar things BUT with n-dimensional arrays. Assume
I have an array of the form

M = ones(5,2)
2×5 Array{Float64,2}:
1.0 1.0 1.0 1.0 1.0
1.0 1.0 1.0 1.0 1.0

first index is particle number, second is dimension. Now I need to be
able to refer to whole particle vectors, and be able to make
modifications to it such that they keep in M. And viceversa: make
modifications in M and get them reflected in the vectors. As an
exemple, one could think of

r = M[:,3]

and then be able to do something like

r = [0.0, 0.0]

and recover

M
2×5 Array{Float64,2}:
1.0 1.0 0.0 1.0 1.0
1.0 1.0 0.0 1.0 1.0

which does not work. And the other way around, make

M[:,2] = [2.0,2.0]

and get
r
2-element Array{Float64,1}:
2.0
2.0

Can this be done? Can this r vector be also part of a type
deffinition?

Thanks a lot,

Ferran.

mohamed82008 · August 7, 2017, 5:04pm

You can use @view and subarrays.

julia> M = ones(2,5)
2×5 Array{Float64,2}:
 1.0  1.0  1.0  1.0  1.0
 1.0  1.0  1.0  1.0  1.0

julia> r = @view M[:,3]
2-element SubArray{Float64,1,Array{Float64,2},Tuple{Base.Slice{Base.OneTo{Int64}},Int64},true}:
 1.0
 1.0

julia> r[:] = [2.0,2.0]
2-element Array{Float64,1}:
 2.0
 2.0

julia> M
2×5 Array{Float64,2}:
 1.0  1.0  2.0  1.0  1.0
 1.0  1.0  2.0  1.0  1.0

Ferran_Mazzanti · August 7, 2017, 6:09pm

Ahhh… that was it I though @view worked in one way only, but now I see it wrks both ways…
Thanks.

davidbp · August 8, 2017, 10:06am

Hola Ferran :),

There are a couple of relevant things

First about

r = M[:,3]
and then be able to do something like
r = [0.0, 0.0]
and recover
M
2×5 Array{Float64,2}:
1.0 1.0 0.0 1.0 1.0
1.0 1.0 0.0 1.0 1.0

In julia, you can declare new variables at anytime and replace whatever they had.
Therefore even if you use views the following code will not do what you expected;

r = view(M,:,3)
r = [0.0, 0.0]
M
2×5 Array{Float64,2}:
 1.0  1.0  1.0  1.0  1.0
 1.0  1.0  1.0  1.0  1.0

Because … first you say r is something of type view and then you assign r (reassign r in this case) with an array.

You coud also do

r = view(M,:,3)
r = "this now is a string! "

And you would probably never thing that you are modifying M.

As mohammed wrote, what is very important is to write r[:]= something.
Moreover you don’t need to use the @view macro you can simply use view(M,:,3)

M = ones(2,5)
r = view(M,:,3)
r[:] = [99,99]
M

Will print

2×5 Array{Float64,2}:
 1.0  1.0  99.0  1.0  1.0
 1.0  1.0  99.0  1.0  1.0

For more details on the difference between assignment and mutation…

About views

Views are usefull for changing the array but also for moving pointers of subarrays.

Let’s say you iterate over a big array in slices, if your array is already in memory, maybe there is no need to create new arrays for every slice you use.

M = ones(784,60000);
@time view(M,:,1:1000);
@time M[:,1:1000];

Will print

julia> @time r1 = view(M,:,1:1000); #makes a pointer to the slice
  0.000053 seconds (40 allocations: 1.063 KiB)

julia> @time  r2 = M[:,1:1000]; #makes a copy of the slice in M
  0.002812 seconds (8 allocations: 5.982 MiB)

And as explained before, if you modify r1 you will modify M but if you modify r2 you will not modify M .

mkborregaard · August 8, 2017, 10:34am

Or r .= [99,99]

davidbp · August 8, 2017, 10:39am

Yes that is much cleaner.

Ferran_Mazzanti · August 10, 2017, 8:05pm

Hi again,

I have tried this within an object in a type definition and something
is not working. So with the same example, I now define

M = ones(2,5)

2×5 Array{Float64,2}:
1.0 1.0 1.0 1.0 1.0
1.0 1.0 1.0 1.0 1.0

type BEAD
r :: Array{Float64,1}
BEAD() = new()
end;

z = BEAD()

BEAD(undef)

but now if I do

z.r = @view M[:,3]

2-element SubArray{Float64,1,Array{Float64,2},Tuple{Colon,Int64},true}:
1.0
1.0

when I try to redefine the contents

z.r[:] = [2.0,2.0]

2-element Array{Float64,1}:
2.0
2.0

I see NO changes in M

2×5 Array{Float64,2}:
1.0 1.0 1.0 1.0 1.0
1.0 1.0 1.0 1.0 1.0

I’m sure this is related to type definitions, but I can’t sget what am
I doing wrong. I’ll appreciate some further help.

And in fact this is not exactly what I want, as in my code I need to
define an array of BEAD() objects, so I would create something of the
form

ZZ = [ BEAD() for i in 1:5 ]

and would like to do

for i in 1:5
ZZ[i].r = @view M[:,i]
end

…and be able to redefine each element in M from the corresponding
elements of the ZZ[i] array. Please notice I’m still in Julia 0.5.2,
but I will be moving to Julia 0.6.0 soon.

Thanks a lot

gracies David

Ferran.

rdeits · August 10, 2017, 8:10pm

Your type BEAD has its field r as an Array{Float64, 1}. So when you do

z.r = @view M[:, 3]

what happens is that:

A view is created representing M[:, 3]
That view is converted to an Array in order to be stored in z.r. That’s what always happens when you try to set a field of a type: Julia will convert whatever you’re setting into the type of that field (or give an error if it can’t be converted)
That conversion to an Array actually copies the data, so it no longer refers to the same data as M.

One way you could solve this would be to parameterize BEAD on the particular type of array:

julia> type BEAD{T <: AbstractArray}
         r::T
       end

julia> M = ones(2, 5)
2×5 Array{Float64,2}:
 1.0  1.0  1.0  1.0  1.0
 1.0  1.0  1.0  1.0  1.0

julia> z = BEAD(@view M[:, 3])
BEAD{SubArray{Float64,1,Array{Float64,2},Tuple{Base.Slice{Base.OneTo{Int64}},Int64},true}}([1.0, 1.0])

julia> z.r[:] = 2
2

julia> M
2×5 Array{Float64,2}:
 1.0  1.0  2.0  1.0  1.0
 1.0  1.0  2.0  1.0  1.0

in this way, the BEAD can actually store the view into M, rather than making a copy.

mohamed82008 · August 11, 2017, 4:16am

Following what @rdeits said, you can do the following.

julia> type BEAD{T <: AbstractArray}
           r::T
       end

julia> M = ones(2,5)
2×5 Array{Float64,2}:
 1.0  1.0  1.0  1.0  1.0
 1.0  1.0  1.0  1.0  1.0

julia> init = @view zeros(1,1)[:,1]
1-element SubArray{Float64,1,Array{Float64,2},Tuple{Base.Slice{Base.OneTo{Int64}},Int64},true}:
 0.0

julia> ZZ = [BEAD(init) for i in 1:5]
5-element Array{BEAD{SubArray{Float64,1,Array{Float64,2},Tuple{Base.Slice{Base.OneTo{Int64}},Int64},true}},1}:
 BEAD{SubArray{Float64,1,Array{Float64,2},Tuple{Base.Slice{Base.OneTo{Int64}},Int64},true}}([0.0])
 BEAD{SubArray{Float64,1,Array{Float64,2},Tuple{Base.Slice{Base.OneTo{Int64}},Int64},true}}([0.0])
 BEAD{SubArray{Float64,1,Array{Float64,2},Tuple{Base.Slice{Base.OneTo{Int64}},Int64},true}}([0.0])
 BEAD{SubArray{Float64,1,Array{Float64,2},Tuple{Base.Slice{Base.OneTo{Int64}},Int64},true}}([0.0])
 BEAD{SubArray{Float64,1,Array{Float64,2},Tuple{Base.Slice{Base.OneTo{Int64}},Int64},true}}([0.0])

julia> for i in 1:5
           ZZ[i].r = @view M[:,i]
       end

julia> ZZ[2].r[1] = 100
100

julia> M
2×5 Array{Float64,2}:
 1.0  100.0  1.0  1.0  1.0
 1.0    1.0  1.0  1.0  1.0

Ferran_Mazzanti · August 12, 2017, 6:19pm

Hi guys again,
thanks to all your input, I now understand what was going on. As suspected, it was just a matter of typing: @view M[:,3] is not producing an Array{Float64,1} but something fancy as

SubArray{Float64,1,Array{Float64,2},Tuple{Colon,Int64},true}
which is amazing by itself If I replace the BEAD definition to

type BEAD
     r :: SubArray{Float64,1,Array{Float64,2},Tuple{Colon,Int64},true}
     BEAD() = new()
end;

then everything works as expected because this type is the same as the one corresponding to slicing the original 2D array M (so no conversion is done whatsoever). Still I understand it is better to use the parametrised version explained above, so you don’t have to make hideous declarations as the above one…

Thanks a lot to everyone,

Ferran.

DNF · August 12, 2017, 6:52pm

Not just that. Now, it will only work for that particular kind of SubArray, and nothing else. It will not work for ordinary slices, for example. Are you sure that’s what you want?

Also, the type keyword is being deprecated. You should use struct or mutable struct instead.

Ferran_Mazzanti · August 12, 2017, 10:38pm

Yes I’m sure this is the only thing I need in my code, but the parametrised form is, as I said, less terrible to use

And I know about struct, etc… but I was coding in Julia 0.5.0. I will be moving to 0.6.0 soon, but at the moment I have quite a bit of coding done in 0.5.x and will have to seat down to port it, but right now I have to produce numbers before doing that.

Thanks for your help,

Ferran.

Ferran_Mazzanti · September 1, 2017, 9:22am

Hi again folks,
I have found that now I need a refined version of the parametrised types mentioned above. The fact is that my type (now mutable struct) has to contain more fields than the parametrised one, something like

mutable struct BEAD_2{T <: AbstractArray}
   x::Int;
   r::T;
end

but now I want to be able to perform incomplete initialisation of objects of the BEAD_2 type. Usually I do such things by creating types of the form

mutable struct TEST{T <: AbstractArray}
   x::Int;
   y::Float64;
   TEST() = new();
end

and then I can do

z=TEST()
z.x=1

and keep z.y uninitialised. I like this form very much because sometimes my types contain many fields and depending on what I’m doing, I must initialise some of them and leave others undefined.

How would I achieve the same thing with the BEAD_2 definition above? I mean, I want to be able to tell what T is (for instance @view M[:,3] in the examples above) but leave all other fields uninitialized…

Thanks again for your help,

Ferran.

favba · September 1, 2017, 12:55pm

Maybe:

mutable struct TEST{T <: AbstractArray}
   x::Int;
   y::T;
   TEST{T}() where T<:AbstractArray = new{T}();
end

Then you do, for example, a=TEST{Vector{Float64}}()

Ferran_Mazzanti · September 1, 2017, 3:40pm

Hi fevba,

that does work with your example

z = TEST{Vector{Float64}}()
> TEST{Array{Float64,1}}(4531977880, #undef)

but when I try

b = TEST{@view M[:,1]}()
> TypeError: Type: in parameter, expected Type, got SubArray{Float64,1,Array{Float64,2},Tuple{Base.Slice{Base.OneTo{Int64}},Int64},true}

Stacktrace:
 [1] include_string(::String, ::String) at ./loading.jl:515

so it does not seem to work with the views Any new idea?

Thanks again,

Ferran.

favba · September 1, 2017, 9:32pm

The parameter must be a type, you are trying to initialize it with a actual subarray.
Try this:

b = TEST{typeof(@view M[:,1])}()

favba · September 1, 2017, 9:51pm

What I just wrote above doesn’t make much sense, because you’d be creating the field already.
If I understand it right, you want to initialize just the array field and leave the other fields undefined, right?
Try this then:

julia> mutable struct TEST{T<:AbstractArray}
       y::T
       x::Int
       TEST{T}(y::T) where T<:AbstractArray = new{T}(y)
       end

julia> TEST(y::T) where T = TEST{T}(y)
TEST
julia> b = TEST(zeros(2,3))
TEST{Array{Float64,2}}([0.0 0.0 0.0; 0.0 0.0 0.0], 140320647153152)

julia> b.x=3
3

The fields you want initialized at the construction of the struct must appear first on the definition of the mutable struct. Then you create a inner constructor that takes only the arguments you want to initialize.

Ferran_Mazzanti · September 2, 2017, 8:44am

Hi favba,
that seems to be working, thanks Still I’d like to understand it. Why do we have to issue the intermediate

 julia> TEST(y::T) where T = TEST{T}(y)

thing?

Best regards and thanks,

Ferran.