# Variables @over dimensions

**URL:** <https://discourse.julialang.org/t/variables-over-dimensions/74730>\
**Category:** General Usage\
**Tags:** dataframes, dataframesmeta\
**Created:** [January 17, 2022, 4:06am UTC](https://discourse.julialang.org/t/variables-over-dimensions/74730 "2022-01-17T04:06:26Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Lincoln\_Hannah](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lincoln_hannah/32/19198_2.png) [@Lincoln\_Hannah](https://discourse.julialang.org/u/Lincoln_Hannah)\
**Post date:** [January 17, 2022, 4:06am UTC](https://discourse.julialang.org/t/variables-over-dimensions/74730/1 "2022-01-17T04:06:26Z")

</div>

Does a package like this exist?

Define variables @over dimensions / keys.  
System then creates the underlying DataFrames.  
PK-FK relationships remove the need for joins and lookup staetemnts.

```julia
@over :X = 0:2 # key :X defined as [0,1,2]
    :Y = :X * 2 # within @over block, like being within @rtransform @astable.
    :Z = :X + :Y # operations are at a row level, each new column can depend on all previous columns.
end # this block creates a DataFrame with columns :X :Y :Z

@over :X # system keeps track of all defined keys.
    :Z_2 = :Z * 2 # can create a block over any previously defined key and new fields.
end

@over :A # key stated but not defined, and hasn't been defined previously
    DataFrame( A=1:10, B=11:20 ) # So, first line must be a dataframe containing the key as a column. 
    :C = mod(:A,3) # operations at row level - like being within @rtransform @astable

    @rsubset :C ∈ [0,1] # @rsubset works as within @chain.
                                   # corresponding rows of the underlying DataFrame are removed.

    @over :C # @over blocks can be nested.
        :D = :C * 2 # this block is similar to @groupby :C; @combine. :C also set as FK against :A
        :Bsum = sum( :A.B ) # So ,variables defined over :A can be used with dot notation
    end # ending of nested block. Subsequent columns are over :A

    :E = :C.D * 2 # columns defined over :C can also be used in @over :A block.
    :BpercentC = :B / :C.Bsum
end

@over :G = 1:1000
    :H = :G * 2
    @FK :A = mod(:G,10) + 1 # Explicitly define :A as FK against :G
    :I = :A.B # variables defined over :A can now be used within the @over :G block
    :J = :C.D * 2 # variables defined over :C can also be used.
end # System combines the relationships ( :G-:A :A-:C )

@over :A
    :J = sum( :G.H ) # variables defined over :G can be used within an @over :A block
end

@over :A, :K = 1:10 # blok defined over multiple dimensios / keys.
                                 # variables within the block defined over cross-product of :A and :K
    :L = :A * :K
    :LdifK = :L -prev(:K,1,:L) # prev(:K,1,:L) - get the value of :L, 1 step back along the :K axis.
                                 # :K must be one of the @over block dimensions.
end

A[1].B # outside of an @over block, dictionary notation can be used.
                                 # This gets the value of the :B column when key :A == 1.

```

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [January 17, 2022, 4:24am UTC](https://discourse.julialang.org/t/variables-over-dimensions/74730/2 "2022-01-17T04:24:22Z")

</div>

don’t quite get the “over dimensions” part of the title, but macro seems to be a bad way of doing this. Your example seems to imply some global scope trick, especially if you want

> system keeps track of all defined keys.  
> can create a block over any previously defined key and new fields.

your first two blocks can be easily done with:

```julia
julia> df = DataFrame(X=0:2); 
df.Y = df.X * 2; 
df.Z = df.X + df.Y;

julia> df.Z_2 = df.Z * 2

julia> df
3×4 DataFrame
 Row │ X Y Z Z_2   
     │ Int64 Int64 Int64 Int64 
─────┼────────────────────────────
   1 │ 0 0 0 0
   2 │ 1 2 3 6
   3 │ 2 4 6 12

```

lines like:

```julia
:C = mod(:A,3) 

```

can be done via broadcast:

```julia
df.C = mod.(df.A, 3)

```

etc.

Macro should not be used when things can be done without considerable more trouble with plain functions because macros are not composable

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [January 17, 2022, 4:42am UTC](https://discourse.julialang.org/t/variables-over-dimensions/74730/3 "2022-01-17T04:42:05Z")

</div>

I have a package that does something similar to what you want, I think. [AddToField.jl](https://github.com/pdeffebach/AddToField.jl). I’m not fully following what you mean but it could be helpful. Something like

```julia
julia> using AddToField

julia> A = 1:100;

julia> map(A) do a
           @addnt begin 
               @add b = a + 1
               @add c = b + a
           end
       end |> DataFrame
100×2 DataFrame
 Row │ b c     
     │ Int64 Int64 
─────┼──────────────
   1 │ 2 3
   2 │ 3 5
   3 │ 4 7
   4 │ 5 9
   5 │ 6 11
...

```

Also note that the `@astable` macro-flag in DataFramesMeta.jl gets somewhat close to the syntax you mention.

If you want to explore this idea more fully I would try to get a non-macro solution working with SplitApplyCombine.jl and see where you land.