Variables @over dimensions

General Usage
,
1

Does a package like this exist?

Define variables @over dimensions / keys.
System then creates the underlying DataFrames.
PK-FK relationships remove the need for joins and lookup staetemnts.

@over :X = 0:2           # key :X  defined as [0,1,2]
    :Y = :X * 2          # within @over block, like being within  @rtransform @astable.
    :Z = :X + :Y         # operations are at a row level, each new column can depend on all previous columns.
end                      # this block creates a DataFrame with columns :X :Y :Z

@over :X                 # system keeps track of all defined keys.
    :Z_2 = :Z * 2        # can create a block over any previously defined key and new fields.
end

@over :A                           # key stated but not defined, and hasn't been defined previously
    DataFrame( A=1:10, B=11:20 )   # So, first line must be a dataframe containing the key as a column. 
    :C = mod(:A,3)                 # operations at row level - like being within  @rtransform @astable

    @rsubset :C ∈ [0,1]            # @rsubset works as within @chain.
                                   # corresponding rows of the underlying DataFrame are removed.

    @over :C                       # @over blocks can be nested.
        :D      = :C * 2           # this block is similar to @groupby :C; @combine.  :C also set as FK against :A
        :Bsum   = sum( :A.B )      # So ,variables defined over :A can be used with dot notation
    end                            # ending of nested block. Subsequent columns are over :A

    :E           = :C.D * 2        # columns defined over :C can also be used in @over :A block.
    :BpercentC   = :B / :C.Bsum
end

@over :G = 1:1000
    :H = :G * 2
    @FK :A = mod(:G,10) + 1      # Explicitly define :A as FK against :G
    :I = :A.B                    # variables defined over :A can now be used within the @over :G block
    :J = :C.D * 2                # variables defined over :C can also be used.
end                              # System combines the relationships ( :G-:A  :A-:C )

@over :A
    :J = sum( :G.H )             # variables defined over :G can be used within an @over :A block
end

@over :A, :K = 1:10              # blok defined over multiple dimensios / keys.
                                 # variables within the block defined over cross-product of :A and :K
    :L      = :A * :K
    :LdifK  = :L -prev(:K,1,:L)  # prev(:K,1,:L) - get the value of :L, 1 step back along the :K axis.
                                 # :K must be one of the @over block dimensions.
end


A[1].B                           # outside of an @over block, dictionary notation can be used.
                                 # This gets the value of the :B column when key :A == 1.
2

don’t quite get the “over dimensions” part of the title, but macro seems to be a bad way of doing this. Your example seems to imply some global scope trick, especially if you want

system keeps track of all defined keys.
can create a block over any previously defined key and new fields.

your first two blocks can be easily done with:

julia> df = DataFrame(X=0:2); 
df.Y = df.X * 2; 
df.Z = df.X + df.Y;

julia> df.Z_2 = df.Z * 2

julia> df
3×4 DataFrame
 Row │ X      Y      Z      Z_2   
     │ Int64  Int64  Int64  Int64 
─────┼────────────────────────────
   1 │     0      0      0      0
   2 │     1      2      3      6
   3 │     2      4      6     12

lines like:

:C = mod(:A,3)

can be done via broadcast:

df.C = mod.(df.A, 3)

etc.

Macro should not be used when things can be done without considerable more trouble with plain functions because macros are not composable

3

I have a package that does something similar to what you want, I think. AddToField.jl. I’m not fully following what you mean but it could be helpful. Something like

julia> using AddToField

julia> A = 1:100;

julia> map(A) do a
           @addnt begin 
               @add b = a + 1
               @add c = b + a
           end
       end |> DataFrame
100×2 DataFrame
 Row │ b      c     
     │ Int64  Int64 
─────┼──────────────
   1 │     2      3
   2 │     3      5
   3 │     4      7
   4 │     5      9
   5 │     6     11
...

Also note that the @astable macro-flag in DataFramesMeta.jl gets somewhat close to the syntax you mention.

If you want to explore this idea more fully I would try to get a non-macro solution working with SplitApplyCombine.jl and see where you land.