# Creating an AbstractDataFrame subtype
**URL:** https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451
**Category:** General Usage
**Tags:** question
**Created:** [March 24, 2020, 2:37pm UTC](https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451 "2020-03-24T14:37:22Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@DrPapa](https://discourse.julialang.org/u/DrPapa)
#### Post date: [March 24, 2020, 2:37pm UTC](https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451/1 "2020-03-24T14:37:22Z")
I wish to create a new subtype of AbstractDataFrame that has a single DataFrame member. I then want to forward the AbstractDataFrame interface to the DataFrame member. Something like:
```julia
using DataFrames
using Lazy
struct DFWrapper <: AbstractDataFrame
df::DataFrame
other
stuff
end
@forward DFWrapper.df DataFrames.describe, Base.summary, Base.hcat, Base.vcat,
Base.repeat, Base.names, DataFrames.rename!, Base.length, Base.size,
Base.first, Base.last, Base.convert, DataFrames.completecases,
DataFrames.dropmissing, DataFrames.dropmissing!, DataFrames.nonunique,
Base.unique, Base.unique!, DataFrames.disallowmissing, DataFrames.disallowmissing!,
DataFrames.allowmissing, DataFrames.allowmissing!, DataFrames.categorical, DataFrames.categorical!,
Base.similar, Base.filter, Base.filter!
DFW = DFWrapper(DataFrame(A=1:10), 1.0, false)
```
I obtained the list of methods being forwarded form [here](https://github.com/JuliaData/DataFrames.jl/blob/master/src/abstractdataframe/abstractdataframe.jl).
Running in the REPL, I get:
```julia
Error showing value of type DFWrapper:
ERROR: StackOverflowError:
Stacktrace:
[1] _check_consistency(::DFWrapper) at /dataframe/dataframe.jl:296 (repeats 80000 times)
```
line 296 is
```julia
_check_consistency(df::AbstractDataFrame) = _check_consistency(parent(df))
```
Forwarding `DataFrames._check_consistency` then produces
```julia
Error showing value of type DFWrapper:
ERROR: MethodError: no method matching getindex(::DFWrapper, ::typeof(!), ::Symbol)
Closest candidates are:
getindex(::AbstractDataFrame, ::Integer, ::Union{Regex, AbstractArray{T,1} where T, All, Between, InvertedIndex}) at /Users/gerlacar/.julia/packages/DataFrames/S3ZFo/src/dataframerow/dataframerow.jl:90
getindex(::AbstractDataFrame, ::Integer, ::Colon) at /Users/gerlacar/.julia/packages/DataFrames/S3ZFo/src/dataframerow/dataframerow.jl:92
getindex(::AbstractDataFrame, ::CartesianIndex{2}) at /Users/gerlacar/.julia/packages/DataFrames/S3ZFo/src/other/broadcasting.jl:3
```
I feel like I am entering a rabbit hole and that there has to be a better way to do this. Any suggestions? Thanks.
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [March 24, 2020, 2:58pm UTC](https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451/2 "2020-03-24T14:58:06Z")
What are you trying to accomplish here? The DataFrames interface is not trivial it to be honest it’s a lot of work to make a sub-dataframe type.
I _think_, but am not certain, that you need to implement `getproperty` yourself, it’s not on the list of forwarded functions. You can see that in the file you linked to they define a lt of functions themselves.
---
### Author:  [@tbeason](https://discourse.julialang.org/u/tbeason)
#### Post date: [March 24, 2020, 3:25pm UTC](https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451/3 "2020-03-24T15:25:33Z")
Why does it even need to be a subtype? Your code works if you just drop `<: AbstractDataFrame`.
EDIT: Well… most of the functions work. Some do not.
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [March 24, 2020, 4:01pm UTC](https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451/4 "2020-03-24T16:01:58Z")
You can roughly learn what is needed to define a custom `AbstractDataFrame` if you look at methods that are implemented for `SubDataFrame`. In general, as @pdeffebach noted, although this is doable it is really a hard thing to do (`AbstractDataFrame` has a lot of “convinience” functionalities and “special cases” that were added over the years per requests of the users and this is simply hard to reproduce with 100% accuracy quickly).
---
### Author:  [@DrPapa](https://discourse.julialang.org/u/DrPapa)
#### Post date: [March 24, 2020, 5:09pm UTC](https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451/5 "2020-03-24T17:09:32Z")
> [@pdeffebach](#):
>
> What are you trying to accomplish here?
This is a great question. I will admit that I took a little bit of random walk to get here, so a coarse correction may be in order. I am writing this for a specific internal application. So, I don’t actually _need_ to implement the entire DataFrame interface.
Maybe its best to explain my path here. I have data stored in CSV files from various tests that I load into a DataFrame. I want to be able to dispatch on this particular data for various plotting recipes so I made a struct like:
```julia
struct DFWrapper
df::DataFrame
other
stuff
end
```
where `other` and `stuff` are common meta data for test data, e.g. test date/time…
I would also like the ability to query and filter `DFWrapper.df` with tools like Query.jl and DataFramesMeta.jl. That led me to [this](https://discourse.julialang.org/t/creating-custom-dataframe-types/4992). I thought “that sounds easy enough”… and here I am. At this point, I don’t think its worth doing this for the project at hand given the feedback.
I am open all ears on better ideas. In essence, I want something that stores a DataFrame, acts like a DataFrame, but that I can also dispatch on.
---
### Author:  [@oxinabox](https://discourse.julialang.org/u/oxinabox)
#### Post date: [March 24, 2020, 5:14pm UTC](https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451/6 "2020-03-24T17:14:25Z")
I suggest that when you want to treat your object `dw::DataFramesWrapper`
as a DataFrame, that you just do:
`dw.df` and work with the `DataFrame` field directly.
Invenia has a a package that subtypes DataFrames, and its just a bit much maintance work for too little gain, so we are looking to retire it.
[https://github.com/invenia/KeyedFrames.jl/issues/19](https://github.com/invenia/KeyedFrames.jl/issues/19)
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [March 24, 2020, 5:16pm UTC](https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451/7 "2020-03-24T17:16:35Z")
100% you want to just work with the `DataFrame` directly and use `x.df` when you do things to your data frame.
---
### Author:  [@DrPapa](https://discourse.julialang.org/u/DrPapa)
#### Post date: [March 24, 2020, 6:26pm UTC](https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451/8 "2020-03-24T18:26:43Z")
This makes a lot of sense. I wondered if I was trying to be cute with this.
I saw that the interface for DataFrames for Query was pretty simple. So, as an exercise I tried extending that to my type and this seems pretty workable.
```julia
using DataFrames
using IterableTables
using Query
using Lazy
struct DFWrapper
df::DataFrame
other
stuff
end
#source
IterableTables.IteratorInterfaceExtensions.isiterable(x::DFWrapper) = true
IterableTables.TableTraits.isiterabletable(x::DFWrapper) = true
function IterableTables.TableTraits.getiterator(dfw::DFWrapper)
return IterableTables.TableTraitsUtils.create_tableiterator(getfield(dfw.df, :columns), names(dfw.df))
end
# Sink
function _DFWrapper(dfw::DFWrapper, x)
cols, names = IterableTables.create_columns_from_iterabletable(x, na_representation=:missing)
df = DataFrames.DataFrame(cols, names)
return DFWrapper(df,dfw.other, dfw.stuff)
end
DFWrapper(dfw::DFWrapper, x::AbstractVector{T}) where {T<:NamedTuple} = _DFWrapper(dfw, x)
function DFWrapper(dfw::DFWrapper, x)
if IterableTables.TableTraits.isiterabletable(x)
return _DFWrapper(dfw, x)
else
df = convert(DataFrames.DataFrame, x)
return DFWrapper(df, dfw.other, dfw.stuff)
end
end
function DFWrapper(dfw::DFWrapper)
return x->DFWrapper(dfw, x)
end
DFW = DFWrapper(DataFrame(A=1:10), 3.0, true)
x = DFW |>
@filter(_.A>5) |> DFWrapper(DFW) # produces DFWrapper(DataFrame(A=6:10), 3.0, true)
```
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [March 24, 2020, 6:29pm UTC](https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451/9 "2020-03-24T18:29:49Z")
Yeah that’s definitely a solution. You can also implement the `Tables.jl` interface and get Query for free.
But still, this seems like overkill a bit.
---
### Author:  [@croberts](https://discourse.julialang.org/u/croberts)
#### Post date: [July 26, 2021, 2:49pm UTC](https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451/10 "2021-07-26T14:49:35Z")
I’m going to necromance this thread.
It would be useful to define
```julia
abstract type SpecializedDataFrame <: AbstractDataFrame end
get_df(special_df::SpecializedDataFrame) = special_df.df
# generically forward methods
# e.g.:
# getindex(specdf::Specialized_df, ...) = getindex(get_df(specdf), ...)
```
Which would allow people to conveniently define AbstractDataFrame datatypes in which a DataFrame is a field. All methods defined for a DataFrame would be applied to a SpecializedDataFrame by using the get\_df() function.
I think this would be very helpful for extending dataframe functionality.
In my case, I want to construct a Panel data frame. A panel has two units of observation: a nominal one (e.g. person id) and an ordinal one (e.g. month, or survey number). Panel data permits many statistical operations that are not feasible with time series, cross-sections, or repeated cross sections.
All functions defined on a dataframe are applicable to a panel. But additional functions are applicable to panels, especially computing lags (julia has a FixedEffectsModels.jl package, which demeans the target variable after grouping by id. This is one way to compute a fixed effects, but it has different asymptotic properties than taking first differences and is not always the correct way to do it. Taking first differences requires computing lags).
If the SpecializedDataFrame abstract type existed, forwarding methods appropriately, it would be nearly trivial to develop a PanelDataFrames package that extends DataFrames to exploit this data structure. It would implement lag operators, allow for lag operators in @formula syntax, and other things. eg:
```julia
abstract type AbstractPanel <: SpecializedDataFrame end
struct PanelDataFrame <: Abstract Panel
df::DataFrame
panel_indices::Vector{Vector{Int}}
end
lag(pdf::PanelDataFrame, var::Symbol, lags=1) = #etc
```
There are many other data structures that can be similarly exploited, so the SpecializedDataFrame type could get a lot of use helping to implement different packages.
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [July 26, 2021, 3:16pm UTC](https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451/11 "2021-07-26T15:16:51Z")
This is a good idea, panel data is an important structure for the Julia data ecosystem to support.
I think the main problem is that there isn’t a well defined `AbstractDataFrame` API. What list of functions does a type _need_ to implement? No one has compiled this list, which will make it very hard to implement a `<:AbstractDataFrame`. This is something that should be done eventually, but requires significant work.
I also think a lot of what you want can be done with grouped data frames. See DataFramesMeta, for example
```julia
julia> begin
using DataFramesMeta, Dates, ShiftedArrays
years = 1985:2000
state_ids = 1:50
df = DataFrame(year = Int[], state_id = Int[])
for i in vec(collect(Base.Iterators.product(years, state_ids)))
push!(df, i)
end
sort!(df, [:year, :state_id])
df.y = randn(nrow(df))
@by df :state_id :y_lag = lag(:y)
end
```
Obviously it would be nice not to have to re-compute the indices of all the groups every time, which is presumably the whole point of a specialized panel data frame. But if you have a persistent `GroupedDataFrame` it might make things easier.
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [July 26, 2021, 3:28pm UTC](https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451/12 "2021-07-26T15:28:20Z")
As I have commented several times. Creation of such a list is not a problem. Just do:
```julia
methodswith(DataFrame)
```
and you see what requires to be implemented. And you do not have to implement everything if you do not want/need to.
The effort that would have to be made is to track down in the code places (there are several), where we explicitly assume that the only `AbstractDataFrame` types are `DataFrame` or `SubDataFrame`, which might work incorrectly if a third type would be introduced. If you feel it is worthwhile - please open an issue, and we can plan fixing all such places.
ah - and if you wanted `SubDataFrame` to be stored inside your wrapper you would need to handle it separately.
---
### Author:  [@croberts](https://discourse.julialang.org/u/croberts)
#### Post date: [July 26, 2021, 3:37pm UTC](https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451/13 "2021-07-26T15:37:03Z")
A follow up question on this.
How is the constructor working (or not working, as it were) in the example OP presented?
As a silly example, I can execute
```julia
struct NumberWrapper <: Number
my_number::Number
end
NumberWrapper(1)
```
without throwing an exception.
But
```julia
struct DataFrameWrapper <: AbstractDataFrame
df::DataFrame
end
df = DataFrame([1 2; 3 4])
DataFrameWrapper(df)
```
generates the stack overflow error OP mentions. Why is \_check\_consistency even called at all when constructing a DataFrameWrapper? Does the DataFrames.jl package somehow define methods for default constructors on AbstractDataFrames (e.g. something like writing a specialized method of Base.new(df::AbstractDataFrame) for AbstractDataFrame types, or something?
The error message only refers to one line – I don’t know what calls \_check\_consistency. I cannot find information in the Julia documentation for constructors that can help to explain this. It seems that there is some crucial information about how Julia constructors work under the hood that I do not understand.
---
### Author:  [@croberts](https://discourse.julialang.org/u/croberts)
#### Post date: [July 26, 2021, 3:44pm UTC](https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451/14 "2021-07-26T15:44:12Z")
Thanks!
---
### Author:  [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)
#### Post date: [July 26, 2021, 3:46pm UTC](https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451/15 "2021-07-26T15:46:03Z")
This is an understandable `StackOverFlow`.
```julia
_check_consistency(df::AbstractDataFrame) = _check_consistency(parent(df))
```
combined with `parent(df::AbstractDataFrame)`
```julia
julia> df = DataFrame(a = [1, 2])
2×1 DataFrame
Row │ a
│ Int64
─────┼───────
1 │ 1
2 │ 2
julia> parent(df)
2×1 DataFrame
Row │ a
│ Int64
─────┼───────
1 │ 1
2 │ 2
julia> @which parent(df)
parent(adf::AbstractDataFrame) in DataFrames at C:\Users\Peter\.julia\packages\DataFrames\nxjiD\src\abstractdataframe\abstractdataframe.jl:1898
```
that’s what’s causing the `StackOverFlow`.
Defining `parent(df::DataFrameWrapper)` fixed this problem (and gets you into printing errors which are easily fixable).
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [July 26, 2021, 3:55pm UTC](https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451/16 "2021-07-26T15:55:09Z")
Yes, for the custom data frame type you need to define `parent` if it is a wrapper around a data frame. `parent` does not need to be defined if your custom data frame would store its columns directly in some way.
---
### Author:  [@croberts](https://discourse.julialang.org/u/croberts)
#### Post date: [July 26, 2021, 4:15pm UTC](https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451/17 "2021-07-26T16:15:05Z")
Thanks!
I understand where the error comes from now. The inner constructor new(.) does not throw the exception. Rather, when a data type is constructed, base.show may be called on it. Base.show(io::IO, df::AbstractDataFrame) calls \_show, which calls \_check\_consistency(df::AbstractDataFrame). Since \_check\_consistency(df::AbstractDataFrame) = \_check\_consistency(parent(df)), and parent(df::AbstractDataFrame)=df, we have an obvious stack overflow.
The calls to show and \_show were not showing up on my stacktrace, so I had assumed the inner constructor was to blame and not the functions called to display what was just (successfully) constructed.
Thanks again!
---
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [July 26, 2021, 4:18pm UTC](https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451/18 "2021-07-26T16:18:35Z")
This is indeed what happens.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [July 27, 2021, 8:06am UTC](https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451/19 "2021-07-27T08:06:59Z")
> [@croberts](#):
>
> All functions defined on a dataframe are applicable to a panel. But additional functions are applicable to panels, especially computing lags (julia has a FixedEffectsModels.jl package, which demeans the target variable after grouping by id.
I may be missing something, but I think that DataFrame should be able to handle panel data just fine. IDs and survey information just become additional columns.
(If this is not the case, an MWE would help focus this discussion.)
---
### Author:  [@nilshg](https://discourse.julialang.org/u/nilshg)
#### Post date: [July 27, 2021, 8:51am UTC](https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451/20 "2021-07-27T08:51:57Z")
I think the point is around making time/unit based operations a bit more convenient in the way they are in say Stata when you’ve used `tsset`. E.g. you might have `lag(::PanelDataFrame, ::String, n)` give a lagged column respecting the groups, which currently would require going via `groupby` (which I personally find unproblematic, but I can understand people doing this a lot and used to working in Stata being a bit put off by some of the verbosity of DataFrames here. It’d certainly be interesting to see some attempts at innovation in this space!
That said this thread also gives me the opportunity to bring up my favourite bit of `pandas` trivia, which is that `pandas` is actually named after `pan`el `da`ta, and did indeed feature a `Panel` class as an alternative to the usual `DataFrame` early on. I remember trying to use it in the early days of my PhD (maybe ~10 years ago?), but found it unnecessarily cumbersome and indeed it was decprecated a few years later (apparently in 2017) - maybe a cautionary tale with respect to the benefits of having panel data as a distinct type, but at the same time Julia isn’t Python of course and might allow for a better design!
[Next page](https://discourse.julialang.org/t/creating-an-abstractdataframe-subtype/36451.md?page=2)