# Enforcing Schema on Data Frame Passed as Function Argument
**URL:** https://discourse.julialang.org/t/enforcing-schema-on-data-frame-passed-as-function-argument/49020
**Category:** General Usage
**Tags:** question, dataframes, function
**Created:** [October 26, 2020, 2:21am UTC](https://discourse.julialang.org/t/enforcing-schema-on-data-frame-passed-as-function-argument/49020 "2020-10-26T02:21:18Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@gyrfalcon](https://discourse.julialang.org/u/gyrfalcon)
#### Post date: [October 26, 2020, 2:21am UTC](https://discourse.julialang.org/t/enforcing-schema-on-data-frame-passed-as-function-argument/49020/1 "2020-10-26T02:21:18Z")
I have been working on some code where `DataFrame` structures seem to apply quite well. However, I was wondering if it is possible to enforce at a minimum a certain pattern of data types, if not both data types and column names, for a `DataFrame` that is passed as an argument to a function. An example with the current behavior and what I might like to happen below:
```julia
using DataFrames
using Dates
# This is what is available, as far as I can tell
function test1(df::DataFrame)
println(first(df, 1))
end
# I have tried this syntax but it does not appear to work, for illustrative purposes only
function test2(df::DataFrame{Date, Int64, Int64, Int64, Int64})
println(first(df, 1))
end
function main()
test_df1 = DataFrame("A" => Date("2020"), "B" => 1, "C" => 2,
"D" => 3, "E" => 4)
test_df2 = DataFrame("A" => Date("2020"), "B" => 1, "C" => 2,
"D" => 3)
test1(test_df1) # This should work fine
test1(test_df2) # This should work fine too
test2(test_df1) # This should work fine
test2(test_df2) # This should not work, as the schema does not match
end
main()
```
Is this something that is possible with a `DataFrame`? If not, what are the potential alternatives? I have considered defining a custom type and then requiring an array of holding that type, but I like the capability that `Dataframe`s have in being able to easily pull out ranges of rows based on tests of one column, which is more difficult with other things. Plus, I can add columns as I please.
---
### Author:  [@jling](https://discourse.julialang.org/u/jling)
#### Post date: [October 26, 2020, 2:33am UTC](https://discourse.julialang.org/t/enforcing-schema-on-data-frame-passed-as-function-argument/49020/2 "2020-10-26T02:33:16Z")
`DataFrame` is not typed like that so it won’t work, you will need to check for each column and do something like:
```julia
julia> tt = eltype.(eachcol(test_df2))
4-element Array{DataType,1}:
Date
Int64
Int64
Int64
```
but this is ~ meaningless because you can permute the order of the columns and it will be the same dataframe (to me at least) as long as each `col` (given a name) has the same type as before.
---
### Author:  [@jocklawrie](https://discourse.julialang.org/u/jocklawrie)
#### Post date: [October 26, 2020, 3:02am UTC](https://discourse.julialang.org/t/enforcing-schema-on-data-frame-passed-as-function-argument/49020/3 "2020-10-26T03:02:18Z")
The `compare` function in [Schemata.jl](https://github.com/JockLawrie/Schemata.jl) compares a table to a schema, where the schema can be specified in code but is more easily specified as a YAML file.
---
### Author:  [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)
#### Post date: [October 26, 2020, 3:51am UTC](https://discourse.julialang.org/t/enforcing-schema-on-data-frame-passed-as-function-argument/49020/4 "2020-10-26T03:51:00Z")
Your other option is to use a data structure that carries the full type information of the columns, e.g.
- Named tuple
- [StructArrays.jl](https://github.com/JuliaArrays/StructArrays.jl)
- [TypedTables.jl](https://github.com/JuliaData/TypedTables.jl)
- [IndexedTables.jl](https://github.com/JuliaData/IndexedTables.jl)
---
### Author:  [@lungben](https://discourse.julialang.org/u/lungben)
#### Post date: [October 26, 2020, 8:17am UTC](https://discourse.julialang.org/t/enforcing-schema-on-data-frame-passed-as-function-argument/49020/5 "2020-10-26T08:17:39Z")
I had the same issue:
> [@Dispatch on DataFrame columns](https://discourse.julialang.org/t/dispatch-on-dataframe-columns/40689):
>
> Hi, I have a function on a DataFrame which should behave differently depending on the names / eltypes of the DataFrame columns. Is this possible using a multiple-dispatch like pattern? Example: using DataFrames function myfunc(df:: AbstractDataFrame) # call this method if columns a + b exist and are numeric df[!, :c] = df.a .\* df.b return df end function myfunc(df:: AbstractDataFrame) # call this method if columns a + c exist and are numeric df[!, :b] = df.c ./ df.a return df …
My solution was to define a Dict with the required column names and types (abstract supertypes, where applicable) and a function to check if they are in the DataFrame (there may be more columns in the DataFrame, which is fine for me):
```julia
const INPUT_ELTYPES = Dict(
:field_a=> AbstractString,
:field_b => ItemTypes,
:field_c=> AbstractString,
:field_x=> Union{AbstractString, Nothing},
:field_y=> Real,
:value => Union{Real, Missing},
)
function check_input_data(df:: AbstractDataFrame)
@assert COLUMN_NAMES ⊆ names(df)
for (col_name, col_type) in INPUT_ELTYPES
@assert eltype(df[!, col_name]) <: col_type
end
end
```
Maybe something like this could be added to DataFrames.jl, or a typed data frame as alternative type. The latter could be just a thin wrapper over the standard DataFrame type (with untyped columns) to avoid recompilations, with the sole purpose of defining schemas.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [October 26, 2020, 10:06am UTC](https://discourse.julialang.org/t/enforcing-schema-on-data-frame-passed-as-function-argument/49020/6 "2020-10-26T10:06:13Z")
> [@gyrfalcon](#):
>
> enforce at a minimum a certain pattern of data types, if not both data types and column names
I would recommend a trait using the `Tables.schema` interface:
```julia
using Tables, DataFrames
f(table) = f_typed(Tables.schema(table), table)
f_typed(::Tables.Schema{(:a,:b),Tuple{Int64,Char}}, table) = "hey"
f(df) # "hey"
```
---
### Author:  [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)
#### Post date: [October 26, 2020, 3:10pm UTC](https://discourse.julialang.org/t/enforcing-schema-on-data-frame-passed-as-function-argument/49020/7 "2020-10-26T15:10:34Z")
There are a couple of other options that haven’t been mentioned.
#### Wrap a data frame in a struct
You could create various custom types with constructors that represent different tables in your data flow. These custom types with constructors can provide some of the safety you are looking for. However, I’ve tried this before and the thing I didn’t like about it is that all the actions become nouns (the constructors) rather than verbs.
#### Let it fail
This is the approach I’ve taken. When I’m writing internal code (in other words, code that is not user facing), I can just assume that I, the programmer, am smart enough to provide the right kind of data frame. In fact, this approach applies to a lot of things. When you are writing internal code, you have to assume to some degree that the developer (not the user) knows what they are doing and is providing the correct input to functions. You typically don’t fill your code with assertions (`@assert`) in every function.
---
### Author:  [@gyrfalcon](https://discourse.julialang.org/u/gyrfalcon)
#### Post date: [October 27, 2020, 12:24am UTC](https://discourse.julialang.org/t/enforcing-schema-on-data-frame-passed-as-function-argument/49020/8 "2020-10-27T00:24:54Z")
I think for now I will be using @CameronBieganek’s suggestion to simply let it fail. The code I am working on now should not be user facing (I think). That said, I think something along the lines of what @lungben described would be best for my particular application, I also don’t care if there is more there, or if they are in a different order, only that a minimum set for the given function is present.