I am trying to construct a DataFrame from an array of named tuples where some key:value pairs may be missing from some tuples. They come from querying Airtable using Airtable,jl. If you have missing values in your table, they are not present in the data returned by the airtable API. This is what a…

One way would be: cols = union(keys.(data)...) df = DataFrame([c => get.(data,c, missing) for c in cols]...) which gives: 4×3 DataFrame Row │ Number Name Address │ Int64 String String? ─────┼────────────────────────────────── 1 │ 4 Abc Efgg 48 Mont Rd …

[image] Dan: cols = union(keys.(data)...) Thank you for your kind reply. get! I forgot about that. Regards David

Probably this scheme is valid also without the intervention of the foraeach function (that is, using only the named tuples and some splatting/broadcasting), but I haven’t found the way yet df=DataFrame() foreach(d->push!(df,d,cols=:union),data) i would have expected that push! worked the same way…

Thanks for your reply. I will give it a go my tomorrow. Julia is an interesting language for sure. Regards David

Yes Tables.dictrowtable is the intended way to handle this: julia> DataFrame(Tables.dictrowtable(data)) 4×3 DataFrame Row │ Number Name Address │ Int64 String String? ─────┼────────────────────────────────── 1 │ 4 Abc Efgg 48 Mont Rd 2 │ 6 Ruf Sly 19A Keke ava …

Of the two unfulfilled expectations, of one (the one related to the function append!(…, cols=:union)) I got an idea of how it works. From the following example I understand that kwarg intervenes to combine internally “homogeneous” blocks (ie a vectors of named tuples with the same fields) but betwe…

[image] rocco_sprmnt21: push!(df,data...,cols=:union) This is not implemented as it strains compiler a lot, foreach should be used instead - just as you have proposed. append!(df1,data[3:4],cols=:union) As I have commented - the reason why this fails is unrelated with DataFrames.jl. This …

[image] bkamins: The problem is that data[3:4] is not a valid Tables.jl table. How to determine that? From reading the docs, it seems to fulfil the requirements for a row-based table.

Isn’t the istable function checking if something is valid Tables.jl table? julia> data = [(Number = 4, Name = "Abc Efgg", Address = "48 Mont Rd"), (Number = 6, Name = "Ruf Sly", Address = "19A Keke ava"), (Number = 10, Name = "Jack Bog"), (Number = 5, Name = "Gid Hoo", Address …

Technically you can check it if you try Tables.columns: julia> Tables.columns(data) ERROR: type NamedTuple has no field Address In the documentation (of dictrowtable) you can read: For “schema-less” input tables, dictrowtable employs a “column unioning” behavior, as opposed to inferring the sche…

[image] rocco_sprmnt21: Isn’t the istable function checking if something is valid Tables.jl table? I would not recommend using Tables.istable in practice (unfortunately). See its docs: Check if an object has specifically defined that it is a table. Note that not all valid tables will return…

Construct DataFrame From Uneven Named Tuples

General Usage

bkamins August 20, 2023, 1:56pm 18

If you feel the behavior should be changed, can you please open an issue in Tables.jl as probably @quinnj should comment on this since he maintains this package.

Topic		Replies	Views
Array of heterogeneous named tuples to DataFrame General Usage	2	447	December 24, 2020
Issue adding a row record of a DataFrame with `String` name to itself General Usage dataframes	5	1141	March 20, 2022
Initializing a dataframe New to Julia	23	11308	March 15, 2020
Appending rows to a dataframe is seemingly inconsistent and confusing Data	11	4992	December 24, 2021
DataFrame construction from array of tuples General Usage data	12	7225	November 28, 2022

Construct DataFrame From Uneven Named Tuples

Related topics