# DataFrames & IJulia?

**URL:** <https://discourse.julialang.org/t/dataframes-ijulia/10118>\
**Category:** Data\
**Created:** [April 2, 2018, 1:04pm UTC](https://discourse.julialang.org/t/dataframes-ijulia/10118 "2018-04-02T13:04:59Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![BLI](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bli/32/37206_2.png) [@BLI](https://discourse.julialang.org/u/BLI)\
**Post date:** [April 2, 2018, 1:04pm UTC](https://discourse.julialang.org/t/dataframes-ijulia/10118/1 "2018-04-02T13:04:59Z")

</div>

I have used pandas in Python to typeset a table in Jupyter notebook (standard Anaconda installation, nothing fancy). The data consisted in a simple list of dictionaries: using pandas, this resulted in a table with dictionary keys as headline, and dictionary values in the rows below.

Question: what is the command for doing the same with data frames in IJulia?

**Example of data** :  
`sr_org.getQuantities()[0:2]`

**leading to** :

` [{'Changeable': 'false', 'Description': 'Initializing temperature in reactor, K', 'Name': 'T', 'Value': None, 'Variability': 'continuous', 'alias': 'noAlias', 'aliasvariable': None}, {'Changeable': 'false', 'Description': 'Initializing concentration of A in reactor, mol/L', 'Name': 'cA', 'Value': None, 'Variability': 'continuous', 'alias': 'noAlias', 'aliasvariable': None}]`

**This looks as follows in Python/Jupyter** (`import pandas as pd`):

 ![image](https://global.discourse-cdn.com/julialang/original/3X/c/0/c02efd975e693642e26bf10cd4ea19cf7bbe571c.png)

---

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [April 2, 2018, 1:35pm UTC](https://discourse.julialang.org/t/dataframes-ijulia/10118/2 "2018-04-02T13:35:05Z")

</div>

I use

```julia
using DataFrames

function df_from_dicts(arr::AbstractArray; missing_value=missing)
    cols = Set{Symbol}()
    for di in arr union!(cols, keys(di)) end
    df = DataFrame()
    for col=cols
        df[col] = [get(di, col, missing_value) for di=arr]
    end
    return df
end

df_from_dicts(your_vector_of_dicts)

```

---

<div class="post-metadata">

**Author:** ![BLI](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bli/32/37206_2.png) [@BLI](https://discourse.julialang.org/u/BLI)\
**Post date:** [April 3, 2018, 8:14pm UTC](https://discourse.julialang.org/t/dataframes-ijulia/10118/3 "2018-04-03T20:14:00Z")

</div>

Thanks, cstjean – I finally figured out your code…

First, I tried with the following example:

```julia
dict1 = Dict("name"=>"William", "surname"=>"Shakespeare", "bornyear"=>1564);
dict2 = Dict("name"=>"Miguel", "surname"=>"de Cervantes", "bornyear"=>1547);
dict3 = Dict("name"=>"William", "surname"=>"Tyndale", "bornyear"=>1484); 
dlist = [dict1,dict2,dict3]; 
df_from_dicts(dlist)

```

Didn’t work, and gave an error message. Then I tried:

```julia
dict1 = Dict(:name=>"William", :surname=>"Shakespeare", :bornyear=>1564);
dict2 = Dict(:name=>"Miguel", :surname=>"de Cervantes", :bornyear=>1547);
dict3 = Dict(:name=>"William", :surname=>"Tyndale", :bornyear=>1484); 
dlist = [dict1,dict2,dict3]; 
df_from_dicts(dlist)

```

and **now** it works.

**Questions** :

1. Is it becoming a _standard_ that `Dict` keys should be symbols? Or is that just a choice?
2. Are there any advantages in using symbols for keys?
3. If no to #1 and #2, how should one generalize the code to handle any case?

-B

---

<div class="post-metadata">

**Author:** ![cstjean](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cstjean/32/1444_2.png) [@cstjean](https://discourse.julialang.org/u/cstjean)\
**Post date:** [April 4, 2018, 12:34am UTC](https://discourse.julialang.org/t/dataframes-ijulia/10118/4 "2018-04-04T00:34:52Z")

</div>

Sorry that my example was a bit terse; I was short on time.

> [@BLI](#):
>
> Is it becoming a standard that Dict keys should be symbols? Or is that just a choice?

DataFrames in Julia use symbols for column names (Pandas uses strings), so it’s logical to use symbols in the input to `df_from_dicts`. Technically, `Dict`s can map from anything to anything.

> [@](#):
>
> Are there any advantages in using symbols for keys?

You mean, in general? AFAIK, symbols in Julia are “interned”, which means that comparing/hashing them is really fast (O(1): like comparing two numbers), whereas string comparison is, I believe, O(N).

> [@](#):
>
> If no to #1 and #2, how should one generalize the code to handle any case?

You could use `df[Symbol(col)] = [get(di, col, missing_value) for di=arr]` in the loop above (and use a `Set{String}()`. As much as possible, I would favour manipulating symbols over strings.

---

<div class="post-metadata">

**Author:** ![BLI](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bli/32/37206_2.png) [@BLI](https://discourse.julialang.org/u/BLI)\
**Post date:** [April 4, 2018, 6:05am UTC](https://discourse.julialang.org/t/dataframes-ijulia/10118/5 "2018-04-04T06:05:11Z")

</div>

Thanks a lot.

---

<div class="post-metadata">

**Author:** ![BLI](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bli/32/37206_2.png) [@BLI](https://discourse.julialang.org/u/BLI)\
**Post date:** [April 7, 2018, 1:29pm UTC](https://discourse.julialang.org/t/dataframes-ijulia/10118/6 "2018-04-07T13:29:43Z")

</div>

OK… a couple of questions about Strings and Symbols… I can convert between strings and symbols as follows:

```julia
julia> sy1 = Symbol("derT")
:derT
julia> String(sy1)
"derT"

```

I can create the same symbol with short-hand colon notation:

```julia
julia> sy2 = :derT
:derT
julia> sy3 = :(derT)
:derT

```

_However_, the following appear to be **different** :

```julia
julia> sy4 = Symbol("der(T)")
Symbol("der(T)")
julia> sy5 = :(der(T))
:(der(T))

```

Attempting to convert back to string leads to:

```julia
julia> String(sy4)
"der(T)"
julia> String(sy5)
MethodError: Cannot `convert` an object of type Expr to an object of type String
This may have arisen from a call to the constructor String(...),
since type constructors fall back to convert methods.

Stacktrace:
 [1] String(::Expr) at .\sysimg.jl:77
 [2] include_string(::String, ::String) at .\loading.jl:522

```

Anyone knows the system/rationale here?

---

<div class="post-metadata">

**Author:** ![jlapeyre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jlapeyre/32/4514_2.png) [@jlapeyre](https://discourse.julialang.org/u/jlapeyre)\
**Post date:** [April 7, 2018, 7:13pm UTC](https://discourse.julialang.org/t/dataframes-ijulia/10118/7 "2018-04-07T19:13:44Z")

</div>

The quoting syntax `:( ... )` returns objects of various types.

Both `:a` and `:(a)` return the symbol `:a`.

`:(1)` returns an `Int` with value `1`.

`:(der(T))` returns an expression, an object of type `Expr`.

```julia
julia> dump(:(der(T)))
Expr
  head: Symbol call
  args: Array{Any}((2,))
    1: Symbol der
    2: Symbol T
  typ: Any

```

These are exactly the objects that Julia returns when parsing code into a syntax tree.

This

```julia
julia> sy4 = Symbol("der(T)")
Symbol("der(T)")

```

creates a `Symbol`. Julia displays all objects as strings in such a way that they can be reconstructed by parsing and evaluating. But, `:(der(T))` gives an expression rather than a symbol. So Julia displays this `Symbol` as `Symbol("der(T)")`.

Finally, you _can_ convert an expression to a string:

```julia
julia> string(:(der(T)))
"der(T)"

```

Note this example uses `string` rather than `String`.

---

<div class="post-metadata">

**Author:** ![BLI](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bli/32/37206_2.png) [@BLI](https://discourse.julialang.org/u/BLI)\
**Post date:** [April 8, 2018, 2:04pm UTC](https://discourse.julialang.org/t/dataframes-ijulia/10118/8 "2018-04-08T14:04:44Z")

</div>

Thanks… the distinction between Symbols and `:(...)` remains somewhat obscure to me… The following two statements lead to different objects:

```julia
julia> Symbol("der(T)")
Symbol("der(T)")
julia> :(der(T))
:(der(T))

```

Both of them are valid `Dict` keys, but none of them are typeset as table headings in `IJulia` using the `df_from_dicts()` function. However, simpler `:(...)` objects are typeset as a table heading…

---

<div class="post-metadata">

**Author:** ![jlapeyre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jlapeyre/32/4514_2.png) [@jlapeyre](https://discourse.julialang.org/u/jlapeyre)\
**Post date:** [April 8, 2018, 4:08pm UTC](https://discourse.julialang.org/t/dataframes-ijulia/10118/9 "2018-04-08T16:08:21Z")

</div>

Yes, it takes a little bit of study. Maybe the best place to start is the “Metaprogramming” section of the Julia manual. In particular, it says

> The : character has two syntactic purposes in Julia.The first form creates a Symbol, an interned string used as one building-block of expressions.

This does not seem quite correct to me. The two purposes are really almost the same. For example `:a` and `:(a)` both return the `Symbol` `a`. And `:1` and `:(1)` both return the integer `1`. It’s just that, if an expression is complicated enough, it must be enclosed in parens in order to be parsed correctly.

_Any_ object can be a `Dict` key in Julia. But only `Symbol`s are allowed as the names of columns in `DataFrames`.

A couple of years ago, IIUC, column names were required to be valid identifiers. Apparently its possible to use any symbol now.

If you want to be sure you are constructing a `Symbol` and not an expression, use `Symbol("...")`. For example `Symbol("a")` and `Symbol("1")` return `Symbol`s, in contrast to the the examples using `:` above.

---

<div class="post-metadata">

**Author:** ![BLI](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bli/32/37206_2.png) [@BLI](https://discourse.julialang.org/u/BLI)\
**Post date:** [April 9, 2018, 6:35am UTC](https://discourse.julialang.org/t/dataframes-ijulia/10118/10 "2018-04-09T06:35:36Z")

</div>

Thanks for tips. OK – so column names in `DataFrames` are symbols, then, thus `Symbol("der(T)")` should work, while `:(der(T))` may not work.