Just for a reference. You can assign to a new column in general like:
df = DataFrame(a=[1,2,3])
df[!, :b] .= 1
works OK. The only problematic cases were zero column data frame and an empty data frame with more than zero columns (they are currently handled in the same way in the PR I have mentioned).
Thanks for the explanation. In your example, my naive expectation would be that df[!,:col] .= 1 creates a column of Int64[] called col with a single element of 1:
df
1×1 DataFrame
│ Row │ a │
│ │ Int64 │
├─────┼───────┤
│ 1 │ 1 │
It is somewhat of an edge case. Although I would expect it to create a new column and assign 1, that may not be consistent with the current behavior, where it would create a column of size R where R is the number of rows. For example:
using DataFrames
df = DataFrame()
R = 10
df[!,:a] = rand( R)
df[!,:b] .= .4
df
10×2 DataFrame
│ Row │ a │ b │
│ │ Float64 │ Float64 │
├─────┼──────────┼─────────┤
│ 1 │ 0.859254 │ 0.4 │
│ 2 │ 0.931467 │ 0.4 │
│ 3 │ 0.276189 │ 0.4 │
│ 4 │ 0.695778 │ 0.4 │
│ 5 │ 0.436747 │ 0.4 │
│ 6 │ 0.478671 │ 0.4 │
│ 7 │ 0.220407 │ 0.4 │
│ 8 │ 0.778795 │ 0.4 │
│ 9 │ 0.614729 │ 0.4 │
│ 10 │ 0.825508 │ 0.4 │
I’m not entirely certain what should be done when R = 0 or R is undefined because the dataframe is empty On one hand, it seems odd to create an empty column when a specific value is being supplied, but assigning the value does not seem entirely consistent with the behavior above either. I guess, between the two options, I might choose to have it create the column and assign the value and use something like df[!,:a] .= Int64[ ] to create an empty column. Of course, that is my somewhat under-informed assessment, and I have not thought about all of the ramifications.
The current behavior is slightly inconvenient because I have to wrap scalars in an array, at least when creating the initial column, e.g. df = DataFrame(); df[!,:a] = [.3]. After that, as you pointed out in your second comment to me, it is possible to assign values subsequently. That’s not a big deal if I always know where the first column will be created. What is more problematic is if v could be a scalar or vector, and the DataFrame is empty. As far as I know, this would require two sets of syntax to handle both cases, whereas before, it handled the assignment of both scalars and vectors.
I agree with you. I’m not sure how often this problem arises in actual code. I have encountered it from time to time in my work. This occurs when I need to extract data from various sources and integrate it with some container iteratively and flexibly (i.e. I may not know what the columns will be or what their sizes will ultimately be).
P.S. I couldn’t adapt your example with push to my problem:
Case 1: scalar
df = DataFrame()
a = 1
push!(df,(a=a,))
df
1×1 DataFrame
│ Row │ a │
│ │ Int64 │
├─────┼───────┤
│ 1 │ 1 │
Case 2: vector
df = DataFrame()
a= [1,2]
push!(df,(a=a,))
df
1×1 DataFrame
│ Row │ a │
│ │ Array… │
├─────┼────────┤
│ 1 │ [1, 2] │
I tried various placements of the "..." operator, but it would not accommodate both cases.
We could treat DataFrame() as having undefined number of rows (as opposed to 0 rows).
Thinking about it we then should also allow:
df = DataFrame()
df[!, :c] .= [1,2,3]
(and create a 3 element vector - now it throws an error; I will update the PR I have mentioned above and we can discuss it further there)
PS. push! is only for scalars
Thanks for your time. I found the discussion helpful for understanding DataFrames.
I have implemented the requested changes in https://github.com/JuliaData/DataFrames.jl/pull/1890. The functionality is tricky so if someone can have a look at it it would be appreciated.
Thank you. I will take a look.
Based on the feedback after 0.19.0 release we have a patch 0.19.1 release that introduces the following changes:
push! and append! now make sure they do not produce an output data frame with unequal number of rows in columns (this was a common problem causing hard-to-catch bugs)join, groupby and show-related functions now check if the data frame they work on is not internally corrupted (most often the problem is unequal number of rows in columns); this allows users to track possible column aliasing related bugs in their code easier0-row (empty) data frame and correctly handles broadcasting into a single cell of a data frame.Thanks for all that reported problems and contributed!
You can see the detailed release notes here.
We have just released a 0.19.2 patch release. The full log of changes can be found here.
The most relevant changes are:
disallowmissing , allowmissing and categorical functions were addedunstack now accepts renamecols keyword argument providing a flexible mechanizm to generate column names of an unstacked data frameI found some weird syntax in DataFrame 0.19.2… is it me doing wrong (i.e. “is there a better way to do it”) or it is supposed to be like this ?
Let’s assume the following df:
df = DataFrame(a=[1,2,3],b=[4,5,6],c=[7,8,9])
# Change column order
df = df[[:b, :a, :c]] # before
df = df[:,[:b, :a, :c]] # now
# Operate on a column whose name is memorised in a variable
C = :c; eltype(df[C]) # before
C = :c; eltype(df[!,C]) # now
# Delete cols
deletecols!(df,:b) # before
select!(df,Not(:b)) # now
What the new ! symbol stand for ? How df[!,:a] is different than df[:,:a] ?
! roughly stands for “a direct access to a column stored in a data frame”, as opposed to : which stands for “access to a copy of a column stored in a data frame”.
The detailed rules are laid out here.
The reason for this change is described above in this thread (around post 51), but in short a direct access to a column lead to many nasty bugs when using DataFrames.jl package (people accessed the column, then mutated it and they were not aware that they also mutated the contents of the source data frame). The ! symbol adds a bit of verbosity, but is meant to warn that the operation is potentially mutating (so this similar with the meaning of ! in Base in function names).
Note that instead of df[!, :col] you can just write df.col.
Also note that:
permutecols! function which works in place; if you want to avoid copying of columns and create a new data frame use df[!, [:b, :a, :c]] (this is potentially unsafe though - as explained above)deletecols! was deprecated because select! just does the job with Not indexing, so there would be a duplication of functionality (and select-family of functions is something that many people already know from other data processing ecosystems).I write to say that in my opinion the new syntax is nice. The symbol ! inside the selection is a bit strange at first, but after you got use it has a lot of sense. The other changes are also good in mi opinion.
I have a small announcement. DataFrames Tutorial is for some time in sync with DataFrames.jl 0.19 release. But I have just added in the 04_loadsave.ipynb notebook examples of integration with JSONTables.jl.
@quinnj has done a fantastic job with JSONTables.jl and with the release 0.1.2 of this package you can safely read and write JSON data (both row- and column- oriented) to a DataFrame.
FWIW, I think it would really be worth adding a Project + Manifest to that repo. Then there is no problem getting the exact versions that were used when running the notebooks.
Good point - I will add this (I started doing these tutorials before Project.toml was an option).
If you have a few minutes to spare, @bkamins, could you cast your expert eye over the wiki books chapter to see if I’ve made any howlers? It dates back a long way (2015) but I’m sentimental and don’t want to just delete the whole section and point to your (much better) material. (Besides, I’d have to redo all the links. 
)
(Just mention any problems, don’t waste time learning the Wikibooks markup language . 
)
Hi All,
After almost half a year from 0.19 release we made it to 0.20. It is really a big release (79 PRs merged since 0.19).
Firstly, I would like to thank all that contributed to it. A chief person to mention is @nalimilan who constantly has been curating the package. The number of contributors in the period was so large that when I thought about listing all who were involved in the transition from 0.19 to 0.20 it was really hard. I have managed to pull out two groups of logins from GitHub (and apologies if I missed someone - I will gladly correct the list; I dropped @ in front of the logins as Discourse disallows me to mention them all directly in this post, as there are too many of them 
):
Both lists were impressive for me (I have not expected such a big contributing community), so it seems that DataFrame.jl is going strong. Thank you all for working on it.
Now, as this release is so big I am summarizing here only the major changes from 0.19 to 0.20:
: as row selectorDataFrame is now correctGroupedDataFrame is now reserveddf[!, cols] is now allowed in setindex! and broadcasting assignmentDataFrameRow
describe now shows actual eltype of the columnallowmissing, disallowmissing and categorical functionsunstack now allows for renaming of the columnsTuple as a value of on keyword argument in join is now deprecatedPair instead)Between, All and Not are allowed in indexingDataFrameRows and DataFrameColumns now have custom show methodsDataFrameRows and DataFrameColumns support getproperty nowcolumnindex from Tables.jl is now exportedcategorical! now allows types as cols argumentmakeunique=true for grouping keys in combine and map
by now has skipmissing keyword argumentpush!, append! and vcat to make them more consistentmapcols now never reuses source columns without copyingsort performancedisallowmissing! and disallowmissing now accept error keyword argumentselect and select! now allow passing multiple columns argumentspermutecols! is deprecated (use select! instead)row_group_slots
join now accepts more than two data frames to be joinedrename! now allows permutation of column names in renamingnames! is deprecated (use rename!)aggregate now accepts skipmissing
gropuby now accepts cols argument to be an empty vectorcopycols in DataFrame constructors is now more permissivecopycols=false)join now allows mixing Symbol and Pair{Symbol, Symbol} in on keyword argumentflatten functiondescribe now has cols keyword argumentArray for data frame and data frame rowrename! and rename now accept strings and integers (apart from Symbols) for renamingio support in describe is deprecatedmelt is now deprecated (use stack with Not selector instead)stackdf and meltdef are now deprecated (use view=true in stack insteead)GroupedDataFrame now supports keys with information on values of grouping columns;GroupedDataFrame indexing (also Tuple or NamedTuple are allowed)Finally, we are fairly close to 1.0 release. Here is a list of current to-do things for 1.0, so as you can see it is not very long (probably it will grow a bit on the way). I plan to have a 0.21 release as a beta before 1.0.
EDIT
The DataFrames Tutorial is now updated and reflects the changes in 0.20.0 release.
Thank you to @bkamins, @quinnj, @nalimilan and other contributors for the continued updates. Over the last few months, updates to DataFrames, CSV and 1.2->1.3 have led to about an 80% speed increase in my data-intensive code, with gains spread fairly evenly across different types of IO operations, computations, and data manipulations.
Whoa… eye-popping improvement! I compared pandas and DataFrames/CSV just now. For 1GB csv file reading, pandas took 13 seconds while DataFrames/CSV took only 1.8 seconds. Thank contributors very much. You are the geniuses.