# Data

**URL:** https://discourse.julialang.org/c/domain/data/16.md?page=33

[Latest](https://discourse.julialang.org/latest.md) · [Categories](https://discourse.julialang.org/categories.md) · [Tags](https://discourse.julialang.org/tags.md)

**Page:** 34

---

## [Returning Array instead of DataFrame with ODBC](https://discourse.julialang.org/t/returning-array-instead-of-dataframe-with-odbc/9489)

<div class="topic-metadata">

**Author:** [@Max-Zara](https://discourse.julialang.org/u/Max-Zara)\
**Replies:** 0\
**Last updated:** [March 4, 2018, 2:44pm UTC](https://discourse.julialang.org/t/returning-array-instead-of-dataframe-with-odbc/9489 "2018-03-04T14:44:04Z")

</div>

Hi, If I want to run a query from ODBC connected to PostgreSQL, but don’t want to receive a DataFrame, how do I go about doing this? using ODBC.query(dsn\_connection, “my query string…”, sink =??) I have tried sink = O…

---

## [KeyedTuple and inference](https://discourse.julialang.org/t/keyedtuple-and-inference/9059)

<div class="topic-metadata">

**Author:** [@bramtayl](https://discourse.julialang.org/u/bramtayl)\
**Replies:** 12\
**Last updated:** [March 3, 2018, 6:17pm UTC](https://discourse.julialang.org/t/keyedtuple-and-inference/9059 "2018-03-03T18:17:36Z")

</div>

I wonder if you’ve tried with the named tuple implementation in Keys.jl (v0.0.1, not master). That’s built on tuples.

---

## [Online/out-of-core machine learning (ML) algorithms needs to compete with H20 & Spark](https://discourse.julialang.org/t/online-out-of-core-machine-learning-ml-algorithms-needs-to-compete-with-h20-spark/9382)

<div class="topic-metadata">

**Author:** [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Replies:** 13\
**Last updated:** [March 1, 2018, 7:05pm UTC](https://discourse.julialang.org/t/online-out-of-core-machine-learning-ml-algorithms-needs-to-compete-with-h20-spark/9382 "2018-03-01T19:05:53Z")

</div>

I was just thinking about what is needed in Julia in order for it to compete with H20 and Spark and be a serious contender in enterprise ML. I think JuliaDB.jl can handle the data manipulation aspects and OnlineStats.jl…

---

## [Field as Variable in Query.jl](https://discourse.julialang.org/t/field-as-variable-in-query-jl/9365)

<div class="topic-metadata">

**Author:** [@RobertR](https://discourse.julialang.org/u/RobertR)\
**Replies:** 5\
**Last updated:** [March 1, 2018, 9:28am UTC](https://discourse.julialang.org/t/field-as-variable-in-query-jl/9365 "2018-03-01T09:28:54Z")

</div>

I’m a bit confused about why something (admittedly stupid) I was trying to do didn’t work. Using Query.jl you can write something like: x = @from i in df begin @where i.age\>50 @select {i.name, i.children} @…

---

## [Data Frames for non null data](https://discourse.julialang.org/t/data-frames-for-non-null-data/9266)

<div class="topic-metadata">

**Author:** [@Jupiterjosh](https://discourse.julialang.org/u/Jupiterjosh)\
**Replies:** 4\
**Last updated:** [February 23, 2018, 8:13pm UTC](https://discourse.julialang.org/t/data-frames-for-non-null-data/9266 "2018-02-23T20:13:36Z")

</div>

Is there a “dataframes” for data that you know will not be null? I have used both datatables and dataframes and like their syntactic sugar, but I find myself fussing with nullable arrays or missing.missing issues too mu…

---

## [How to permute the rows of a DataFrame in-place efficiently?](https://discourse.julialang.org/t/how-to-permute-the-rows-of-a-dataframe-in-place-efficiently/8825)

<div class="topic-metadata">

**Author:** [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Replies:** 11\
**Last updated:** [February 20, 2018, 11:29pm UTC](https://discourse.julialang.org/t/how-to-permute-the-rows-of-a-dataframe-in-place-efficiently/8825 "2018-02-20T23:29:27Z")

</div>

I have a vector of row numbers and I want to use it to permute a DataFrame’s rows. Here is an MVE using StatsBase df = DataFrame(a = rand(1\_000\_000)) r=sample(1:size(df,1), size(df,1), replace=false) @time df = df\[r,:\]…

---

## [How to read a table with missing values using CSV?](https://discourse.julialang.org/t/how-to-read-a-table-with-missing-values-using-csv/7556)

<div class="topic-metadata">

**Author:** [@Fred](https://discourse.julialang.org/u/Fred)\
**Replies:** 9\
**Last updated:** [February 20, 2018, 4:16pm UTC](https://discourse.julialang.org/t/how-to-read-a-table-with-missing-values-using-csv/7556 "2018-02-20T16:16:35Z")

</div>

Hi, I read a table where some values are missing using : f = CSV.read(“data/$file”) When the missing values is ‘NA’ in the table, I obtain : ERROR: CSV.ParsingException(“error parsing a Float64 value on column 27, ro…

---

## [Fill!() vs for loop](https://discourse.julialang.org/t/fill-vs-for-loop/9194)

<div class="topic-metadata">

**Author:** [@abshej](https://discourse.julialang.org/u/abshej)\
**Replies:** 15\
**Last updated:** [February 20, 2018, 10:56am UTC](https://discourse.julialang.org/t/fill-vs-for-loop/9194 "2018-02-20T10:56:58Z")

</div>

Technically there shouldnt be a difference between using fill! to make every element of a matrix a SMCSC and using a for loop to do so. But using fill!() messes up code. As in it seems like two copies are being made som…

---

## [Performing multiple melt/stack operations on a dataset simultaneously](https://discourse.julialang.org/t/performing-multiple-melt-stack-operations-on-a-dataset-simultaneously/8947)

<div class="topic-metadata">

**Author:** [@Raf](https://discourse.julialang.org/u/Raf)\
**Replies:** 5\
**Last updated:** [February 16, 2018, 9:18am UTC](https://discourse.julialang.org/t/performing-multiple-melt-stack-operations-on-a-dataset-simultaneously/8947 "2018-02-16T09:18:54Z")

</div>

I’m importing some R data so decided to learn DataFrames manipulation in julia instead of doing it in R. I have a wide dataframe with some initial columns followed by four variables that span 12 columns, one for each mon…

---

## [Count number of substrings in dataframe column](https://discourse.julialang.org/t/count-number-of-substrings-in-dataframe-column/9036)

<div class="topic-metadata">

**Author:** [@jacobcvt12](https://discourse.julialang.org/u/jacobcvt12)\
**Replies:** 8\
**Last updated:** [February 13, 2018, 11:07pm UTC](https://discourse.julialang.org/t/count-number-of-substrings-in-dataframe-column/9036 "2018-02-13T23:07:38Z")

</div>

I have a dataset (read in from a CSV). I’m trying to count how many observations contain a substring (for simplicity, let’s call it “X”). In R, I can do this sum(stringr::str\_count(data$CITY, "X")) \> 0 I’m having troub…

---

## [Recommended way to save and read DataFrames in JSON format](https://discourse.julialang.org/t/recommended-way-to-save-and-read-dataframes-in-json-format/8990)

<div class="topic-metadata">

**Author:** [@piever](https://discourse.julialang.org/u/piever)\
**Replies:** 8\
**Last updated:** [February 12, 2018, 6:33pm UTC](https://discourse.julialang.org/t/recommended-way-to-save-and-read-dataframes-in-json-format/8990 "2018-02-12T18:33:24Z")

</div>

I need to save some tabular data in JSON format (from a game engine) to read it in Julia as a DataFrame (or IndexedTable). I’m trying to understand what is the best way to save this kind of data in JSON. The following tw…

---

## [Is there a way that I can attach string labels to integer values in CategoricalArrays?](https://discourse.julialang.org/t/is-there-a-way-that-i-can-attach-string-labels-to-integer-values-in-categoricalarrays/8970)

<div class="topic-metadata">

**Author:** [@mwsohn](https://discourse.julialang.org/u/mwsohn)\
**Replies:** 9\
**Last updated:** [February 11, 2018, 10:24pm UTC](https://discourse.julialang.org/t/is-there-a-way-that-i-can-attach-string-labels-to-integer-values-in-categoricalarrays/8970 "2018-02-11T22:24:34Z")

</div>

I noticed that CategoricalPool type allows labels for categorical values. I am wondering whether there is a way that I can convert an integer variable to a Categorical Array with string labels corresponding to each integ…

---

## [Iterate over all numeric columns in DataFrames](https://discourse.julialang.org/t/iterate-over-all-numeric-columns-in-dataframes/8916)

<div class="topic-metadata">

**Author:** [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Replies:** 21\
**Last updated:** [February 11, 2018, 6:12am UTC](https://discourse.julialang.org/t/iterate-over-all-numeric-columns-in-dataframes/8916 "2018-02-11T06:12:58Z")

</div>

beginner’s questions: how do I iterate over all numeric columns in a DataFrame, e.g., to calculate means or sum-cubed? bonus question: can DataFrames consider NaN to be a missing value for Float32 and Float64, too,…

---

## [Group, Mutate, Ungroup](https://discourse.julialang.org/t/group-mutate-ungroup/8819)

<div class="topic-metadata">

**Author:** [@RandomString123](https://discourse.julialang.org/u/RandomString123)\
**Replies:** 2\
**Last updated:** [February 4, 2018, 11:18pm UTC](https://discourse.julialang.org/t/group-mutate-ungroup/8819 "2018-02-04T23:18:35Z")

</div>

Is there a way developed to support the following workflow? I realize this example is trivial but it could be applied to more complex workflows like subgroup moving averages. And while there are ways to run calculations…

---

## [Concatenate CSV data into Feather using DataStreams](https://discourse.julialang.org/t/concatenate-csv-data-into-feather-using-datastreams/3690)

<div class="topic-metadata">

**Author:** [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Replies:** 5\
**Last updated:** [February 1, 2018, 6:50am UTC](https://discourse.julialang.org/t/concatenate-csv-data-into-feather-using-datastreams/3690 "2018-02-01T06:50:32Z")

</div>

I have about 400G of data in 15 .csv.gz files, overall 2e9 records with 20 columns, same schema in each file. I would like to read all of them and save in a single .feather file for further analysis. I could not figure …

---

## [Various by-group strategies compared](https://discourse.julialang.org/t/various-by-group-strategies-compared/6594)

<div class="topic-metadata">

**Author:** [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Replies:** 36\
**Last updated:** [January 30, 2018, 11:28pm UTC](https://discourse.julialang.org/t/various-by-group-strategies-compared/6594 "2018-01-30T23:28:40Z")

</div>

Update 12th Nov 2017 2nd Update Implemented a multi-threaded that is almost 2x faster than data.table! Implemented a new radixgroup that is even faster than the previous method by 18% on 250 million element arrays. Up…

---

## [Julia Support for File Loading](https://discourse.julialang.org/t/julia-support-for-file-loading/8673)

<div class="topic-metadata">

**Author:** [@RandomString123](https://discourse.julialang.org/u/RandomString123)\
**Replies:** 22\
**Last updated:** [January 30, 2018, 2:07pm UTC](https://discourse.julialang.org/t/julia-support-for-file-loading/8673 "2018-01-30T14:07:58Z")

</div>

As a first learning project in Julia I have started to try to convert a project I did in R into Julia. And it turns out I quickly ran into a couple roadblocks. All of which are related to reading in legacy data files l…

---

## [Replicating a useful Stata Workflow with DataFramesMeta](https://discourse.julialang.org/t/replicating-a-useful-stata-workflow-with-dataframesmeta/8671)

<div class="topic-metadata">

**Author:** [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Replies:** 6\
**Last updated:** [January 30, 2018, 2:07pm UTC](https://discourse.julialang.org/t/replicating-a-useful-stata-workflow-with-dataframesmeta/8671 "2018-01-30T14:07:20Z")

</div>

One thing that is necessary when working with survey data is being able create new variables by standardizing capping existing variables variables. One workflow might look like this: sysuse auto local variables\_to\_cap …

---

## [Progress towards faster \`sortperm\` for Strings](https://discourse.julialang.org/t/progress-towards-faster-sortperm-for-strings/8505)

<div class="topic-metadata">

**Author:** [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Replies:** 16\
**Last updated:** [January 29, 2018, 4:45am UTC](https://discourse.julialang.org/t/progress-towards-faster-sortperm-for-strings/8505 "2018-01-29T04:45:39Z")

</div>

I have just implemented an idea I had to make sortperm faster for Strings. It is implemented in SortingLab.jl’s fsortperm; and I got the performance on a particular benchmark to be 7x faster than Base.sortperm, see code …

---

## [Append!() function extremely slow in DataFrames + CSV](https://discourse.julialang.org/t/append-function-extremely-slow-in-dataframes-csv/8396)

<div class="topic-metadata">

**Author:** [@Fred](https://discourse.julialang.org/u/Fred)\
**Replies:** 14\
**Last updated:** [January 16, 2018, 2:21pm UTC](https://discourse.julialang.org/t/append-function-extremely-slow-in-dataframes-csv/8396 "2018-01-16T14:21:16Z")

</div>

Hi, When I switched to CSV instead of readtable() to read tabular data, I was surprised to see an extreme slowdown. I attached a small benchmark which seems to indicate that the slowdown is in the append!() function. T…

---

## [Write CSV with nulls from dataframe](https://discourse.julialang.org/t/write-csv-with-nulls-from-dataframe/8167)

<div class="topic-metadata">

**Author:** [@jonjilla](https://discourse.julialang.org/u/jonjilla)\
**Replies:** 3\
**Last updated:** [January 5, 2018, 7:49am UTC](https://discourse.julialang.org/t/write-csv-with-nulls-from-dataframe/8167 "2018-01-05T07:49:43Z")

</div>

I’m trying to write a CSV file using the CSV.jl package but am getting this error julia\> CSV.write(“/tmp/out.csv”,testmap) ERROR: TypeError: streamto!: in typeassert, expected String, got DataArrays.NAtype I thought t…

---

## [ANN: SASLib.jl](https://discourse.julialang.org/t/ann-saslib-jl/8030)

<div class="topic-metadata">

**Author:** [@tk3369](https://discourse.julialang.org/u/tk3369)\
**Replies:** 14\
**Last updated:** [December 30, 2017, 12:45am UTC](https://discourse.julialang.org/t/ann-saslib-jl/8030 "2017-12-30T00:45:50Z")

</div>

Hello everyone! I am happy to announce a new package SASLib.jl. Please see the github project page for more information. -Tom https://github.com/tk3369/SASLib.jl

---

## [Replace missing values with .= in DataFrame](https://discourse.julialang.org/t/replace-missing-values-with-in-dataframe/8039)

<div class="topic-metadata">

**Author:** [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Replies:** 2\
**Last updated:** [December 28, 2017, 3:37pm UTC](https://discourse.julialang.org/t/replace-missing-values-with-in-dataframe/8039 "2017-12-28T15:37:35Z")

</div>

The following is an MWE of a problem in which I wanted to replace some missing values for a specific set of observations in a DataFrame. It works with =, but not with the .=. Is this a bug? If not, can someone please cl…

---

## [Is anyone familiar with MapD.com?](https://discourse.julialang.org/t/is-anyone-familiar-with-mapd-com/8038)

<div class="topic-metadata">

**Author:** [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Replies:** 2\
**Last updated:** [December 28, 2017, 2:43pm UTC](https://discourse.julialang.org/t/is-anyone-familiar-with-mapd-com/8038 "2017-12-28T14:43:42Z")

</div>

Does anyone in the Julia community deal with MapD.com at all? It is a GPU database company and it uses LLVM which seems to be a natural fit for Julia.

---

## [Pandas.series.str.extract equivalent?](https://discourse.julialang.org/t/pandas-series-str-extract-equivalent/8016)

<div class="topic-metadata">

**Author:** [@magrathean](https://discourse.julialang.org/u/magrathean)\
**Replies:** 2\
**Last updated:** [December 27, 2017, 5:18pm UTC](https://discourse.julialang.org/t/pandas-series-str-extract-equivalent/8016 "2017-12-27T17:18:01Z")

</div>

How do I do the following pandas manipulation on a Julia Dataframe column: \>\>\> s = Series(\['a1', 'b2', 'c3'\]) \>\>\> s.str.extract('(?P\<letter\>\[ab\])(?P\<digit\>\\d)') letter digit 0 a 1 1 b 2 2 NaN Na…

---

## [Announcement: An Update on DataFrames Future Plans](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646)

<div class="topic-metadata">

**Author:** [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Replies:** 40\
**Last updated:** [December 27, 2017, 5:03pm UTC](https://discourse.julialang.org/t/announcement-an-update-on-dataframes-future-plans/4646 "2017-12-27T17:03:42Z")

</div>

This post summarizes the current roadmap for DataFrames.jl and the data ecosystem in general for the Julia 1.0 milestone. Plans have changed significantly since the previous post. Indeed, it has appeared that representin…

---

## [JLD unable to load files from separate folders? ](https://discourse.julialang.org/t/jld-unable-to-load-files-from-separate-folders/7886)

<div class="topic-metadata">

**Author:** [@gaomrx](https://discourse.julialang.org/u/gaomrx)\
**Replies:** 0\
**Last updated:** [December 20, 2017, 8:48pm UTC](https://discourse.julialang.org/t/jld-unable-to-load-files-from-separate-folders/7886 "2017-12-20T20:48:35Z")

</div>

Duplicate of open issue #201 on Github

---

## [DataFrame unmelt-like operation](https://discourse.julialang.org/t/dataframe-unmelt-like-operation/7875)

<div class="topic-metadata">

**Author:** [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Replies:** 1\
**Last updated:** [December 20, 2017, 11:53am UTC](https://discourse.julialang.org/t/dataframe-unmelt-like-operation/7875 "2017-12-20T11:53:49Z")

</div>

I have a DataFrame with observations for 2 individuals in a household, and want to “unmelt” it so that they end up in columns, one for each person. MWE: julia\> df = DataFrame(household = repeat(1:5, inner = 2), …

---

## [DataFrames 0.11 released](https://discourse.julialang.org/t/dataframes-0-11-released/7296)

<div class="topic-metadata">

**Author:** [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Replies:** 23\
**Last updated:** [December 19, 2017, 1:19pm UTC](https://discourse.julialang.org/t/dataframes-0-11-released/7296 "2017-12-19T13:19:00Z")

</div>

After a long and complex development period, we are glad to announce that version 0.11.0 of DataFrames has been released. Among other features listed in the release notes, the major change introduced by this version is t…

---

## [Compatibility of Query and Union{T, Missing}](https://discourse.julialang.org/t/compatibility-of-query-and-union-t-missing/7362)

<div class="topic-metadata">

**Author:** [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Replies:** 3\
**Last updated:** [November 28, 2017, 5:13pm UTC](https://discourse.julialang.org/t/compatibility-of-query-and-union-t-missing/7362 "2017-11-28T17:13:19Z")

</div>

I just posted this on the slack channel re status of Query/IterableTables/Missings: I don’t see a path for Query and iterable tables to use missing in the julia 1.0 time frame. My read of that discussion so far is that…

[Previous page](https://discourse.julialang.org/c/domain/data/16.md?page=32)

[Next page](https://discourse.julialang.org/c/domain/data/16.md?page=34)