# Data

**URL:** https://discourse.julialang.org/c/domain/data/16.md?page=26

[Latest](https://discourse.julialang.org/latest.md) · [Categories](https://discourse.julialang.org/categories.md) · [Tags](https://discourse.julialang.org/tags.md)

**Page:** 27

---

## [Dataset access - Python](https://discourse.julialang.org/t/dataset-access-python/30048)

<div class="topic-metadata">

**Author:** [@johnh](https://discourse.julialang.org/u/johnh)\
**Replies:** 7\
**Last updated:** [October 18, 2019, 9:46pm UTC](https://discourse.julialang.org/t/dataset-access-python/30048 "2019-10-18T21:46:50Z")

</div>

@oxinabox will set me right here. I recall looking at https://github.com/oxinabox/DataDepsGenerators.jl There is a Python standard for accessing datasets which was mentioned at the time. Or am I imagining this? I ask s…

---

## [\[DataFrames Question\]: hash-based row indexing for DataFrames package](https://discourse.julialang.org/t/dataframes-question-hash-based-row-indexing-for-dataframes-package/29925)

<div class="topic-metadata">

**Author:** [@Sijun](https://discourse.julialang.org/u/Sijun)\
**Replies:** 16\
**Last updated:** [October 16, 2019, 1:58am UTC](https://discourse.julialang.org/t/dataframes-question-hash-based-row-indexing-for-dataframes-package/29925 "2019-10-16T01:58:04Z")

</div>

Question: Any plan for hash-based indexing for DataFrame Suppose the following table is given (actual table that I work on contains about a million records): df = DataFrame(id=\[7,1,5,3,4,6,2\], val=\[5,6,9,3,4,10,4\]) Now…

---

## [DataFrames, best way to import from JSON format file](https://discourse.julialang.org/t/dataframes-best-way-to-import-from-json-format-file/29946)

<div class="topic-metadata">

**Author:** [@4aHxKzD](https://discourse.julialang.org/u/4aHxKzD)\
**Replies:** 6\
**Last updated:** [October 15, 2019, 10:55pm UTC](https://discourse.julialang.org/t/dataframes-best-way-to-import-from-json-format-file/29946 "2019-10-15T22:55:03Z")

</div>

Hi. I’m coming from Python and would like to know what is the current “standard” on how to import from a JSON file to a DataFrame data structure, i.e., the equivalent to pandas.read\_json().

---

## [Difference between Tables.jl and DataFrames.jl](https://discourse.julialang.org/t/difference-between-tables-jl-and-dataframes-jl/29944)

<div class="topic-metadata">

**Author:** [@4aHxKzD](https://discourse.julialang.org/u/4aHxKzD)\
**Replies:** 1\
**Last updated:** [October 15, 2019, 3:16pm UTC](https://discourse.julialang.org/t/difference-between-tables-jl-and-dataframes-jl/29944 "2019-10-15T15:16:17Z")

</div>

Hi. Is anyone here kind enough to educate me about what is the difference between Tables.jl and DataFrames.jl packages?

---

## [DataFrames: removing columns with many missing values](https://discourse.julialang.org/t/dataframes-removing-columns-with-many-missing-values/29907)

<div class="topic-metadata">

**Author:** [@johnbb](https://discourse.julialang.org/u/johnbb)\
**Replies:** 2\
**Last updated:** [October 14, 2019, 7:33pm UTC](https://discourse.julialang.org/t/dataframes-removing-columns-with-many-missing-values/29907 "2019-10-14T19:33:08Z")

</div>

How can I remove columns with more than 10% missing data from a DataFrame? In R it would be x\[, colMeans(is.na(x)) \< 0.1\]. Thanks!

---

## [Serious group-by performance issue with Query.jl](https://discourse.julialang.org/t/serious-group-by-performance-issue-with-query-jl/29070)

<div class="topic-metadata">

**Author:** [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Replies:** 26\
**Last updated:** [October 13, 2019, 5:34am UTC](https://discourse.julialang.org/t/serious-group-by-performance-issue-with-query-jl/29070 "2019-10-13T05:34:41Z")

</div>

Wanted to raise a performance issue at Query.jl but got directed here. I think Query.jl group-by is really inefficient and that’s why I never warmed to it. I use FileIO.jl but never Query.jl due to performance issues. E.…

---

## [Hierarchical or multi-index for data frames](https://discourse.julialang.org/t/hierarchical-or-multi-index-for-data-frames/17076)

<div class="topic-metadata">

**Author:** [@MatthiasKauer](https://discourse.julialang.org/u/MatthiasKauer)\
**Replies:** 10\
**Last updated:** [October 9, 2019, 8:23am UTC](https://discourse.julialang.org/t/hierarchical-or-multi-index-for-data-frames/17076 "2019-10-09T08:23:06Z")

</div>

Hi, I’ve been looking around for data frame implementations. I found that GitHub - JuliaData/DataFrames.jl: In-memory tabular data in Julia does not support a hierarchical or multi-index approach. For clarification, I’…

---

## [Ignore consecutive whitespaces with CSV.read(...)](https://discourse.julialang.org/t/ignore-consecutive-whitespaces-with-csv-read/11304)

<div class="topic-metadata">

**Author:** [@e3c6](https://discourse.julialang.org/u/e3c6)\
**Replies:** 18\
**Last updated:** [October 8, 2019, 4:34pm UTC](https://discourse.julialang.org/t/ignore-consecutive-whitespaces-with-csv-read/11304 "2019-10-08T16:34:02Z")

</div>

I am reading a file with CSV.read(...) that contains many consecutive spaces between columns because the colmuns are padded. I passed the keyword argument delim=' ', but it thinks that every space counts as a new column…

---

## [Serious performance issue with MySQL.jl](https://discourse.julialang.org/t/serious-performance-issue-with-mysql-jl/29289)

<div class="topic-metadata">

**Author:** [@Sh4pe](https://discourse.julialang.org/u/Sh4pe)\
**Replies:** 2\
**Last updated:** [October 3, 2019, 9:31pm UTC](https://discourse.julialang.org/t/serious-performance-issue-with-mysql-jl/29289 "2019-10-03T21:31:24Z")

</div>

Hi there! I hope that any MySQL.jl experts are out there. I’ve posted this issue already as #130 there and I hope to increase visibility by adding a topic here as well. In this test on my fork of MySQL.jl, I’ve managed…

---

## [Parallel DataFrame processing](https://discourse.julialang.org/t/parallel-dataframe-processing/29397)

<div class="topic-metadata">

**Author:** [@Fred](https://discourse.julialang.org/u/Fred)\
**Replies:** 2\
**Last updated:** [October 3, 2019, 9:07am UTC](https://discourse.julialang.org/t/parallel-dataframe-processing/29397 "2019-10-03T09:07:43Z")

</div>

Hi ! I try to process DataFrame columns in parallel. I first think using pmap() but it was not possible because it is not just applying a function to each columns. So I created a toy example here producing my error. It…

---

## [JDF - an experimental DataFrame serialization format is ready for beta testing](https://discourse.julialang.org/t/jdf-an-experimental-dataframe-serialization-format-is-ready-for-beta-testing/28722)

<div class="topic-metadata">

**Author:** [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Replies:** 8\
**Last updated:** [September 15, 2019, 11:01pm UTC](https://discourse.julialang.org/t/jdf-an-experimental-dataframe-serialization-format-is-ready-for-beta-testing/28722 "2019-09-15T23:01:56Z")

</div>

https://github.com/xiaodaigh/JDF.jl I wanted to contribute to Parquet.jl and Arrow.jl but don’t feel confident that I know enough about serialization and disk formats. And reading the Parquet and Arrow docs still leaves…

---

## [Queryverse: How to @select columns with specific eltype? (using \`oftype\` function)](https://discourse.julialang.org/t/queryverse-how-to-select-columns-with-specific-eltype-using-oftype-function/28709)

<div class="topic-metadata">

**Author:** [@Anselm\_Jeong](https://discourse.julialang.org/u/Anselm_Jeong)\
**Replies:** 3\
**Last updated:** [September 14, 2019, 10:14am UTC](https://discourse.julialang.org/t/queryverse-how-to-select-columns-with-specific-eltype-using-oftype-function/28709 "2019-09-14T10:14:47Z")

</div>

Dear all Queryverse users. In an auxilary package QueryOperators.NamedTupleUtilities, there is a helper function named oftype. I guess it is related with selecting columns with specific eltype. But I cannot figure out h…

---

## [How to quickly bulk insert into postgres](https://discourse.julialang.org/t/how-to-quickly-bulk-insert-into-postgres/15194)

<div class="topic-metadata">

**Author:** [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Replies:** 14\
**Last updated:** [September 10, 2019, 1:03pm UTC](https://discourse.julialang.org/t/how-to-quickly-bulk-insert-into-postgres/15194 "2019-09-10T13:03:48Z")

</div>

This is sort of a shot in the dark, I don’t have any reason to believe there’s a better solution to this but I thought I’d give it a try. I’m trying to insert data into tables on a postgres database. I’m using the exce…

---

## [How can I handle uuid and jsob in PGSQL](https://discourse.julialang.org/t/how-can-i-handle-uuid-and-jsob-in-pgsql/28539)

<div class="topic-metadata">

**Author:** [@hasanOryx](https://discourse.julialang.org/u/hasanOryx)\
**Replies:** 4\
**Last updated:** [September 9, 2019, 5:35pm UTC](https://discourse.julialang.org/t/how-can-i-handle-uuid-and-jsob-in-pgsql/28539 "2019-09-09T17:35:28Z")

</div>

In my database, I’ve the below structure: CREATE TABLE public.devices ( id uuid, data jsonb ) When I run my query using this package, I got the below, how can I read the output properly?

---

## [Converting a Matrix to a DataFrame](https://discourse.julialang.org/t/converting-a-matrix-to-a-dataframe/6114)

<div class="topic-metadata">

**Author:** [@sylvaticus](https://discourse.julialang.org/u/sylvaticus)\
**Replies:** 4\
**Last updated:** [September 8, 2019, 4:05pm UTC](https://discourse.julialang.org/t/converting-a-matrix-to-a-dataframe/6114 "2019-09-08T16:05:14Z")

</div>

I am struggling to convert a Matrix where I have the first row as header and possible NA values in a DataFrame. Actually I did find at least two ways, and the resulting df seems ok, but in reality the individual columns…

---

## [Review inefficient pipeline?](https://discourse.julialang.org/t/review-inefficient-pipeline/28215)

<div class="topic-metadata">

**Author:** [@danriggins](https://discourse.julialang.org/u/danriggins)\
**Replies:** 2\
**Last updated:** [September 5, 2019, 11:06pm UTC](https://discourse.julialang.org/t/review-inefficient-pipeline/28215 "2019-09-05T23:06:12Z")

</div>

Hi, I’m a new learner, trying to learn/practice data ingestion in Julia. I was wondering if anyone could review this pipeline I created to see if there’s a more efficient way to do it. I ultimately want a JuliaDB table, …

---

## [String Index for DataFrames](https://discourse.julialang.org/t/string-index-for-dataframes/28390)

<div class="topic-metadata">

**Author:** [@yakyak](https://discourse.julialang.org/u/yakyak)\
**Replies:** 1\
**Last updated:** [September 4, 2019, 10:27pm UTC](https://discourse.julialang.org/t/string-index-for-dataframes/28390 "2019-09-04T22:27:52Z")

</div>

I am using the DataFrames package, and I would like to set a string column of it as the index. For example, let x = │ Row │ name │ val│ │ │ String │ String │ ├─────┼────────┼────────┤ │ 1 │ A │ 1│ │ 2 │ B…

---

## [Group-by performance benchmarks and recommendations](https://discourse.julialang.org/t/group-by-performance-benchmarks-and-recommendations/9313)

<div class="topic-metadata">

**Author:** [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Replies:** 12\
**Last updated:** [September 2, 2019, 9:58pm UTC](https://discourse.julialang.org/t/group-by-performance-benchmarks-and-recommendations/9313 "2019-09-02T21:58:48Z")

</div>

I have been trying to improve Julia’s DataFrame group-by for a while now and I think I am able to synthesized my thinking into APIs. Here are my synthesized recommendation (as of 25th Feb 2018) Recommendation Why? B…

---

## [First part benchmarking medium-data tools](https://discourse.julialang.org/t/first-part-benchmarking-medium-data-tools/28260)

<div class="topic-metadata">

**Author:** [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Replies:** 4\
**Last updated:** [September 1, 2019, 9:41pm UTC](https://discourse.julialang.org/t/first-part-benchmarking-medium-data-tools/28260 "2019-09-01T21:41:51Z")

</div>

I compared and contrast JuliaDB with Dask and disk.frame. http://diskframe.com/articles/vs-dask-juliadb.html Probably more to come, but JuliaDB needs to mature before it can be a contender in these benchmarks.

---

## [Most efficient way to compute number of rows in each group using JuliaDB?](https://discourse.julialang.org/t/most-efficient-way-to-compute-number-of-rows-in-each-group-using-juliadb/28256)

<div class="topic-metadata">

**Author:** [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Replies:** 0\
**Last updated:** [September 1, 2019, 8:01am UTC](https://discourse.julialang.org/t/most-efficient-way-to-compute-number-of-rows-in-each-group-using-juliadb/28256 "2019-09-01T08:01:37Z")

</div>

Suppose a is a JuliaDB table, is the below the best (most efficient) way to compute the number of rows per group? using LambdaFn, DataFrames, JuliaDB a = DataFrame(Column1 = rand(\["a", "b", "c"\], 1\_000), b = rand(1000))…

---

## [Trying to analyse Fannie Mae data with JuliaDB](https://discourse.julialang.org/t/trying-to-analyse-fannie-mae-data-with-juliadb/21410)

<div class="topic-metadata">

**Author:** [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Replies:** 14\
**Last updated:** [August 31, 2019, 11:27pm UTC](https://discourse.julialang.org/t/trying-to-analyse-fannie-mae-data-with-juliadb/21410 "2019-08-31T23:27:45Z")

</div>

I am doing my best to analyse Fannie Mae data with JuliaDB. However the f First Step is to download the Fannie Mae Data and unzip them all Now I tried to load the data using loadtable but it didn’t work. So I split th…

---

## [How to quickly bulk insert into postgres 1](https://discourse.julialang.org/t/how-to-quickly-bulk-insert-into-postgres-1/27784)

<div class="topic-metadata">

**Author:** [@datnt2509](https://discourse.julialang.org/u/datnt2509)\
**Replies:** 3\
**Last updated:** [August 29, 2019, 2:12pm UTC](https://discourse.julialang.org/t/how-to-quickly-bulk-insert-into-postgres-1/27784 "2019-08-29T14:12:54Z")

</div>

This is sort of a shot in the dark, I don’t have any reason to believe there’s a better solution to this but I thought I’d give it a try. I’m trying to insert data into tables on a postgres database. I’m using the excel…

---

## [How to "print" a dataframe representation in a string?](https://discourse.julialang.org/t/how-to-print-a-dataframe-representation-in-a-string/28155)

<div class="topic-metadata">

**Author:** [@sylvaticus](https://discourse.julialang.org/u/sylvaticus)\
**Replies:** 3\
**Last updated:** [August 29, 2019, 9:22am UTC](https://discourse.julialang.org/t/how-to-print-a-dataframe-representation-in-a-string/28155 "2019-08-29T09:22:05Z")

</div>

How do I “print” a representation of a dataframe in within a String, e.g. this would not work: df = DataFrame(c1=\["a","b"\],c2=\[1,2\]) myStr = """ My dataframe: $(show(df,allrows=true, allcols=true)) bla,bla,bla... """

---

## [Reading .csv.gz with CSV does not find readavailable(::GZipStream)](https://discourse.julialang.org/t/reading-csv-gz-with-csv-does-not-find-readavailable-gzipstream/27966)

<div class="topic-metadata">

**Author:** [@hmmueller](https://discourse.julialang.org/u/hmmueller)\
**Replies:** 4\
**Last updated:** [August 28, 2019, 5:49pm UTC](https://discourse.julialang.org/t/reading-csv-gz-with-csv-does-not-find-readavailable-gzipstream/27966 "2019-08-28T17:49:38Z")

</div>

I had an original piece of code like this, which worked nicely: filepaths = \[joinpath(root, f) for (root, dirs, files) in walkdir(root) for f in files\[occursin.(fnfeature,…

---

## [Accumulating distributed data](https://discourse.julialang.org/t/accumulating-distributed-data/28117)

<div class="topic-metadata">

**Author:** [@grahamas](https://discourse.julialang.org/u/grahamas)\
**Replies:** 0\
**Last updated:** [August 28, 2019, 3:56pm UTC](https://discourse.julialang.org/t/accumulating-distributed-data/28117 "2019-08-28T15:56:52Z")

</div>

I’m running distributed simulations and hoping to save the results as I go – the aggregate results are too big to hold in memory. Can someone point me to a good way to do this that doesn’t amount to saving each run in a…

---

## [JuliaDB saving arrays](https://discourse.julialang.org/t/juliadb-saving-arrays/28059)

<div class="topic-metadata">

**Author:** [@grahamas](https://discourse.julialang.org/u/grahamas)\
**Replies:** 1\
**Last updated:** [August 27, 2019, 7:35pm UTC](https://discourse.julialang.org/t/juliadb-saving-arrays/28059 "2019-08-27T19:35:37Z")

</div>

I’m running a parameter sweep solving differential equations and trying to save the results, and have a couple questions: When I try to save a toy example as follows, I get an error that \_impl is not defined for my arg…

---

## [DataFrame or DataFrameRow to Dict and JSON](https://discourse.julialang.org/t/dataframe-or-dataframerow-to-dict-and-json/28018)

<div class="topic-metadata">

**Author:** [@milesf](https://discourse.julialang.org/u/milesf)\
**Replies:** 1\
**Last updated:** [August 27, 2019, 6:00am UTC](https://discourse.julialang.org/t/dataframe-or-dataframerow-to-dict-and-json/28018 "2019-08-27T06:00:21Z")

</div>

Wanted to double-check if there are any recommended ways to convert a DataFrame (or DataFrameRow) into a Dict (for eventually conversion to JSON). This is the inverse of this question. “List of Dicts” JSON format would …

---

## [My experiences reading CSVs from the Fannie Mae datasets](https://discourse.julialang.org/t/my-experiences-reading-csvs-from-the-fannie-mae-datasets/8737)

<div class="topic-metadata">

**Author:** [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Replies:** 62\
**Last updated:** [August 26, 2019, 7:41am UTC](https://discourse.julialang.org/t/my-experiences-reading-csvs-from-the-fannie-mae-datasets/8737 "2019-08-26T07:41:02Z")

</div>

I am pretty excited about the Fannie Mae data being released so I tried to load the data into Julia. Unfortunately, it’s too much work trying to produce a MWE as being able to load large datasets is part of the point. Bu…

---

## [Common interface for chunked arrays?](https://discourse.julialang.org/t/common-interface-for-chunked-arrays/26009)

<div class="topic-metadata">

**Author:** [@fabiangans](https://discourse.julialang.org/u/fabiangans)\
**Replies:** 10\
**Last updated:** [August 26, 2019, 6:56am UTC](https://discourse.julialang.org/t/common-interface-for-chunked-arrays/26009 "2019-08-26T06:56:16Z")

</div>

Hi all, there is quite a zoo of data formats that allow you to store multidimensional arrays tiled into chunks, with or without compression. Examples are HDF5.jl, NetCDF.jl, Zarr.jl, or BigArrays.jl, maybe there will be…

---

## [Trouble creating new column in grouped object](https://discourse.julialang.org/t/trouble-creating-new-column-in-grouped-object/27982)

<div class="topic-metadata">

**Author:** [@tlnagy](https://discourse.julialang.org/u/tlnagy)\
**Replies:** 1\
**Last updated:** [August 26, 2019, 5:15am UTC](https://discourse.julialang.org/t/trouble-creating-new-column-in-grouped-object/27982 "2019-08-26T05:15:28Z")

</div>

using Query using DataFrames ex\_df = DataFrame(id = \[1,1,1,2,2,2,2\], trip= \["a", "a", "b", "a", "a", "b", "b"\], values=rand(7)) function consect(d) # 0 if consecutive values are the same, 1 otherwise Int.(view(…

[Previous page](https://discourse.julialang.org/c/domain/data/16.md?page=25)

[Next page](https://discourse.julialang.org/c/domain/data/16.md?page=27)