# Data

**URL:** https://discourse.julialang.org/c/domain/data/16.md?page=30

[Latest](https://discourse.julialang.org/latest.md) · [Categories](https://discourse.julialang.org/categories.md) · [Tags](https://discourse.julialang.org/tags.md)

**Page:** 31

---

## [\[ANN\] Schemata.jl ready on Julia 1.0](https://discourse.julialang.org/t/ann-schemata-jl-ready-on-julia-1-0/16675)

<div class="topic-metadata">

**Author:** [@jocklawrie](https://discourse.julialang.org/u/jocklawrie)\
**Replies:** 0\
**Last updated:** [October 23, 2018, 10:41am UTC](https://discourse.julialang.org/t/ann-schemata-jl-ready-on-julia-1-0/16675 "2018-10-23T10:41:27Z")

</div>

Hi all, After an alpha release last November, Schemata.jl is finally ready for 1.0. From the README: A Schema is a specification of a data set. It exists independently of any particular data set, and therefore can b…

---

## [DataFrame group by first column, and sort by last column](https://discourse.julialang.org/t/dataframe-group-by-first-column-and-sort-by-last-column/16357)

<div class="topic-metadata">

**Author:** [@mbeach42](https://discourse.julialang.org/u/mbeach42)\
**Replies:** 4\
**Last updated:** [October 16, 2018, 12:26am UTC](https://discourse.julialang.org/t/dataframe-group-by-first-column-and-sort-by-last-column/16357 "2018-10-16T00:26:48Z")

</div>

Hi, Probably a simple question, but with DataFrames.jl I’m having trouble returning the data frame grouped by the first column, the sorted by the last column. Here’s a MWE using DataFrames A = rand(5,5) A\[:,1\] = \[…

---

## [DataFrames.jl - Vectorized row-wise function application](https://discourse.julialang.org/t/dataframes-jl-vectorized-row-wise-function-application/14513)

<div class="topic-metadata">

**Author:** [@mdsalerno](https://discourse.julialang.org/u/mdsalerno)\
**Replies:** 3\
**Last updated:** [October 13, 2018, 10:06pm UTC](https://discourse.julialang.org/t/dataframes-jl-vectorized-row-wise-function-application/14513 "2018-10-13T22:06:22Z")

</div>

What are efficient ways to row-wise apply a function f over a DataFrame? Looking through the DataFrames.jl documentation, I have found plenty of examples of column-wise aggregation. However, say that I want to take one…

---

## [Newbie using CSV with categorical=true](https://discourse.julialang.org/t/newbie-using-csv-with-categorical-true/16259)

<div class="topic-metadata">

**Author:** [@ilanggear](https://discourse.julialang.org/u/ilanggear)\
**Replies:** 7\
**Last updated:** [October 13, 2018, 3:31pm UTC](https://discourse.julialang.org/t/newbie-using-csv-with-categorical-true/16259 "2018-10-13T15:31:51Z")

</div>

How do I use the categorical=true feature of CSV.validate(…)? I’ve tried a bunch of variations, and keep getting the error “ERROR: TypeError: in setfield!, expected Union{Missing, CategoricalString{UInt32}}, got String”…

---

## [Gzipped (.csv.gz) writing?](https://discourse.julialang.org/t/gzipped-csv-gz-writing/11767)

<div class="topic-metadata">

**Author:** [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Replies:** 6\
**Last updated:** [October 9, 2018, 8:13am UTC](https://discourse.julialang.org/t/gzipped-csv-gz-writing/11767 "2018-10-09T08:13:35Z")

</div>

What is or will be the recommended way to write a data frame to a .csv.gz file? (I tried GZip.open with writetable but this failed.) regards, /iaw

---

## [Permission for download datasets from other peoples websites](https://discourse.julialang.org/t/permission-for-download-datasets-from-other-peoples-websites/15973)

<div class="topic-metadata">

**Author:** [@simonschoelly](https://discourse.julialang.org/u/simonschoelly)\
**Replies:** 9\
**Last updated:** [October 9, 2018, 1:33am UTC](https://discourse.julialang.org/t/permission-for-download-datasets-from-other-peoples-websites/15973 "2018-10-09T01:33:20Z")

</div>

I’m developing a a package that downloads some datasets (using DataDeps) that are publicly available on the internet. I was wondering if there might me some legal or ethical problems with that. Should I ask the authors…

---

## [Bioformats in Julia?](https://discourse.julialang.org/t/bioformats-in-julia/2440)

<div class="topic-metadata">

**Author:** [@yalexand](https://discourse.julialang.org/u/yalexand)\
**Replies:** 66\
**Last updated:** [October 5, 2018, 3:04pm UTC](https://discourse.julialang.org/t/bioformats-in-julia/2440 "2018-10-05T15:04:40Z")

</div>

Hi All, I look how Julia may be suitable for my current project (related to Bioimaging), and as I see, there is no connection to Bioformats. Bioformats is Biomicroscopy format crunching I/O software easily accessible f…

---

## [New package ASDF.jl](https://discourse.julialang.org/t/new-package-asdf-jl/15808)

<div class="topic-metadata">

**Author:** [@schnetter](https://discourse.julialang.org/u/schnetter)\
**Replies:** 0\
**Last updated:** [October 3, 2018, 12:39am UTC](https://discourse.julialang.org/t/new-package-asdf-jl/15808 "2018-10-03T00:39:55Z")

</div>

I have created a package ASDF.jl https://github.com/eschnett/ASDF.jl that implements the ASDF (Advanced Scientific Data Format) file format https://github.com/spacetelescope/asdf-standard. This file format has similariti…

---

## [Announce: A different way to read JSON data, LazyJSON.jl](https://discourse.julialang.org/t/announce-a-different-way-to-read-json-data-lazyjson-jl/9046)

<div class="topic-metadata">

**Author:** [@samoconnor](https://discourse.julialang.org/u/samoconnor)\
**Replies:** 19\
**Last updated:** [October 2, 2018, 2:07am UTC](https://discourse.julialang.org/t/announce-a-different-way-to-read-json-data-lazyjson-jl/9046 "2018-10-02T02:07:34Z")

</div>

LazyJSON.jl implements yet another a different way of reading JSON data in Julia. I wrote this as a proof of concept and it is probably not production ready, but if you work with JSON data in a performance sensitive appl…

---

## [CSV.write() to Unix Pipe (e.g., lz4 or bzip2)](https://discourse.julialang.org/t/csv-write-to-unix-pipe-e-g-lz4-or-bzip2/15759)

<div class="topic-metadata">

**Author:** [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Replies:** 6\
**Last updated:** [October 1, 2018, 11:32pm UTC](https://discourse.julialang.org/t/csv-write-to-unix-pipe-e-g-lz4-or-bzip2/15759 "2018-10-01T23:32:43Z")

</div>

I am wondering whether CSV.write() can write to a different non-gzip compressor. julia\> using DataFrames, CSV julia\> textbz2= open("test.txt.bz2", "w") IOStream(\<file test.txt.bz2\>) julia\> open( \`bzip2 -c\`, "w", text…

---

## [Is post deleted deleted deleted](https://discourse.julialang.org/t/is-post-deleted-deleted-deleted/15746)

<div class="topic-metadata">

**Author:** [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Replies:** 0\
**Last updated:** [October 1, 2018, 5:36pm UTC](https://discourse.julialang.org/t/is-post-deleted-deleted-deleted/15746 "2018-10-01T17:36:57Z")

</div>

deleted deleteddeleteddeleteddeleted

---

## [Row wise operations in DataFrames](https://discourse.julialang.org/t/row-wise-operations-in-dataframes/15682)

<div class="topic-metadata">

**Author:** [@cosmia](https://discourse.julialang.org/u/cosmia)\
**Replies:** 3\
**Last updated:** [September 30, 2018, 7:29pm UTC](https://discourse.julialang.org/t/row-wise-operations-in-dataframes/15682 "2018-09-30T19:29:24Z")

</div>

Hey guys, I have a very big DataFrame, where I want to do row wise linear algebra operations if certain conditions are met. What i need is that for every row, check if column (variable) is 1 or 0, and for each case, do …

---

## [Problem with unzippig file and reading it with CSV](https://discourse.julialang.org/t/problem-with-unzippig-file-and-reading-it-with-csv/15645)

<div class="topic-metadata">

**Author:** [@alejandromerchan](https://discourse.julialang.org/u/alejandromerchan)\
**Replies:** 2\
**Last updated:** [September 29, 2018, 5:35am UTC](https://discourse.julialang.org/t/problem-with-unzippig-file-and-reading-it-with-csv/15645 "2018-09-29T05:35:57Z")

</div>

I’m having issues now with a code that was working before and will appreciate any help to solve this, because despite some comments from @quinnj on the Slack, I still haven’t been able to figure it out. I’m downloading …

---

## [Anyone can help explain about these statements?](https://discourse.julialang.org/t/anyone-can-help-explain-about-these-statements/15545)

<div class="topic-metadata">

**Author:** [@LeoK987](https://discourse.julialang.org/u/LeoK987)\
**Replies:** 12\
**Last updated:** [September 28, 2018, 1:25am UTC](https://discourse.julialang.org/t/anyone-can-help-explain-about-these-statements/15545 "2018-09-28T01:25:05Z")

</div>

I am trying to find out an issue that causes julia to freeze when my large script uses with new releases of CSV and Tables. I tried to cook up simple demo code but none could reproduce the issue. So I tried to dig into…

---

## [Query.jl - filtering on missing data](https://discourse.julialang.org/t/query-jl-filtering-on-missing-data/14898)

<div class="topic-metadata">

**Author:** [@mdsalerno](https://discourse.julialang.org/u/mdsalerno)\
**Replies:** 7\
**Last updated:** [September 21, 2018, 5:08pm UTC](https://discourse.julialang.org/t/query-jl-filtering-on-missing-data/14898 "2018-09-21T17:08:58Z")

</div>

What is the correct way to filter on missing values? The following does not work properly: result = @from r in df begin @where !ismissing(r.A) @select r @collect DataFrame end When I run such a query, the @…

---

## [HDF and JLD have been unbuildable on macs for months... Not sure anyone is addressing](https://discourse.julialang.org/t/hdf-and-jld-have-been-unbuildable-on-macs-for-months-not-sure-anyone-is-addressing/13468)

<div class="topic-metadata">

**Author:** [@nickeubank](https://discourse.julialang.org/u/nickeubank)\
**Replies:** 5\
**Last updated:** [September 20, 2018, 2:29pm UTC](https://discourse.julialang.org/t/hdf-and-jld-have-been-unbuildable-on-macs-for-months-not-sure-anyone-is-addressing/13468 "2018-09-20T14:29:21Z")

</div>

So for several months now HDF.jl and JLD.jl are unbuildable on macs because of a problem with Homebrew.jl. It’s been brought up in a number of places by a number of people (here, here, here, here, here, here, and more I’…

---

## [Reading large Excel files (with UTF-8 entries) for caching and later processing?](https://discourse.julialang.org/t/reading-large-excel-files-with-utf-8-entries-for-caching-and-later-processing/15165)

<div class="topic-metadata">

**Author:** [@robertfeldt](https://discourse.julialang.org/u/robertfeldt)\
**Replies:** 8\
**Last updated:** [September 20, 2018, 12:25pm UTC](https://discourse.julialang.org/t/reading-large-excel-files-with-utf-8-entries-for-caching-and-later-processing/15165 "2018-09-20T12:25:42Z")

</div>

There seem to be many options for how to read Excel files into Julia and caching them for later/future processing. Can someone recommend a few good ways? Would ExcelFiles.jl and JuliaDB.jl be a good first try? Or is some…

---

## [Memory leak in 1.0 while using IJulia, DataFrames and LibPQ?](https://discourse.julialang.org/t/memory-leak-in-1-0-while-using-ijulia-dataframes-and-libpq/15163)

<div class="topic-metadata">

**Author:** [@johann.spies](https://discourse.julialang.org/u/johann.spies)\
**Replies:** 1\
**Last updated:** [September 20, 2018, 3:30am UTC](https://discourse.julialang.org/t/memory-leak-in-1-0-while-using-ijulia-dataframes-and-libpq/15163 "2018-09-20T03:30:32Z")

</div>

I just had the experience that running a query to a PostgreSQL database using IJulia and LibPQ brought my computer to a halt and I had to push the power button to shut it off. This happened in 1.0. The memory usage was …

---

## [Open Data Access Protocol](https://discourse.julialang.org/t/open-data-access-protocol/13158)

<div class="topic-metadata">

**Author:** [@johnh](https://discourse.julialang.org/u/johnh)\
**Replies:** 8\
**Last updated:** [September 19, 2018, 2:24pm UTC](https://discourse.julialang.org/t/open-data-access-protocol/13158 "2018-09-19T14:24:24Z")

</div>

Is anyone able to use OpenDAP with Julia https://www.opendap.org/ I am at JuliaCON and maybe we could chat? Or maybe look at this at the Hackathon tomorrow? I would like to download a dataset of sea surface temperatur…

---

## [Dates - Randomly generating DateTime values](https://discourse.julialang.org/t/dates-randomly-generating-datetime-values/15156)

<div class="topic-metadata">

**Author:** [@mdsalerno](https://discourse.julialang.org/u/mdsalerno)\
**Replies:** 2\
**Last updated:** [September 19, 2018, 8:16am UTC](https://discourse.julialang.org/t/dates-randomly-generating-datetime-values/15156 "2018-09-19T08:16:26Z")

</div>

I have two DateTimes, dt1 \< dt2. What is a reasonable way to randomly generate a DateTime that is in between dt1 and dt2? I started off by trying dt3 = rand(dt1.instant.periods.value : dt2.instant.periods.value). Thi…

---

## [DataFrames.jl - Choosing between the core functions and available libraries (Query.jl, DataFramesMeta.jl, etc)](https://discourse.julialang.org/t/dataframes-jl-choosing-between-the-core-functions-and-available-libraries-query-jl-dataframesmeta-jl-etc/14938)

<div class="topic-metadata">

**Author:** [@mdsalerno](https://discourse.julialang.org/u/mdsalerno)\
**Replies:** 10\
**Last updated:** [September 15, 2018, 12:00am UTC](https://discourse.julialang.org/t/dataframes-jl-choosing-between-the-core-functions-and-available-libraries-query-jl-dataframesmeta-jl-etc/14938 "2018-09-15T00:00:57Z")

</div>

In addition to the core functions in DataFrames.jl, I have come across libraries such as Query.jl and DataFramesMeta.jl which provide general querying and manipulation of DataFrames. Are there compelling reasons to use …

---

## [Large dataframe. fast row selection](https://discourse.julialang.org/t/large-dataframe-fast-row-selection/14849)

<div class="topic-metadata">

**Author:** [@grandemundo82](https://discourse.julialang.org/u/grandemundo82)\
**Replies:** 5\
**Last updated:** [September 13, 2018, 5:49pm UTC](https://discourse.julialang.org/t/large-dataframe-fast-row-selection/14849 "2018-09-13T17:49:23Z")

</div>

I have a very large dataframe and would like to know the fastest way to select a subset of rows. I tried Query package. It is elegant but really slow compared to ugly/brute force solution. Any hints? Thank you. Much appr…

---

## [Single value in DataFrame-structure](https://discourse.julialang.org/t/single-value-in-dataframe-structure/14904)

<div class="topic-metadata">

**Author:** [@sebastianskejoe](https://discourse.julialang.org/u/sebastianskejoe)\
**Replies:** 2\
**Last updated:** [September 13, 2018, 12:47pm UTC](https://discourse.julialang.org/t/single-value-in-dataframe-structure/14904 "2018-09-13T12:47:16Z")

</div>

I use a variable v to index one or more columns of a DataFrame. Sometimes v is a Symbol and sometimes v is an array of symbols. Is there a single syntax to always return the value(s) in a DataFrame-structure, even when v …

---

## [Applying function to DataFrame with missing values in columns?](https://discourse.julialang.org/t/applying-function-to-dataframe-with-missing-values-in-columns/14700)

<div class="topic-metadata">

**Author:** [@onetonfoot](https://discourse.julialang.org/u/onetonfoot)\
**Replies:** 3\
**Last updated:** [September 9, 2018, 11:44am UTC](https://discourse.julialang.org/t/applying-function-to-dataframe-with-missing-values-in-columns/14700 "2018-09-09T11:44:17Z")

</div>

Let say I’ve got a data frame like this: df = DataFrame(:x =\> \[1 ,3, missing\], :y =\> \["Some ascii text" ,"Some chinese text 恒基",missing\]) In simple cases, map seems to do what I want and just ignores/returns the missin…

---

## [CSV.read with really small decimal value](https://discourse.julialang.org/t/csv-read-with-really-small-decimal-value/14274)

<div class="topic-metadata">

**Author:** [@youngjae.woo](https://discourse.julialang.org/u/youngjae.woo)\
**Replies:** 3\
**Last updated:** [September 3, 2018, 10:49pm UTC](https://discourse.julialang.org/t/csv-read-with-really-small-decimal-value/14274 "2018-09-03T22:49:21Z")

</div>

CSV.read runs into the following error if there is a value below 5e-324. ERROR: MethodError: no method matching /(::Base.TwicePrecision{Float64}, ::Base.TwicePrecision{Float64}) Closest candidates are: /(::Base.TwiceP…

---

## [Issue with HDF5/Homebrew on 0.6.4](https://discourse.julialang.org/t/issue-with-hdf5-homebrew-on-0-6-4/13791)

<div class="topic-metadata">

**Author:** [@gideonsimpson](https://discourse.julialang.org/u/gideonsimpson)\
**Replies:** 0\
**Last updated:** [August 20, 2018, 9:56pm UTC](https://discourse.julialang.org/t/issue-with-hdf5-homebrew-on-0-6-4/13791 "2018-08-20T21:56:46Z")

</div>

The following is preventing me from getting HDF5 up and running. This used to work for me, but I had to reinstall 0.6.4 after finding I couldn’t get HDF5 in 0.7 working, and now HDF5 won’t install. INFO: Building HDF5 …

---

## [HDF5 error on 0.6.4](https://discourse.julialang.org/t/hdf5-error-on-0-6-4/14319)

<div class="topic-metadata">

**Author:** [@gideonsimpson](https://discourse.julialang.org/u/gideonsimpson)\
**Replies:** 0\
**Last updated:** [August 30, 2018, 2:10pm UTC](https://discourse.julialang.org/t/hdf5-error-on-0-6-4/14319 "2018-08-30T14:10:50Z")

</div>

I’ve been trying to convert some data form .jld to .jld2 and when reading in one of my files, I encountered the following error: julia\> load("/Users/gideonsimpson/filename1.jld") HDF5-DIAG: Error detected in HDF5 (1.10.…

---

## [Ingesting data to JuliaDB without .csv files](https://discourse.julialang.org/t/ingesting-data-to-juliadb-without-csv-files/11347)

<div class="topic-metadata">

**Author:** [@ElOceanografo](https://discourse.julialang.org/u/ElOceanografo)\
**Replies:** 4\
**Last updated:** [August 30, 2018, 5:52am UTC](https://discourse.julialang.org/t/ingesting-data-to-juliadb-without-csv-files/11347 "2018-08-30T05:52:20Z")

</div>

I’ve got a large dataset of several million short audio clips (i.e., waveforms in 1D arrays, plus metadata). Total size is about 500 GB, stored across 140,000 .jld files. I need to go through and calculate a set of sum…

---

## [Julia program reading CSV file from stdin](https://discourse.julialang.org/t/julia-program-reading-csv-file-from-stdin/14208)

<div class="topic-metadata">

**Author:** [@iwelch](https://discourse.julialang.org/u/iwelch)\
**Replies:** 13\
**Last updated:** [August 29, 2018, 7:56pm UTC](https://discourse.julialang.org/t/julia-program-reading-csv-file-from-stdin/14208 "2018-08-29T19:56:56Z")

</div>

I am wondering whether it is possible to embed a CSV file in a julia program. It almost works with CSV.read(), except I do not see a way to signal “EOF” (like with ^D on the REPL). julia\> using DataFrames, CSV julia\> …

---

## [ANN: Feather.jl v0.4.0 (lazy edition)](https://discourse.julialang.org/t/ann-feather-jl-v0-4-0-lazy-edition/14219)

<div class="topic-metadata">

**Author:** [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Replies:** 2\
**Last updated:** [August 29, 2018, 1:23pm UTC](https://discourse.julialang.org/t/ann-feather-jl-v0-4-0-lazy-edition/14219 "2018-08-29T13:23:14Z")

</div>

Hello all, it’s finally time to announce something that we’ve been working on and using extensively for a good while now, that is the new Feather.jl. (We still have to fix our docs, see latest for the v0.4.0 docs.) For…

[Previous page](https://discourse.julialang.org/c/domain/data/16.md?page=29)

[Next page](https://discourse.julialang.org/c/domain/data/16.md?page=31)