# Data

**URL:** https://discourse.julialang.org/c/domain/data/16.md?page=36

[Latest](https://discourse.julialang.org/latest.md) · [Categories](https://discourse.julialang.org/categories.md) · [Tags](https://discourse.julialang.org/tags.md)

**Page:** 37

---

## [How is the data ecosystem right now for large datasets?](https://discourse.julialang.org/t/how-is-the-data-ecosystem-right-now-for-large-datasets/4281)

<div class="topic-metadata">

**Author:** [@aaowens](https://discourse.julialang.org/u/aaowens)\
**Replies:** 35\
**Last updated:** [July 13, 2017, 3:08pm UTC](https://discourse.julialang.org/t/how-is-the-data-ecosystem-right-now-for-large-datasets/4281 "2017-07-13T15:08:42Z")

</div>

I am currently working with a dataset of several hundred million observations in Stata, which as I am aware is not particularly efficient at handling datasets this large. It can take close to an hour to process a few spl…

---

## [How to filter a df with startswith() in a field?](https://discourse.julialang.org/t/how-to-filter-a-df-with-startswith-in-a-field/4837)

<div class="topic-metadata">

**Author:** [@sylvaticus](https://discourse.julialang.org/u/sylvaticus)\
**Replies:** 1\
**Last updated:** [July 13, 2017, 2:06pm UTC](https://discourse.julialang.org/t/how-to-filter-a-df-with-startswith-in-a-field/4837 "2017-07-13T14:06:30Z")

</div>

Nor these two approach works: df = DataFrame(field1 = \["abc", "abb", "acc"\]) pattern = "ab" ps = sizeof(pattern) filteredDf = myData\[df\[:field1\]\[1:ps\] .== pattern), :\] filteredDf = myData\[startswith(df\[:field1\],patte…

---

## [PriorityQueue and containers](https://discourse.julialang.org/t/priorityqueue-and-containers/4796)

<div class="topic-metadata">

**Author:** [@sambitdash](https://discourse.julialang.org/u/sambitdash)\
**Replies:** 3\
**Last updated:** [July 12, 2017, 5:18pm UTC](https://discourse.julialang.org/t/priorityqueue-and-containers/4796 "2017-07-12T17:18:06Z")

</div>

Hi All, While looking at PriorityQueue implementation in package DataStructures, I realized the algorithm is biased to underlying implementation. The implementation maintains a Dict and index array as a paired heap arr…

---

## [Converting from Wide to Long format](https://discourse.julialang.org/t/converting-from-wide-to-long-format/4811)

<div class="topic-metadata">

**Author:** [@Juan](https://discourse.julialang.org/u/Juan)\
**Replies:** 1\
**Last updated:** [July 12, 2017, 3:43pm UTC](https://discourse.julialang.org/t/converting-from-wide-to-long-format/4811 "2017-07-12T15:43:05Z")

</div>

Hello. Some time ago I dealt with the problem of converting a large dataset from Wide format to Long format, sometimes called pivoting. I did that first steps in R but it’s too slow. Let’s say we have this toy example…

---

## [Help with a little query](https://discourse.julialang.org/t/help-with-a-little-query/4723)

<div class="topic-metadata">

**Author:** [@ssfrr](https://discourse.julialang.org/u/ssfrr)\
**Replies:** 8\
**Last updated:** [July 10, 2017, 12:23pm UTC](https://discourse.julialang.org/t/help-with-a-little-query/4723 "2017-07-10T12:23:59Z")

</div>

I don’t generally work with much tabular data, so I could use some input on the best way to do a little CSV processing task that popped up today. The data I’m working with is here, and is evaluation metrics for a bunch …

---

## [Getting our act together in the data ecosystem](https://discourse.julialang.org/t/getting-our-act-together-in-the-data-ecosystem/4522)

<div class="topic-metadata">

**Author:** [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Replies:** 4\
**Last updated:** [July 4, 2017, 5:55pm UTC](https://discourse.julialang.org/t/getting-our-act-together-in-the-data-ecosystem/4522 "2017-07-04T17:55:45Z")

</div>

Lately I have been guilty of running off and doing-my-own-thing while waiting for the data ecosystem to sort itself out when I could be making more valuable contributions that would both achieve the things I am trying to…

---

## [Announcement: DataFrames Future Plans](https://discourse.julialang.org/t/announcement-dataframes-future-plans/266)

<div class="topic-metadata">

**Author:** [@nalimilan](https://discourse.julialang.org/u/nalimilan)\
**Replies:** 26\
**Last updated:** [July 4, 2017, 5:54pm UTC](https://discourse.julialang.org/t/announcement-dataframes-future-plans/266 "2017-07-04T17:54:45Z")

</div>

UPDATE: the plan described below is going to be implemented in a different way from what was initially announced. The DataFrames package will remain the same as it is now: instead, the new framework will be provided by t…

---

## [CSV parsing performance](https://discourse.julialang.org/t/csv-parsing-performance/4619)

<div class="topic-metadata">

**Author:** [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)\
**Replies:** 4\
**Last updated:** [July 4, 2017, 6:55am UTC](https://discourse.julialang.org/t/csv-parsing-performance/4619 "2017-07-04T06:55:27Z")

</div>

I’m playing around with CSV parsers right now (CSV.jl and TextParse.jl), trying to figure out which one is faster. @quinnj showed some impressive performance numbers in his juliacon talk that indicated CSV.jl is really g…

---

## [Query.jl: Selection using Base functions and possibly missing values](https://discourse.julialang.org/t/query-jl-selection-using-base-functions-and-possibly-missing-values/4557)

<div class="topic-metadata">

**Author:** [@tcovert](https://discourse.julialang.org/u/tcovert)\
**Replies:** 3\
**Last updated:** [June 30, 2017, 7:16am UTC](https://discourse.julialang.org/t/query-jl-selection-using-base-functions-and-possibly-missing-values/4557 "2017-06-30T07:16:51Z")

</div>

Suppose I have a DataFrame with two fields: idx and date. The date field has missing values (in the DataFrames sense) and is currently stored in the DataFrame as a string. Is there a query statement that I can write wh…

---

## [Unsure about Query error](https://discourse.julialang.org/t/unsure-about-query-error/4432)

<div class="topic-metadata">

**Author:** [@zorp](https://discourse.julialang.org/u/zorp)\
**Replies:** 2\
**Last updated:** [June 23, 2017, 10:00pm UTC](https://discourse.julialang.org/t/unsure-about-query-error/4432 "2017-06-23T22:00:06Z")

</div>

I’ve started learning Julia today and have been trying to use it to work on some botany questions that I’ve been thinking about for a time now. I’ve stored two CSVs relevant to the problems in variables (which are DataFr…

---

## [\[ANN\] NOAAData.jl - Access NOAA Climate Daily & Monthly Data](https://discourse.julialang.org/t/ann-noaadata-jl-access-noaa-climate-daily-monthly-data/4288)

<div class="topic-metadata">

**Author:** [@pazzo83](https://discourse.julialang.org/u/pazzo83)\
**Replies:** 0\
**Last updated:** [June 15, 2017, 4:02pm UTC](https://discourse.julialang.org/t/ann-noaadata-jl-access-noaa-climate-daily-monthly-data/4288 "2017-06-15T16:02:36Z")

</div>

Just wanted to announce a small package I just released that wraps the NOAA climate API (Web Services API (version 2) Documentation | Climate Data Online (CDO) | National Climatic Data Center (NCDC)) called NOAAData.jl. …

---

## [How to single out valid data from DataArray?](https://discourse.julialang.org/t/how-to-single-out-valid-data-from-dataarray/4137)

<div class="topic-metadata">

**Author:** [@jonalm](https://discourse.julialang.org/u/jonalm)\
**Replies:** 4\
**Last updated:** [June 8, 2017, 9:25pm UTC](https://discourse.julialang.org/t/how-to-single-out-valid-data-from-dataarray/4137 "2017-06-08T21:25:58Z")

</div>

Hi there, I am working with DataArrays, and would like to calculate the mean of the DataArray content while ignoring NA, NaN and also Inf values. This seems like a basic use case, so I apologies in advance for failing i…

---

## [ISC on fMRI data and related problems](https://discourse.julialang.org/t/isc-on-fmri-data-and-related-problems/3945)

<div class="topic-metadata">

**Author:** [@AShedko](https://discourse.julialang.org/u/AShedko)\
**Replies:** 0\
**Last updated:** [May 26, 2017, 8:09pm UTC](https://discourse.julialang.org/t/isc-on-fmri-data-and-related-problems/3945 "2017-05-26T20:09:04Z")

</div>

ISC (inter-subject correlation) is a medical imaging method that gives you the segments of the brain that behave similarly when under similar stimuli. I am essentially calculating a voxelwise correlation of a bunch of 5…

---

## [Proposed overhaul of DataStreams](https://discourse.julialang.org/t/proposed-overhaul-of-datastreams/3677)

<div class="topic-metadata">

**Author:** [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Replies:** 7\
**Last updated:** [May 25, 2017, 8:43pm UTC](https://discourse.julialang.org/t/proposed-overhaul-of-datastreams/3677 "2017-05-25T20:43:14Z")

</div>

The DataStreams package is a great idea, and I’ve very much embraced it, but I think most agree it’s time for a major overhaul as it’s confusing to users and doesn’t cover enough use cases. I’ve written my ideas out in …

---

## [Union type data frame implementation?](https://discourse.julialang.org/t/union-type-data-frame-implementation/3925)

<div class="topic-metadata">

**Author:** [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Replies:** 4\
**Last updated:** [May 25, 2017, 7:24pm UTC](https://discourse.julialang.org/t/union-type-data-frame-implementation/3925 "2017-05-25T19:24:44Z")

</div>

I’m a little hesitant to ask mildly terrified of asking, but has any thought been given to writing a dataframe implementation using union types, as we’ve been told that updates in the handling of union types will render …

---

## [Pipe to CSV.Source](https://discourse.julialang.org/t/pipe-to-csv-source/3764)

<div class="topic-metadata">

**Author:** [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Replies:** 4\
**Last updated:** [May 22, 2017, 10:23pm UTC](https://discourse.julialang.org/t/pipe-to-csv-source/3764 "2017-05-22T22:23:17Z")

</div>

I have a gzip-compressed CSV file in which uses commas for the decimal mark. I can uncompress and then use tr on the command line, but I would prefer doing that using a pipe from Julia. I tried using CSV using DataStrea…

---

## [Custom DataFrame column sort order](https://discourse.julialang.org/t/custom-dataframe-column-sort-order/3716)

<div class="topic-metadata">

**Author:** [@sylvaticus](https://discourse.julialang.org/u/sylvaticus)\
**Replies:** 5\
**Last updated:** [May 16, 2017, 1:41pm UTC](https://discourse.julialang.org/t/custom-dataframe-column-sort-order/3716 "2017-05-16T13:41:22Z")

</div>

Is it possible to sort a DataFrame column based on custom sort order, e.g. df = DataFrame(col1= \['a','b','c'\], col2 = \[1,2,3\]) sort!(df, col1=custom\_order('a','c','b'))

---

## [Iterate delimited file as NamedTuples](https://discourse.julialang.org/t/iterate-delimited-file-as-namedtuples/3616)

<div class="topic-metadata">

**Author:** [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Replies:** 6\
**Last updated:** [May 13, 2017, 6:16am UTC](https://discourse.julialang.org/t/iterate-delimited-file-as-namedtuples/3616 "2017-05-13T06:16:34Z")

</div>

I have (gzip compressed) files that have ;-delimited rows. I would like to iterate over the rows of each file so that in the body of the iteration, I would have a NamedTuple of the raw fields, ie unparsed. This is becaus…

---

## [Help with DataStreams toy problem](https://discourse.julialang.org/t/help-with-datastreams-toy-problem/3675)

<div class="topic-metadata">

**Author:** [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Replies:** 10\
**Last updated:** [May 12, 2017, 7:45pm UTC](https://discourse.julialang.org/t/help-with-datastreams-toy-problem/3675 "2017-05-12T19:45:50Z")

</div>

I am trying to learn my way around DataStreams. I thought I would implement a simple source which just gives random values (code below). Yet when I write a CSV from it, the file only contains the header. What am I doing …

---

## [Julia 0.6 Unicode Parsing Problem](https://discourse.julialang.org/t/julia-0-6-unicode-parsing-problem/3649)

<div class="topic-metadata">

**Author:** [@Daneel](https://discourse.julialang.org/u/Daneel)\
**Replies:** 4\
**Last updated:** [May 12, 2017, 7:55am UTC](https://discourse.julialang.org/t/julia-0-6-unicode-parsing-problem/3649 "2017-05-12T07:55:24Z")

</div>

Julia 0.6.0-rc1.0 can’t properly load a Unicode character in a settings file I have, specifically “³” (U+00b3). Julia doesn’t have a problem if it’s a UTF-8 Unicode file but the settings files we have are generated in IS…

---

## [JuliaDB - Support for reading GZ file extension](https://discourse.julialang.org/t/juliadb-support-for-reading-gz-file-extension/3654)

<div class="topic-metadata">

**Author:** [@hpoit](https://discourse.julialang.org/u/hpoit)\
**Replies:** 1\
**Last updated:** [May 11, 2017, 11:04pm UTC](https://discourse.julialang.org/t/juliadb-support-for-reading-gz-file-extension/3654 "2017-05-11T23:04:55Z")

</div>

@shashi, does \_csvread from TextParse parsing abstract strings pose a problem for I/O streams? I’m not sure if we should be discussing this here instead of on the Github issues I created, but since you brought it here I…

---

## [Pulling array data from IndexedTables or JuliaDB](https://discourse.julialang.org/t/pulling-array-data-from-indexedtables-or-juliadb/3651)

<div class="topic-metadata">

**Author:** [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)\
**Replies:** 1\
**Last updated:** [May 11, 2017, 5:46pm UTC](https://discourse.julialang.org/t/pulling-array-data-from-indexedtables-or-juliadb/3651 "2017-05-11T17:46:39Z")

</div>

Typically, to do anything useful with tabular data it has to be converted to a plain-old array, often of floats. And, far worse, usually you have to take the output from doing something to said array back into the data …

---

## [How to subset a stream I/O and pass it to TensorFlow.jl? (question has been updated)](https://discourse.julialang.org/t/how-to-subset-a-stream-i-o-and-pass-it-to-tensorflow-jl-question-has-been-updated/3502)

<div class="topic-metadata">

**Author:** [@hpoit](https://discourse.julialang.org/u/hpoit)\
**Replies:** 61\
**Last updated:** [May 6, 2017, 8:56pm UTC](https://discourse.julialang.org/t/how-to-subset-a-stream-i-o-and-pass-it-to-tensorflow-jl-question-has-been-updated/3502 "2017-05-06T20:56:56Z")

</div>

Hello! I’m not sure what the best way to sort is in this case but this is the plan: I would like to turn the below block of code into a loop (I was advised by @ChrisRackauckas not to loop through rows, and to use DataFra…

---

## [Python train\_test\_split vs Julia splitobs](https://discourse.julialang.org/t/python-train-test-split-vs-julia-splitobs/2697)

<div class="topic-metadata">

**Author:** [@JuliaCaesar](https://discourse.julialang.org/u/JuliaCaesar)\
**Replies:** 11\
**Last updated:** [May 5, 2017, 10:49pm UTC](https://discourse.julialang.org/t/python-train-test-split-vs-julia-splitobs/2697 "2017-05-05T22:49:21Z")

</div>

Hi, I have difficulties switching from Python’s train\_test\_split() to Julia’s splitdata() . After many tests, I wonder if those commands are similar. Is it possible for someone to give some explanations about them ?

---

## [New Package: Journal.jl](https://discourse.julialang.org/t/new-package-journal-jl/3507)

<div class="topic-metadata">

**Author:** [@joshbode](https://discourse.julialang.org/u/joshbode)\
**Replies:** 0\
**Last updated:** [May 3, 2017, 11:56pm UTC](https://discourse.julialang.org/t/new-package-journal-jl/3507 "2017-05-03T23:56:33Z")

</div>

I would like to announce a new logging package we have developed and are using in a number of internal projects: Journal.jl is currently unregistered but development has stabilised. Any feedback would be appreciated …

---

## [How to df\[:newConcatenatedCol\] = df\[:stringCol\] \* string(df\[:intCol\])?](https://discourse.julialang.org/t/how-to-df-newconcatenatedcol-df-stringcol-string-df-intcol/2207)

<div class="topic-metadata">

**Author:** [@sylvaticus](https://discourse.julialang.org/u/sylvaticus)\
**Replies:** 5\
**Last updated:** [May 2, 2017, 3:22am UTC](https://discourse.julialang.org/t/how-to-df-newconcatenatedcol-df-stringcol-string-df-intcol/2207 "2017-05-02T03:22:53Z")

</div>

How to convert a column of integers (or floats) in a column of strings and concatenate it to other string column(s) ? e.g.: df = DataFrame(a=\["aa","ab","ac"\],year=\[2015,2016,2017\]) df\[:c\] = df\[:a\] \* " " \* string(df\[:ye…

---

## [Julia: DataFramesMeta Transformation](https://discourse.julialang.org/t/julia-dataframesmeta-transformation/3435)

<div class="topic-metadata">

**Author:** [@alasaadstat](https://discourse.julialang.org/u/alasaadstat)\
**Replies:** 4\
**Last updated:** [April 30, 2017, 6:54pm UTC](https://discourse.julialang.org/t/julia-dataframesmeta-transformation/3435 "2017-04-30T18:54:46Z")

</div>

This question is already posted at StackOverflow on this link: DataFramesMeta Transformation. @ChrisRackauckas introduced me to this forum and so I’m hoping someone might be able to help me about my problem. I’m trying t…

---

## [Regular Expression](https://discourse.julialang.org/t/regular-expression/753)

<div class="topic-metadata">

**Author:** [@alphasig329](https://discourse.julialang.org/u/alphasig329)\
**Replies:** 4\
**Last updated:** [April 20, 2017, 7:41pm UTC](https://discourse.julialang.org/t/regular-expression/753 "2017-04-20T19:41:42Z")

</div>

Hello, I am trying to convert my data management code from R to Julia with some minor challenges. In particular, I am trying to use regular expression to extract patterns between two words. For example, test = “Top Ski…

---

## [How to get ODBC to work with postgresql](https://discourse.julialang.org/t/how-to-get-odbc-to-work-with-postgresql/3169)

<div class="topic-metadata">

**Author:** [@johann.spies](https://discourse.julialang.org/u/johann.spies)\
**Replies:** 1\
**Last updated:** [April 20, 2017, 9:40am UTC](https://discourse.julialang.org/t/how-to-get-odbc-to-work-with-postgresql/3169 "2017-04-20T09:40:16Z")

</div>

Operating system: Debian Testing/Sid Julia version: 4.7.2 and 0.6.0-pre.beta.91. The same happens on both versions. I have installed unixodbc and odbc-postgresql and can connect to my database with isql. Following th…

---

## [ODBC initialization issues](https://discourse.julialang.org/t/odbc-initialization-issues/3241)

<div class="topic-metadata">

**Author:** [@jkbest2](https://discourse.julialang.org/u/jkbest2)\
**Replies:** 7\
**Last updated:** [April 18, 2017, 6:10am UTC](https://discourse.julialang.org/t/odbc-initialization-issues/3241 "2017-04-18T06:10:38Z")

</div>

I have successfully used ODBC.jl to retrieve data from a PostgreSQL server in the past (on Ubuntu). I’m trying to recreate my setup on Arch Linux, and running into problems when I load the ODBC package. I am using the sa…

[Previous page](https://discourse.julialang.org/c/domain/data/16.md?page=35)

[Next page](https://discourse.julialang.org/c/domain/data/16.md?page=37)