# \#big-data

**URL:** https://discourse.julialang.org/tag/big-data/301.md

[Latest](https://discourse.julialang.org/latest.md) · [Categories](https://discourse.julialang.org/categories.md) · [Tags](https://discourse.julialang.org/tags.md)

---

## [Fully-remote health informatics developer analyst ($84k-$107k)](https://discourse.julialang.org/t/fully-remote-health-informatics-developer-analyst-84k-107k/133988)

<div class="topic-metadata">

**Author:** [@Ashlin\_Harris](https://discourse.julialang.org/u/Ashlin_Harris)\
**Replies:** 0\
**Last updated:** [November 19, 2025, 12:21pm UTC](https://discourse.julialang.org/t/fully-remote-health-informatics-developer-analyst-84k-107k/133988 "2025-11-19T12:21:53Z")

</div>

The Tufts Clinical and Translational Science Institute is hiring a developer analyst for their Health Informatics team. Our role at TuftsMedicine is essentially to deliver focused datasets to clinicians based on their re…

---

## [Recommendations for larger than memory data](https://discourse.julialang.org/t/recommendations-for-larger-than-memory-data/129015)

<div class="topic-metadata">

**Author:** [@RobertGregg](https://discourse.julialang.org/u/RobertGregg)\
**Replies:** 5\
**Last updated:** [October 17, 2025, 12:08pm UTC](https://discourse.julialang.org/t/recommendations-for-larger-than-memory-data/129015 "2025-10-17T12:08:24Z")

</div>

I’m starting to work with a large dataset that does not fit into memory. The data is approximately 5000 samples, 1 million features, and two time points. The features are continuous values that range from 0 to 1. I’m try…

---

## [Julia Consulting by Julius Technologies](https://discourse.julialang.org/t/julia-consulting-by-julius-technologies/105121)

<div class="topic-metadata">

**Author:** [@James\_Lee](https://discourse.julialang.org/u/James_Lee)\
**Replies:** 1\
**Last updated:** [February 28, 2025, 6:23pm UTC](https://discourse.julialang.org/t/julia-consulting-by-julius-technologies/105121 "2025-02-28T18:23:41Z")

</div>

I am thrilled to introduce Julius Technologies Consulting, the consulting arm for Julius Technologies. As you may already know, Julius Technologies offers a low-code graph computing solution that makes it simple for dev…

---

## [Struggling with Julia and large datasets](https://discourse.julialang.org/t/struggling-with-julia-and-large-datasets/56916)

<div class="topic-metadata">

**Author:** [@sampope](https://discourse.julialang.org/u/sampope)\
**Replies:** 67\
**Last updated:** [October 17, 2024, 11:42pm UTC](https://discourse.julialang.org/t/struggling-with-julia-and-large-datasets/56916 "2024-10-17T23:42:00Z")

</div>

TLDR: Skip to the 3rd paragraph for the questions. backstory I’m working with a decent sized dataset, around 100 billion log lines from sensor data, updating at 258 million lines per day at nanosecond resolution. Origin…

---

## [Nimble, newest optimized big-data storage format](https://discourse.julialang.org/t/nimble-newest-optimized-big-data-storage-format/112810)

<div class="topic-metadata">

**Author:** [@Dan](https://discourse.julialang.org/u/Dan)\
**Replies:** 3\
**Last updated:** [April 11, 2024, 12:19pm UTC](https://discourse.julialang.org/t/nimble-newest-optimized-big-data-storage-format/112810 "2024-04-11T12:19:53Z")

</div>

Interesting talk about an upcoming open-source data format from Meta called Nimble. It has some nice optimizations. Julia is pretty good with data and with optimizations, so it might be interesting to support this format…

---

## [Reading a few rows from a BIG CSV file](https://discourse.julialang.org/t/reading-a-few-rows-from-a-big-csv-file/68611)

<div class="topic-metadata">

**Author:** [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Replies:** 39\
**Last updated:** [January 18, 2024, 4:35pm UTC](https://discourse.julialang.org/t/reading-a-few-rows-from-a-big-csv-file/68611 "2024-01-18T16:35:17Z")

</div>

I’m reading a Census bureau CSV file that’s 2.2M rows long. I’d like to just read the first hundred rows to check my stuff works… df = Iterators.take(CSV.Rows("filename.csv"),100) |\> DataFrame Doesn’t terminate in any…

---

## [GPUDirect?](https://discourse.julialang.org/t/gpudirect/46424)

<div class="topic-metadata">

**Author:** [@Raf](https://discourse.julialang.org/u/Raf)\
**Replies:** 4\
**Last updated:** [November 12, 2022, 5:21pm UTC](https://discourse.julialang.org/t/gpudirect/46424 "2022-11-12T17:21:22Z")

</div>

Has anyone experimented with direct transfer using nvidias GPUDirect? It would be great to have a direct HDF5 to GPU pipeline.

---

## [Open Data Specialist, HHMI Janelia Research Campus](https://discourse.julialang.org/t/open-data-specialist-hhmi-janelia-research-campus/82321)

<div class="topic-metadata">

**Author:** [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Replies:** 1\
**Last updated:** [October 21, 2022, 12:33am UTC](https://discourse.julialang.org/t/open-data-specialist-hhmi-janelia-research-campus/82321 "2022-10-21T00:33:31Z")

</div>

An exciting Open Science position @HHMIJanelia – We are looking for an Open Data Specialist to identify, develop, and execute best practices for data formatting, metadata, organization, interoperability, rapid diss…

---

## [Help needed for parsing a database](https://discourse.julialang.org/t/help-needed-for-parsing-a-database/87675)

<div class="topic-metadata">

**Author:** [@Juan\_Mac\_Donagh](https://discourse.julialang.org/u/Juan_Mac_Donagh)\
**Replies:** 5\
**Last updated:** [September 23, 2022, 8:11pm UTC](https://discourse.julialang.org/t/help-needed-for-parsing-a-database/87675 "2022-09-23T20:11:01Z")

</div>

Hi all, I need some help parsing a file, I can’t figure out how to do it. My data looks like this (it’s a 4gb file): 101m A 1 MET 0 P02185 M 1 101m A 2 VAL 1 P02185 V 2 101m…

---

## [\[ANN\] Spark.jl, reborn](https://discourse.julialang.org/t/ann-spark-jl-reborn/82275)

<div class="topic-metadata">

**Author:** [@dfdx](https://discourse.julialang.org/u/dfdx)\
**Replies:** 0\
**Last updated:** [June 5, 2022, 3:51pm UTC](https://discourse.julialang.org/t/ann-spark-jl-reborn/82275 "2022-06-05T15:51:14Z")

</div>

I’m pleased to announce a new release of Spark.jl - Julia interface to Apache Spark. Apache Spark is a ubiquitous distributed data processing framework used by thousands of organizations for large scale data engineering…

---

## [Data Storage Quo Vadis under Julia: HDF5 - JLD2 - MAT -](https://discourse.julialang.org/t/data-storage-quo-vadis-under-julia-hdf5-jld2-mat/80159)

<div class="topic-metadata">

**Author:** [@ellocco](https://discourse.julialang.org/u/ellocco)\
**Replies:** 0\
**Last updated:** [April 27, 2022, 5:35pm UTC](https://discourse.julialang.org/t/data-storage-quo-vadis-under-julia-hdf5-jld2-mat/80159 "2022-04-27T17:35:45Z")

</div>

The good news: Julia can beat the phyton data-compression package “hdf5storage”. But it is useful to have a closer look into the matter. And, if you store data, it should be better or at least as good as hdf5storage. …

---

## [Segmentation fault when applying PCA to a big dataset on an HPC](https://discourse.julialang.org/t/segmentation-fault-when-applying-pca-to-a-big-dataset-on-an-hpc/76900)

<div class="topic-metadata">

**Author:** [@Ivan](https://discourse.julialang.org/u/Ivan)\
**Replies:** 2\
**Last updated:** [February 22, 2022, 7:15pm UTC](https://discourse.julialang.org/t/segmentation-fault-when-applying-pca-to-a-big-dataset-on-an-hpc/76900 "2022-02-22T19:15:11Z")

</div>

I understand the following may be due to the fact that I am using an HPC remote service. If that’s the case, I would appreciate comments on where to ask this question best. I have a big array (2765 x 12212224), and I wa…

---

## [Reading and processing multiple very large Wav files](https://discourse.julialang.org/t/reading-and-processing-multiple-very-large-wav-files/27158)

<div class="topic-metadata">

**Author:** [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Replies:** 10\
**Last updated:** [February 22, 2022, 2:30am UTC](https://discourse.julialang.org/t/reading-and-processing-multiple-very-large-wav-files/27158 "2022-02-22T02:30:33Z")

</div>

New job, new problems! :slight\_smile: I’m currently trying to figure out the best way to read a folder of very large WAV sound files and process them. Reading an entire file causes me to run out of memory. Thankfully, t…

---

## [Any equivalent of Google translate (like in gsheet) for DataFrames?](https://discourse.julialang.org/t/any-equivalent-of-google-translate-like-in-gsheet-for-dataframes/73904)

<div class="topic-metadata">

**Author:** [@o314](https://discourse.julialang.org/u/o314)\
**Replies:** 5\
**Last updated:** [January 3, 2022, 9:10am UTC](https://discourse.julialang.org/t/any-equivalent-of-google-translate-like-in-gsheet-for-dataframes/73904 "2022-01-03T09:10:50Z")

</div>

Hello there, I’m used to googletranslating cell in google sheet, a solution that works quite well. I wonder if there is any simple solution that can be used with DataFrames to translate some cell ? The fact is i need …

---

## [Developing a Beginner's Roadmap to Learn Julia High Performance Computing for Data Science](https://discourse.julialang.org/t/developing-a-beginners-roadmap-to-learn-julia-high-performance-computing-for-data-science/63970)

<div class="topic-metadata">

**Author:** [@TheCedarPrince](https://discourse.julialang.org/u/TheCedarPrince)\
**Replies:** 30\
**Last updated:** [July 5, 2021, 11:53pm UTC](https://discourse.julialang.org/t/developing-a-beginners-roadmap-to-learn-julia-high-performance-computing-for-data-science/63970 "2021-07-05T23:53:19Z")

</div>

Hi all :wave: Hope you are doing well! In alignment with both my professional and personal interests, I am slowly starting to turn my attention to the domain of High Performance Computing for the purposes of: Designin…

---

## [Storing big data file for fast access?](https://discourse.julialang.org/t/storing-big-data-file-for-fast-access/54992)

<div class="topic-metadata">

**Author:** [@mihrits](https://discourse.julialang.org/u/mihrits)\
**Replies:** 8\
**Last updated:** [February 10, 2021, 7:31pm UTC](https://discourse.julialang.org/t/storing-big-data-file-for-fast-access/54992 "2021-02-10T19:31:15Z")

</div>

I have a somewhat big .txt data file (~70 MB, ~1M lines, ~30 columns) that I plan to study during the next few months. Data are mostly floats, but also strings and integers. And I’ll probably work with more files like th…

---

## [Can Julia efficiently make use of 20+ cores for transforming hundreds of millions of rows for machine learning?](https://discourse.julialang.org/t/can-julia-efficiently-make-use-of-20-cores-for-transforming-hundreds-of-millions-of-rows-for-machine-learning/50968)

<div class="topic-metadata">

**Author:** [@Kevin\_Shen](https://discourse.julialang.org/u/Kevin_Shen)\
**Replies:** 27\
**Last updated:** [December 1, 2020, 3:15pm UTC](https://discourse.julialang.org/t/can-julia-efficiently-make-use-of-20-cores-for-transforming-hundreds-of-millions-of-rows-for-machine-learning/50968 "2020-12-01T15:15:30Z")

</div>

I’ve seen similar questions posted but I wanted to ask my question to see if I can get more precise answers. I train machine learning models on datasets with hundreds of millions of rows (10^8). On this order of magnitu…

---

## [Repartitioning 2TB of csv into parquets](https://discourse.julialang.org/t/repartitioning-2tb-of-csv-into-parquets/27716)

<div class="topic-metadata">

**Author:** [@gabomgp](https://discourse.julialang.org/u/gabomgp)\
**Replies:** 21\
**Last updated:** [June 25, 2020, 6:09pm UTC](https://discourse.julialang.org/t/repartitioning-2tb-of-csv-into-parquets/27716 "2020-06-25T18:09:31Z")

</div>

Good morning from Colombia. I’m beginner in big data/data science, and i’m trying to do the next task: We have 2 TB of CSV from one table. We want to try to use a SQL Layer to query that data. Currently, the data is s…

---

## [\[ANN\] DataConvenience v0.1.2](https://discourse.julialang.org/t/ann-dataconvenience-v0-1-2/37854)

<div class="topic-metadata">

**Author:** [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Replies:** 0\
**Last updated:** [April 19, 2020, 4:05pm UTC](https://discourse.julialang.org/t/ann-dataconvenience-v0-1-2/37854 "2020-04-19T16:05:39Z")

</div>

A number of posts have been asking for a CSV chunk reader and the new major feature for DataConvenience is reasonably fast chunk reader based on CSV.jl. See GitHub - xiaodaigh/DataConvenience.jl: Convenience functions m…

---

## [Lazy database join](https://discourse.julialang.org/t/lazy-database-join/37304)

<div class="topic-metadata">

**Author:** [@e3c6](https://discourse.julialang.org/u/e3c6)\
**Replies:** 4\
**Last updated:** [April 10, 2020, 6:54am UTC](https://discourse.julialang.org/t/lazy-database-join/37304 "2020-04-10T06:54:05Z")

</div>

I have a very large database A, and a much smaller one B. Since A is so large, I cannot load it in memory at once. I keep A as a gzip compressed CSV file on disk, and I want to load it lazily. On the other hand B is smal…

---

## [Working with BigInt type](https://discourse.julialang.org/t/working-with-bigint-type/36935)

<div class="topic-metadata">

**Author:** [@sergevic](https://discourse.julialang.org/u/sergevic)\
**Replies:** 2\
**Last updated:** [April 3, 2020, 3:32pm UTC](https://discourse.julialang.org/t/working-with-bigint-type/36935 "2020-04-03T15:32:43Z")

</div>

I have a set of p=100 covariates with some covariance matrix given for example by: using LinearAlgebra L=Symmetric(rand(100,100)) I need to extract submatrices from L and perform some operations. However, I have a very…

---

## [Repeatedly optimizing an LP with different data (changing coefficients/constraints)](https://discourse.julialang.org/t/repeatedly-optimizing-an-lp-with-different-data-changing-coefficients-constraints/36125)

<div class="topic-metadata">

**Author:** [@Smith](https://discourse.julialang.org/u/Smith)\
**Replies:** 7\
**Last updated:** [March 19, 2020, 8:57am UTC](https://discourse.julialang.org/t/repeatedly-optimizing-an-lp-with-different-data-changing-coefficients-constraints/36125 "2020-03-19T08:57:08Z")

</div>

Hi, I would like to solve an LP model for different sets of data and then write results to a CSV file. For illustration, assume the following code for one particular instance using JuMP using GLPK c = \[1,1\] A = \[1,1\] b …

---

## [Density-based clustering with incomplete distance matrix](https://discourse.julialang.org/t/density-based-clustering-with-incomplete-distance-matrix/35675)

<div class="topic-metadata">

**Author:** [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Replies:** 0\
**Last updated:** [March 7, 2020, 4:12am UTC](https://discourse.julialang.org/t/density-based-clustering-with-incomplete-distance-matrix/35675 "2020-03-07T04:12:26Z")

</div>

I am looking for a clustering algorithm (probably density based) that works with custom distances (I have my own non-Euclidean distance metric) and can work on very large datasets. I can evaluate nearest neighbors quickl…

---

## [Postdoc @ LANL](https://discourse.julialang.org/t/postdoc-lanl/31109)

<div class="topic-metadata">

**Author:** [@monty](https://discourse.julialang.org/u/monty)\
**Replies:** 23\
**Last updated:** [January 17, 2020, 4:47pm UTC](https://discourse.julialang.org/t/postdoc-lanl/31109 "2020-01-17T16:47:02Z")

</div>

The Computational Earth Science Group at Los Alamos National Laboratory (LANL) has immediate openings for Postdoctoral Research Associates with applied math and computational science skills. The postdoctoral research wil…