Good morning from Colombia. I’m beginner in big data/data science, and i’m trying to do the next task: We have 2 TB of CSV from one table. We want to try to use a SQL Layer to query that data. Currently, the data is stored in Timescale, but Timescale doesn't compress data and the used SSD space …

I want to partition the csv by rows. So, is possible to write parquets in Julia?

Using the CSV.jl package, you can use the CSV.Rows(file) structure to efficiently iterate rows; it shouldn’t matter how big the file is, CSV.Rows can efficiently handle iterating rows. Unfortunately, the Parquet.jl package doesn’t support writing parquet files yet. The Feather.jl package, however, …

[image] gabomgp: To use Pandas, in a very large machine (64 cores, more than 450 GB of RAM). The problem was that Pandas doesn’t scale to large machines. This seems like a reasonable[1] approach, though. Can you elaborate in which way Pandas “doesn’t scale”? [1] Yes, there’s probably a smart…

In my use case: When i’m reading CSV, the reading is not using multiple cores; same when i’m grouping (for partitioning), or sorting (i’m sorting for better compression). I’m parallelize using a Pool, but: 2.1) The CSV are very large sometimes, and reading many CSV in the same time, consume too m…

Is possible to write in streamming the CSV, but without a iterator? I’m thinking in something as: Read in streamming the CSV. With each row: 2.1) Check if the partition key has a csv file created. 2.2) If the csv file is created: append the row to the file. 2.3) If not created: create the file…

Once CSV.jl updates so that CSV.Rows accepts types then I will register this branch https://github.com/xiaodaigh/DataConvenience.jl/tree/CSV-chunk-reader#csv-chunk-reader which has a CSV chunk reader by wrapping some of CSV.jl’s functionalities (so thanks to CSV.jl).

Would DatFramesDBs be of any help? [image] [ANN] DataFrameDBs.jl Data Hi all! Julia is my hobby and I had some free time for the last 4 weeks, so here is the first results of my experiments GitHub - waralex/DataFrameDBs.jl: The DateFrameDBs is the prototype of persistent…

If you are considering using an sql layer, I would recommend ClickHouse ( https://clickhouse.tech/ ). DataFrameDBs is still under development and I would not recommend it to a person who needs to analyze data right now. But yes, it was for such cases that I started to develop it

In case it might help someone: this would also be super simple to achieve with Apache Spark .

Thanks you, I’m the original poster of the question… (my original account is lost). I tried with clickhouse months ago and i can confirm that is an excelent tool to play with medium data (some terabytes of info) in a easy way. Really, pandas or vaex are not competition to clickhouse, agregating and…

Repartitioning 2TB of csv into parquets

Specific Domains Data

johnh April 17, 2020, 8:56am 9

Would DatFramesDBs be of any help?

1 Like

Topic		Replies	Views
Is there a way to stream a table with 1 billion rows from SQL Server to Parquet efficently in Julia New to Julia	1	4110	November 16, 2017
Partition a large CSV file into smaller files without loading into memory General Usage question , csv , io	6	3736	March 10, 2019
Writing Parquet files General Usage	28	5503	November 12, 2020
Struggling with Julia and large datasets General Usage question , big-data	67	11619	October 17, 2024
Write Large Parquet to S3 General Usage parquet	6	378	August 9, 2023

Repartitioning 2TB of csv into parquets

Related topics