Benchmarks of Various Formats for Tabular Data

lungben
Fri, 20 Nov 2020 11:13:17 +0000
Data

Hi,

here is a Pluto.jl notebook for benchmarking of the read and write performance, as well as file sizes of various formats for tabular data.

gist.github.com

https://gist.github.com/lungben/7d967eb5058bbe5708bb08fb1aeb2815

table_benchmarks.jl

### A Pluto.jl notebook ###
# v0.12.11

using Markdown
using InteractiveUtils

# This Pluto notebook uses @bind for interactivity. When running this notebook outside of Pluto, the following 'mock version' of @bind gives bound variables a default value (instead of an error).
macro bind(def, element)
    quote
        local el = $(esc(element))
This file has been truncated. show original

The following formats / packages are compared:

CSV via GitHub - JuliaData/CSV.jl: Utility library for working with CSV and other delimited files in the Julia programming language

JSON via GitHub - JuliaData/JSONTables.jl: JSON3.jl + Tables.jl

Zipped CSV via GitHub - fhs/ZipFile.jl: Read/Write ZIP archives in Julia

JDF via GitHub - xiaodaigh/JDF.jl: Julia DataFrames serialization format

Parquet via GitHub - JuliaIO/Parquet.jl: Julia implementation of Parquet columnar file format reader

Apache Arrow via GitHub - apache/arrow-julia: Official Julia implementation of Apache Arrow

Excel (xlsx) via GitHub - felipenoris/XLSX.jl: Excel file reader and writer for the Julia language.

SQLite via GitHub - JuliaDatabases/SQLite.jl: A Julia interface to the SQLite library

I always used the default configuration of each package, i.e. multithreading or compression is only used if it is switched on by default.

On my machine, Arrow is fastest, followed by JDF (note that Arrow is not compressed per default by JDF is).

2 posts - 2 participants