# Data

**URL:** https://discourse.julialang.org/c/domain/data/16.md?page=1

[Latest](https://discourse.julialang.org/latest.md) · [Categories](https://discourse.julialang.org/categories.md) · [Tags](https://discourse.julialang.org/tags.md)

**Page:** 2

---

## [Genie app with live, shared data](https://discourse.julialang.org/t/genie-app-with-live-shared-data/129030)

<div class="topic-metadata">

**Author:** [@Joseph\_Bradley](https://discourse.julialang.org/u/Joseph_Bradley)\
**Replies:** 0\
**Last updated:** [May 15, 2025, 11:11am UTC](https://discourse.julialang.org/t/genie-app-with-live-shared-data/129030 "2025-05-15T11:11:27Z")

</div>

I’m trying to make a Genie app that will let multiple users collaborate on a web app. It’s just a project tracker, but Excel isn’t quite enough for what we want. There will be a table df that exists on the cloud somewhe…

---

## [Good ways to plot DataFrames content?](https://discourse.julialang.org/t/good-ways-to-plot-dataframes-content/128877)

<div class="topic-metadata">

**Author:** [@Torkel](https://discourse.julialang.org/u/Torkel)\
**Replies:** 12\
**Last updated:** [May 12, 2025, 9:26am UTC](https://discourse.julialang.org/t/good-ways-to-plot-dataframes-content/128877 "2025-05-12T09:26:58Z")

</div>

I have never really gotten along with DataFrames very well (and to be fair, not used it that much). I figured I should ask if there are any straightforward recipes (or methods) for plotting the content of a DataFrame? (…

---

## [SQLite and DBInterface prepared statement error](https://discourse.julialang.org/t/sqlite-and-dbinterface-prepared-statement-error/128882)

<div class="topic-metadata">

**Author:** [@slwu89](https://discourse.julialang.org/u/slwu89)\
**Replies:** 5\
**Last updated:** [May 9, 2025, 10:32pm UTC](https://discourse.julialang.org/t/sqlite-and-dbinterface-prepared-statement-error/128882 "2025-05-09T22:32:14Z")

</div>

Hi all. Does SQLite.jl support DBInterface.jl prepared statements? My MWE is below. When running the DBInterface.prepare line I get an error (pasted below the MWE). On \[0aa819cd\] SQLite v1.6.1. When using DuckDB.jl as th…

---

## [Parquet2.jl](https://discourse.julialang.org/t/parquet2-jl/128639)

<div class="topic-metadata">

**Author:** [@Sam\_P](https://discourse.julialang.org/u/Sam_P)\
**Replies:** 3\
**Last updated:** [May 6, 2025, 10:00pm UTC](https://discourse.julialang.org/t/parquet2-jl/128639 "2025-05-06T22:00:34Z")

</div>

Hi all, just a quick question regarding Parquet2.jl. Love the package and it has been a life-saver more than once. However, when starting a new repo I noticed that it is blocking LazyArrays from updating. LazyArrays v1.…

---

## [Functionality like tidyverse's any\_of()?](https://discourse.julialang.org/t/functionality-like-tidyverses-any-of/128293)

<div class="topic-metadata">

**Author:** [@amrods](https://discourse.julialang.org/u/amrods)\
**Replies:** 2\
**Last updated:** [May 1, 2025, 2:30pm UTC](https://discourse.julialang.org/t/functionality-like-tidyverses-any-of/128293 "2025-05-01T14:30:36Z")

</div>

Is there any way to achieve functionality like tidyverse’s any\_of()? My use case is that I am reading only certain columns of a file, but there could be columns specified in the call, but non-existent in the file: vars…

---

## [Another UndefRefError: access to undefined reference](https://discourse.julialang.org/t/another-undefreferror-access-to-undefined-reference/126039)

<div class="topic-metadata">

**Author:** [@steveo314](https://discourse.julialang.org/u/steveo314)\
**Replies:** 21\
**Last updated:** [April 21, 2025, 12:41am UTC](https://discourse.julialang.org/t/another-undefreferror-access-to-undefined-reference/126039 "2025-04-21T00:41:35Z")

</div>

Is there any way to check a Dataframe to see why this error would be returned when running a function on a DF? The error message is too vague. I got this error when running: unique!(df,\[col1,col2,col3\]) I’ve only have …

---

## [Divide collection into two groups](https://discourse.julialang.org/t/divide-collection-into-two-groups/127983)

<div class="topic-metadata">

**Author:** [@HugoTrentesaux](https://discourse.julialang.org/u/HugoTrentesaux)\
**Replies:** 1\
**Last updated:** [April 11, 2025, 4:48pm UTC](https://discourse.julialang.org/t/divide-collection-into-two-groups/127983 "2025-04-11T16:48:04Z")

</div>

I can filter a group based on a condition: using DataPipes a = \[1,2,5,5,3,9,8\] large = @p filter(\_\>4, a) # 5,5,9,8 small = @p filter(\_\<=4, a) # 1,2,3 But is there a way to achieve the same in a single action? # I woul…

---

## [How to identify clusters of values in a dataframe (or grouped dataframe)](https://discourse.julialang.org/t/how-to-identify-clusters-of-values-in-a-dataframe-or-grouped-dataframe/127557)

<div class="topic-metadata">

**Author:** [@rj3838](https://discourse.julialang.org/u/rj3838)\
**Replies:** 2\
**Last updated:** [April 3, 2025, 8:31pm UTC](https://discourse.julialang.org/t/how-to-identify-clusters-of-values-in-a-dataframe-or-grouped-dataframe/127557 "2025-04-03T20:31:52Z")

</div>

I am trying to identify clusters of values within a dataframe (it’s actually grouped but I’m handling it in sections). The data is of road defects so the actual data is large, I know that isn’t relevant for this but it g…

---

## [Encrypt/Decrypt Serialized Julia Objects](https://discourse.julialang.org/t/encrypt-decrypt-serialized-julia-objects/127608)

<div class="topic-metadata">

**Author:** [@Deduction42](https://discourse.julialang.org/u/Deduction42)\
**Replies:** 2\
**Last updated:** [April 2, 2025, 4:56pm UTC](https://discourse.julialang.org/t/encrypt-decrypt-serialized-julia-objects/127608 "2025-04-02T16:56:56Z")

</div>

I am currently working with serialized objects, but would like to have them safely stored in a file system without the host being able to see what is inside the objects. The files would be generated and consumed by the s…

---

## [When is it worth it to setup a database rather than to just organize the data into files?](https://discourse.julialang.org/t/when-is-it-worth-it-to-setup-a-database-rather-than-to-just-organize-the-data-into-files/127539)

<div class="topic-metadata">

**Author:** [@AwesomeQuest](https://discourse.julialang.org/u/AwesomeQuest)\
**Replies:** 2\
**Last updated:** [March 31, 2025, 2:30pm UTC](https://discourse.julialang.org/t/when-is-it-worth-it-to-setup-a-database-rather-than-to-just-organize-the-data-into-files/127539 "2025-03-31T14:30:38Z")

</div>

I’m building a library of example spectra for chemical analysis and I’m wondering what the benefits would be to setting up a database in mySQL or something with these spectra rather than just having files in folders with…

---

## [Reading data with 0 row groups](https://discourse.julialang.org/t/reading-data-with-0-row-groups/127318)

<div class="topic-metadata">

**Author:** [@phchavesmaia](https://discourse.julialang.org/u/phchavesmaia)\
**Replies:** 0\
**Last updated:** [March 24, 2025, 7:07pm UTC](https://discourse.julialang.org/t/reading-data-with-0-row-groups/127318 "2025-03-24T19:07:45Z")

</div>

Hello everyone, I am trying to work with big data in Julia. I just exported a .parquet file/folder using python’s dask library and wanted to load it in Julia. It follows a representation of my code using Parquet2: Data…

---

## [DuckDB, FunSQL.jl, reading data](https://discourse.julialang.org/t/duckdb-funsql-jl-reading-data/127119)

<div class="topic-metadata">

**Author:** [@jerlich](https://discourse.julialang.org/u/jerlich)\
**Replies:** 0\
**Last updated:** [March 18, 2025, 9:20pm UTC](https://discourse.julialang.org/t/duckdb-funsql-jl-reading-data/127119 "2025-03-18T21:20:34Z")

</div>

Expanding Man Wednesday at 2:27 PM using DuckDB to interact with files, granting full query ability for files, really is great. The only thing that sucks about it is having to deal with SQL. I’m increasingly tempted to …

---

## [Is it hard to support Julia UDFs in DuckDB?](https://discourse.julialang.org/t/is-it-hard-to-support-julia-udfs-in-duckdb/118509)

<div class="topic-metadata">

**Author:** [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Replies:** 42\
**Last updated:** [March 18, 2025, 1:27am UTC](https://discourse.julialang.org/t/is-it-hard-to-support-julia-udfs-in-duckdb/118509 "2025-03-18T01:27:37Z")

</div>

As far as I can tell there’s no UDF in Julia for DuckDB. Gemini says it’s easy to add but a google search turns up this Github issue which seems to suggest that it is available and need some work. Does anyone know what…

---

## [Supporting appending \`Vector\` in DuckDB](https://discourse.julialang.org/t/supporting-appending-vector-in-duckdb/126511)

<div class="topic-metadata">

**Author:** [@slwu89](https://discourse.julialang.org/u/slwu89)\
**Replies:** 2\
**Last updated:** [March 11, 2025, 11:06pm UTC](https://discourse.julialang.org/t/supporting-appending-vector-in-duckdb/126511 "2025-03-11T23:06:55Z")

</div>

Hi all, following up from How to insert nested/composite types into DuckDB from a DataFrame? - #3 by slwu89 I’m peeking into what it would take to support inserting, say Vector{Int} into a DuckDB database using the appen…

---

## [Creating DataFrame from vector of rows, and fixing type](https://discourse.julialang.org/t/creating-dataframe-from-vector-of-rows-and-fixing-type/126591)

<div class="topic-metadata">

**Author:** [@digital\_carver](https://discourse.julialang.org/u/digital_carver)\
**Replies:** 2\
**Last updated:** [March 6, 2025, 7:38pm UTC](https://discourse.julialang.org/t/creating-dataframe-from-vector-of-rows-and-fixing-type/126591 "2025-03-06T19:38:17Z")

</div>

I have some data in a Vector{Vector{SubString{String}}} that I wanted to turn into a DataFrame. While DataFrame constructor accepts a vector of vectors, it seems that it assumes the inner vectors are columns (which is se…

---

## [Best data structure for mapping between nodes and (known) cliques in a graph, that is easily updated after bisection?](https://discourse.julialang.org/t/best-data-structure-for-mapping-between-nodes-and-known-cliques-in-a-graph-that-is-easily-updated-after-bisection/126622)

<div class="topic-metadata">

**Author:** [@MarkusGabl](https://discourse.julialang.org/u/MarkusGabl)\
**Replies:** 4\
**Last updated:** [March 6, 2025, 3:38pm UTC](https://discourse.julialang.org/t/best-data-structure-for-mapping-between-nodes-and-known-cliques-in-a-graph-that-is-easily-updated-after-bisection/126622 "2025-03-06T15:38:45Z")

</div>

Hi Julia community I have a question regrading the best data structure for a certain purpose. I have a graph with of nodes i=1,…,n and edges and from the way I construct the graph I know all the maximum cliques c=1,…,k…

---

## [How to insert nested/composite types into DuckDB from a DataFrame?](https://discourse.julialang.org/t/how-to-insert-nested-composite-types-into-duckdb-from-a-dataframe/126440)

<div class="topic-metadata">

**Author:** [@slwu89](https://discourse.julialang.org/u/slwu89)\
**Replies:** 2\
**Last updated:** [March 3, 2025, 7:01pm UTC](https://discourse.julialang.org/t/how-to-insert-nested-composite-types-into-duckdb-from-a-dataframe/126440 "2025-03-03T19:01:04Z")

</div>

Hi all, it seems DuckDB can support composite types as in Redirecting… I want to know how to insert into columns with one of these types from a DataFrame. I’ve tried the Appender API and directly using prepared statemen…

---

## [OPC UA package? ( Open Platform Communications Unified Architecture)](https://discourse.julialang.org/t/opc-ua-package-open-platform-communications-unified-architecture/94336)

<div class="topic-metadata">

**Author:** [@ohmsweetohm1](https://discourse.julialang.org/u/ohmsweetohm1)\
**Replies:** 15\
**Last updated:** [February 24, 2025, 8:29pm UTC](https://discourse.julialang.org/t/opc-ua-package-open-platform-communications-unified-architecture/94336 "2025-02-24T20:29:56Z")

</div>

Is there a Julia version of python-opcua or something equivalent?

---

## [How do I properly close a DuckDb database?](https://discourse.julialang.org/t/how-do-i-properly-close-a-duckdb-database/100598)

<div class="topic-metadata">

**Author:** [@kobusherbst](https://discourse.julialang.org/u/kobusherbst)\
**Replies:** 20\
**Last updated:** [February 21, 2025, 1:44am UTC](https://discourse.julialang.org/t/how-do-i-properly-close-a-duckdb-database/100598 "2025-02-21T01:44:28Z")

</div>

I cannot run the following code twice in one julia session: using DuckDB using DBInterface db = DuckDB.DB("C:\\\\Temp\\\\test.duckdb") t = DBInterface.execute(db, "SELECT 1") DuckDB.close\_database(db) On the second run …

---

## [Another sorting algorithm](https://discourse.julialang.org/t/another-sorting-algorithm/125884)

<div class="topic-metadata">

**Author:** [@xiaodai](https://discourse.julialang.org/u/xiaodai)\
**Replies:** 1\
**Last updated:** [February 14, 2025, 10:02pm UTC](https://discourse.julialang.org/t/another-sorting-algorithm/125884 "2025-02-14T22:02:53Z")

</div>

Julia has the base sort which has massive improved and also a couple of more experimental libraries including SortingAlgorithms.jl and SortingLab.jl. I saw a sorting algorithm on Hacker News today which genuinely sounds…

---

## [Arrow's DictEncode to CategoricalArray?](https://discourse.julialang.org/t/arrows-dictencode-to-categoricalarray/112451)

<div class="topic-metadata">

**Author:** [@elenev](https://discourse.julialang.org/u/elenev)\
**Replies:** 18\
**Last updated:** [February 11, 2025, 2:30pm UTC](https://discourse.julialang.org/t/arrows-dictencode-to-categoricalarray/112451 "2025-02-11T14:30:28Z")

</div>

I am using Arrow.jl to store a dataframe to disk. The dataframe has some columns of CategoricalArray{T} type, where T is either String or Int64. When I read the dataframe back in, the categorical-ness of these columns i…

---

## [Working with JDBC in Julia](https://discourse.julialang.org/t/working-with-jdbc-in-julia/125030)

<div class="topic-metadata">

**Author:** [@GregorM](https://discourse.julialang.org/u/GregorM)\
**Replies:** 1\
**Last updated:** [January 22, 2025, 10:37am UTC](https://discourse.julialang.org/t/working-with-jdbc-in-julia/125030 "2025-01-22T10:37:09Z")

</div>

In the last couple of years I have been using Julia for various data projects in different companies. What I’m always struggling with is getting all the needed data out of a database. In most cases databases can be acces…

---

## [Planing on building an ETL Framework](https://discourse.julialang.org/t/planing-on-building-an-etl-framework/124747)

<div class="topic-metadata">

**Author:** [@th0rben](https://discourse.julialang.org/u/th0rben)\
**Replies:** 2\
**Last updated:** [January 14, 2025, 7:56pm UTC](https://discourse.julialang.org/t/planing-on-building-an-etl-framework/124747 "2025-01-14T19:56:27Z")

</div>

I am planing to build a package for ETL processes as my Bachelor thesis. My plan is to to have the package be a simple way to build data pipelines following ETL. Having a simple way to load the data from different sourc…

---

## [DuckDB load problem](https://discourse.julialang.org/t/duckdb-load-problem/124467)

<div class="topic-metadata">

**Author:** [@kobusherbst](https://discourse.julialang.org/u/kobusherbst)\
**Replies:** 3\
**Last updated:** [January 7, 2025, 11:01am UTC](https://discourse.julialang.org/t/duckdb-load-problem/124467 "2025-01-07T11:01:33Z")

</div>

I created a new project, with just DuckDB and DBInterface packages. When executing the following code: using DuckDB # create a new in-memory database con = DBInterface.connect(DuckDB.DB, ":memory:") fails with the fol…

---

## [Note on eachindex behavior with GroupedDataFrame](https://discourse.julialang.org/t/note-on-eachindex-behavior-with-groupeddataframe/124439)

<div class="topic-metadata">

**Author:** [@George9000](https://discourse.julialang.org/u/George9000)\
**Replies:** 1\
**Last updated:** [January 5, 2025, 10:56am UTC](https://discourse.julialang.org/t/note-on-eachindex-behavior-with-groupeddataframe/124439 "2025-01-05T10:56:04Z")

</div>

eachindex used on a GroupedDataFrame, returns a GroupKey, such as GroupKey: (site = 'b',). When using the indices to create figure layouts in Makie or other plotting packages, (Axis(fig\[i, 1\]), one should explicitly ask…

---

## [Is it possible to join two dataframes that returns a view of the original two?](https://discourse.julialang.org/t/is-it-possible-to-join-two-dataframes-that-returns-a-view-of-the-original-two/124296)

<div class="topic-metadata">

**Author:** [@okatsn](https://discourse.julialang.org/u/okatsn)\
**Replies:** 1\
**Last updated:** [December 31, 2024, 2:36am UTC](https://discourse.julialang.org/t/is-it-possible-to-join-two-dataframes-that-returns-a-view-of-the-original-two/124296 "2024-12-31T02:36:32Z")

</div>

In julia programming, is it possible to join two dataframes, where the joined dataframe is essentially a “view” of the original two? For example, julia\> name = DataFrame(ID=\[1, 2, 3\], Name=\["John Doe", "Jane Doe", "J…

---

## [How to \`combine\` row vectors](https://discourse.julialang.org/t/how-to-combine-row-vectors/123940)

<div class="topic-metadata">

**Author:** [@alex-s-gardner](https://discourse.julialang.org/u/alex-s-gardner)\
**Replies:** 5\
**Last updated:** [December 18, 2024, 9:24pm UTC](https://discourse.julialang.org/t/how-to-combine-row-vectors/123940 "2024-12-18T21:24:58Z")

</div>

Hi All, Just wondering how I can apply a function to rows containing vectors when doing a groupby → combine: using DataFrames df = DataFrame(group = \["A", "A", "B", "B"\], v1 = \[ \[1, 2\], \[3, 4\], \[5, 6\], \[7, 8\] \], v2 = \[…

---

## [How to execute .sql file](https://discourse.julialang.org/t/how-to-execute-sql-file/123675)

<div class="topic-metadata">

**Author:** [@slwu89](https://discourse.julialang.org/u/slwu89)\
**Replies:** 10\
**Last updated:** [December 11, 2024, 2:55pm UTC](https://discourse.julialang.org/t/how-to-execute-sql-file/123675 "2024-12-11T14:55:41Z")

</div>

Hi, I am just getting started with using databases in Julia and am confused how I could read SQL commands from a file (for example, test.sql as below). I am using DuckDB.jl but the question is generic across SQLite.jl, e…

---

## [Rebuilding Data from Heatmaps](https://discourse.julialang.org/t/rebuilding-data-from-heatmaps/122291)

<div class="topic-metadata">

**Author:** [@Deceneu](https://discourse.julialang.org/u/Deceneu)\
**Replies:** 6\
**Last updated:** [December 6, 2024, 10:19am UTC](https://discourse.julialang.org/t/rebuilding-data-from-heatmaps/122291 "2024-12-06T10:19:52Z")

</div>

Hello, first time posting, might be in the wrong category. I need to extract scalar values from heatmaps saved as images. I don’t have access to the raw data, only compressed JPGs with inlayed colorbars. I would think …

---

## [DataFrames: Which formats (if any) support saving of metadata](https://discourse.julialang.org/t/dataframes-which-formats-if-any-support-saving-of-metadata/123134)

<div class="topic-metadata">

**Author:** [@alex-s-gardner](https://discourse.julialang.org/u/alex-s-gardner)\
**Replies:** 6\
**Last updated:** [November 27, 2024, 5:23pm UTC](https://discourse.julialang.org/t/dataframes-which-formats-if-any-support-saving-of-metadata/123134 "2024-11-27T17:23:37Z")

</div>

Are there file formats that support saving of metadata (global and columnar)?

[Previous page](https://discourse.julialang.org/c/domain/data/16.md)

[Next page](https://discourse.julialang.org/c/domain/data/16.md?page=2)