# JuliaData BoF @ JuliaCon2023 discussion
**URL:** https://discourse.julialang.org/t/juliadata-bof-juliacon2023-discussion/102521
**Category:** Data
**Tags:** discussion
**Created:** [August 5, 2023, 5:10pm UTC](https://discourse.julialang.org/t/juliadata-bof-juliacon2023-discussion/102521 "2023-08-05T17:10:33Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@bkamins](https://discourse.julialang.org/u/bkamins)
#### Post date: [August 5, 2023, 5:10pm UTC](https://discourse.julialang.org/t/juliadata-bof-juliacon2023-discussion/102521/1 "2023-08-05T17:10:33Z")
Below I recap a summary from the BoF Data Ecosystem discussion during JuliaCon2023 (for sure, it is probably opinionated, so feel free to comment to add your thoughts).
I recap things I took out as most important to work on from this discussion (and other related discussions I had during the conference).
Four major areas were discussed:
1. It would be great to have robust distributed tabular data processing. It becomes increasingly important as the ability to work with out-of-core is a must for many users. DTables.jl is a good candidate, but it needs polishing. Contributions/friendly user tests are welcome.
2. A coupled topic is a flexible and easy way to read/store distributed data (especially from the cloud). @quinnj mentioned CloudStore.jl as a package to fill this gap.
3. Decoupling the “DataFrames.jl query language” from the `DataFrame` object would be good. In this way, other tabular data formats could use the same interface. This would also make the meta-packages (DataFramesMeta.jl, DataFrameMacros.jl, Tidier.jl) work automatically with different backends. @krynju has gone through this process for DTables.jl, which was doable and not very hard (though not all operations are supported; this is expected, though, as different data storage formats might impose additional constraints on allowed operations in comparison to fully-flexible DataFrames.jl). In general, it does not have to be strictly this language but some other transformation specification language (that, in particular, would allow lazy query processing). Also, it would be good to have a two-way translator between such a language and SQL.
4. It would be good to have Polars.jl bindings + ensure that DuckDB.jl bindings are stable and maintained (this was not discussed much during the meeting, but I had such discussions separately). I agree with this opinion. These two technologies will continue to improve in the future, and many potential users will want to keep using them when switching to Julia from e.g. Python.
---
### Author:  [@rssdev10](https://discourse.julialang.org/u/rssdev10)
#### Post date: [August 13, 2023, 11:48pm UTC](https://discourse.julialang.org/t/juliadata-bof-juliacon2023-discussion/102521/2 "2023-08-13T23:48:28Z")
Lazy processing…
Lazy means keeping a chain of functions and applying them with an actual data when needed. So there are few interesting possibilities here:
- easy to draw a diagram of the table transformation starting from the `DataFrame()` to the point where the transformation is applied. It helps to debug code.
- transform operators can be wrapped by an external executor/optimizer (a kind of logical execution plan);
- operators can be run in a distributed mode on a cluster with automatic load balancing and collecting results…;
- operators can be optimized before execution;
- operators can be partially converted to an SQL query, providing seamless integration with an SQL DB. In this case, an initial data request and some operations can be performed by a DB, but others can be performed locally with Julia only;
- lazy operators with DB cursor support can provide processing of large tables;
- …
---
### Author:  [@juliohm](https://discourse.julialang.org/u/juliohm)
#### Post date: [August 14, 2023, 2:05am UTC](https://discourse.julialang.org/t/juliadata-bof-juliacon2023-discussion/102521/3 "2023-08-14T02:05:10Z")
> [@rssdev10](#):
>
> Lazy processing…
Regarding lazy transforms over tables, check:
> **[GitHub - JuliaML/TableTransforms.jl: Transforms and pipelines with tabular...](https://github.com/JuliaML/TableTransforms.jl)**
>
> Transforms and pipelines with tabular data in Julia - GitHub - JuliaML/TableTransforms.jl: Transforms and pipelines with tabular data in Julia
> [@rssdev10](#):
>
> easy to draw a diagram of the table transformation starting from the `DataFrame()` to the point where the transformation is applied. It helps to debug code.
Here is the display of pipelines:
```julia
julia> using TableTransforms
julia> pipe = (Select(1:5) → PCA()) ⊔ (Interquartile() → ColTable())
ParallelTableTransform
├─ SequentialTransform
│ ├─ Select([1, 2, 3, 4, 5], nothing)
│ ├─ ZScore(all)
│ └─ EigenAnalysis(:V, nothing, 1.0)
└─ SequentialTransform
├─ Scale(all, 0.25, 0.75)
└─ ColTable()
```
> [@rssdev10](#):
>
> operators can be run in a distributed mode on a cluster with automatic load balancing and collecting results…;
We currently use Transducers.jl to run some transforms in parallel over the columns of the input table or over the branches of parallel pipelines. It would be nice to consider Dagger.jl for parallel over rows.