# \[ANN\] TableTransforms.jl

**URL:** <https://discourse.julialang.org/t/ann-tabletransforms-jl/70572>\
**Category:** Data\
**Tags:** package, announcement, data, machine-learning, tables\
**Created:** [October 28, 2021, 5:26pm UTC](https://discourse.julialang.org/t/ann-tabletransforms-jl/70572 "2021-10-28T17:26:20Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [October 28, 2021, 5:26pm UTC](https://discourse.julialang.org/t/ann-tabletransforms-jl/70572/1 "2021-10-28T17:26:20Z")

</div>

TableTransforms.jl is a new package for transforms and pipelines commonly used in statistics and machine learning. It works with general Tables.jl and has some unique features compared to previous attempts:

> **[GitHub - JuliaML/TableTransforms.jl: Transforms and pipelines with tabular...](https://github.com/JuliaML/TableTransforms.jl)**
>
> Transforms and pipelines with tabular data in Julia - GitHub - JuliaML/TableTransforms.jl: Transforms and pipelines with tabular data in Julia

The package has been submitted for registration and should be available soon as a dependency for other packages. We invite the community to contribute with more tests.

---

<div class="post-metadata">

**Author:** ![datnamer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datnamer/32/3471_2.png) [@datnamer](https://discourse.julialang.org/u/datnamer)\
**Post date:** [October 28, 2021, 5:27pm UTC](https://discourse.julialang.org/t/ann-tabletransforms-jl/70572/2 "2021-10-28T17:27:42Z")

</div>

Can this integrate with MLJ?

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [October 28, 2021, 5:35pm UTC](https://discourse.julialang.org/t/ann-tabletransforms-jl/70572/3 "2021-10-28T17:35:47Z")

</div>

In my opinion, the question should be: _Can MLJ consume TableTransforms.jl?_

We are writing a package for transforms with tabular data that is self-contained and has a clean and well-defined API. That way many Julia users can contribute new transforms with ease.

MLJ is a large project with many complex interconnections. If they think TableTransforms.jl should be a dependency, we wlll be happy to help.

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [October 28, 2021, 5:41pm UTC](https://discourse.julialang.org/t/ann-tabletransforms-jl/70572/4 "2021-10-28T17:41:45Z")

</div>

How does it compare to [https://github.com/invenia/FeatureTransforms.jl](https://github.com/invenia/FeatureTransforms.jl) ?

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [October 28, 2021, 5:43pm UTC](https://discourse.julialang.org/t/ann-tabletransforms-jl/70572/5 "2021-10-28T17:43:30Z")

</div>

I explained it in the README @oxinabox . I started TableTransforms.jl because of limitations in the current design of FeatureTransforms.jl. Basically we can revert arbitrarily complex pipelines and exploit multiples threads via the awesome Transducers.jl ❤

BTW, I tried to contribute to FeatureTransforms.jl before starting the new approach. We realized that a fresh start was really needed to improve the status quo without breaking people’s code.

Also, we do not support arrays by design. This is to make sure that the code doesn’t get messy with keyword arguments. Finally, our transforms are very cheap structs without any reference to the data. So we can create pipelines completely detached from a source.

---

<div class="post-metadata">

**Author:** ![oxinabox](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oxinabox/32/206603_2.png) [@oxinabox](https://discourse.julialang.org/u/oxinabox)\
**Post date:** [October 28, 2021, 5:51pm UTC](https://discourse.julialang.org/t/ann-tabletransforms-jl/70572/6 "2021-10-28T17:51:04Z")

</div>

> [@juliohm](#):
>
> BTW, I tried to contribute to FeatureTransforms.jl before starting the new approach. We realized that a fresh start was really needed to improve the status quo without breaking people’s code.

Right.  
That would be a disadvantage of the fact that we run FeatureTransformations.jl in a lot of production code.  
Breaking changes, require a lot of coordination. (else you end up having to maintain a backports branch forever)

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [October 28, 2021, 5:56pm UTC](https://discourse.julialang.org/t/ann-tabletransforms-jl/70572/7 "2021-10-28T17:56:25Z")

</div>

> [@juliohm](#):
>
> Basically we can revert arbitrarily complex pipelines

Off the top of my head, I can’t think of a reason why I would need to invert a feature transformation pipeline. Target transformations need to be invertible, but I can’t think of a reason why feature transformations need to be invertible. Data normally flows through the pipeline in only one direction. Can you give an example where the invertibility of feature transformations comes in handy?

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [October 28, 2021, 6:00pm UTC](https://discourse.julialang.org/t/ann-tabletransforms-jl/70572/8 "2021-10-28T18:00:33Z")

</div>

In geostatistical modeling for example, we need to run the pipeline forward to get clean, pretty, uncorrelated Gaussian, do some additional modeling regarding geospatial correlation, and then revert the estimates. This is a pretty standard workflow in this field.

I can imagine other situations where users are interested in doing analysis on PCA space and then coming back to original ranges to show results, generate insight.

---

<div class="post-metadata">

**Author:** ![datnamer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/datnamer/32/3471_2.png) [@datnamer](https://discourse.julialang.org/u/datnamer)\
**Post date:** [October 28, 2021, 6:04pm UTC](https://discourse.julialang.org/t/ann-tabletransforms-jl/70572/9 "2021-10-28T18:04:22Z")

</div>

Agreed, and that’s what I meant

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [October 28, 2021, 8:48pm UTC](https://discourse.julialang.org/t/ann-tabletransforms-jl/70572/10 "2021-10-28T20:48:11Z")

</div>

Some bikeshedding: How about `invert` and `isinvertible` instead of `revert` and `isrevertible`?

Also, what’s the difference between running

```julia
newtable, cache = apply(pipeline, oldtable)
original = revert(pipeline, newtable, cache)

```

as opposed to just keeping `oldtable` around if you need it? As far as I can tell, `original == oldtable`.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [October 28, 2021, 8:55pm UTC](https://discourse.julialang.org/t/ann-tabletransforms-jl/70572/11 "2021-10-28T20:55:56Z")

</div>

Thinking about it some more… I don’t see a way in your package to specify which columns you want to transform, so in order to make a true pipeline you would need to add a `Select()` transformer, so you could do something like this:

```julia
Select(:a) → ZScore()

```

But `Select(:a)` is not invertible, so there goes all your invertibility out the window.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [October 28, 2021, 8:57pm UTC](https://discourse.julialang.org/t/ann-tabletransforms-jl/70572/12 "2021-10-28T20:57:56Z")

</div>

Initially I had used `isinvertible`, but then I realized that the concept we want here is revertibility and not invertibility. We want to go forward and backward in a pipeline, and the concept of inverse is slightly different. Some of these transforms are revertible but not invertible.

Regarding the cache choices, some transforms like `Center` and `ZScore` only need to keep track of mu and sigma. In order to save memory in pipelines, we just cache the minimum amount of information necessary to revert the transform. `Sequential` transforms constructed with `\to` for example have a cache that is a sequence of caches.

Regarding the `Select` that would be a nice addition. We had other names in mind though like `RowView`, `ColView` and `View`. `Select` is not invertible, but we can save the other columns and restore later, so it is revertible.

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [October 28, 2021, 9:03pm UTC](https://discourse.julialang.org/t/ann-tabletransforms-jl/70572/13 "2021-10-28T21:03:12Z")

</div>

> [@juliohm](#):
>
> Some of these transforms are revertible but not invertible.

Interesting. Which transformations are revertible but not invertible?

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [October 28, 2021, 9:07pm UTC](https://discourse.julialang.org/t/ann-tabletransforms-jl/70572/14 "2021-10-28T21:07:36Z")

</div>

For example, `Parallel` transforms take a single table as input, run multiple transforms in parallel and concatenate the columns. It is revertible because one can pick any of the transforms that is revertible and recover the input table, but it is not invertible because there may be multiple paths to revert, each producing a slightly different input table. For example, `PCA` reconstruction may not be perfect sometimes.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [October 29, 2021, 4:20am UTC](https://discourse.julialang.org/t/ann-tabletransforms-jl/70572/15 "2021-10-29T04:20:06Z")

</div>

@CameronBieganek I think we can actually provide a revertible `Select`, we just need to save the other columns and restore later. I will try to add this transform in the following days together with a `Discard`. Just need to decide what is the most appropriate name for these transforms.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [October 29, 2021, 2:13pm UTC](https://discourse.julialang.org/t/ann-tabletransforms-jl/70572/16 "2021-10-29T14:13:35Z")

</div>

Select/Reject transforms added. Adding tests now to make sure that order is preserved in the revert step.

---

<div class="post-metadata">

**Author:** ![bgctw](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bgctw/32/22050_2.png) [@bgctw](https://discourse.julialang.org/u/bgctw)\
**Post date:** [February 11, 2022, 7:35pm UTC](https://discourse.julialang.org/t/ann-tabletransforms-jl/70572/17 "2022-02-11T19:35:02Z")

</div>

Where can I find documentation in addition to the Readme? e.g. on the several available Transforms.

Do I need to resort to general ML documentation or to docu of FeatureTransforms.jl?

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [February 11, 2022, 7:41pm UTC](https://discourse.julialang.org/t/ann-tabletransforms-jl/70572/18 "2022-02-11T19:41:34Z")

</div>

Did you try the docstrings of each transform? For example, `?PCA`. Type question mark followed by the name of the transform you are interested.

---

<div class="post-metadata">

**Author:** ![bgctw](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bgctw/32/22050_2.png) [@bgctw](https://discourse.julialang.org/u/bgctw)\
**Post date:** [February 11, 2022, 7:50pm UTC](https://discourse.julialang.org/t/ann-tabletransforms-jl/70572/19 "2022-02-11T19:50:37Z")

</div>

`?Quantile` gives me “The quantile transform to a given distribution.”

I am not an ML person but start to explore how I can transform parameters before a Bayesian inversion. I thought that TableTransforms might be a more general alternative to [TransformVariables.jl](https://tamaspapp.eu/TransformVariables.jl/stable/) but the current docu does not allow me evaluating this.

---

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [February 11, 2022, 8:05pm UTC](https://discourse.julialang.org/t/ann-tabletransforms-jl/70572/20 "2022-02-11T20:05:59Z")

</div>

Yes, unfortunately the docstrings aren’t ideal. The Quantile transform is a transform that converts the CDF of the input to any given CDF using inverse sampling. I think the closest wikipedia page is

> **[Inverse transform sampling](https://en.wikipedia.org/wiki/Inverse_transform_sampling)**
>
> Inverse transform sampling (also known as inversion sampling, the inverse probability integral transform, the inverse transformation method, Smirnov transform, or the golden rule) is a basic method for pseudo-random number sampling, i.e., for generating sample numbers at random from any probability distribution given its cumulative distribution function.
> Inverse transformation sampling takes uniform samples of a number 
>   
>     
>       
> u
>       
>     
> {\\displaystyle u}
>   
> between 0 and...

The idea is that you can convert between a CDF1 to a uniform CDF and then to a CDF2. So your transform object `Quantile(Normal())` will convert the original CDF to a Normal CDF. You can try any continuous distribution from Distributions.jl as the argument to the `Quantile` transform.

[Next page](https://discourse.julialang.org/t/ann-tabletransforms-jl/70572.md?page=2)