# Functor preprocessing pattern

**URL:** <https://discourse.julialang.org/t/functor-preprocessing-pattern/86200>\
**Category:** New to Julia\
**Tags:** functors\
**Created:** [August 23, 2022, 2:29pm UTC](https://discourse.julialang.org/t/functor-preprocessing-pattern/86200 "2022-08-23T14:29:48Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![ctrebbau](https://avatars.discourse-cdn.com/v4/letter/c/5f9b8f/32.png) [@ctrebbau](https://discourse.julialang.org/u/ctrebbau)\
**Post date:** [August 23, 2022, 2:29pm UTC](https://discourse.julialang.org/t/functor-preprocessing-pattern/86200/1 "2022-08-23T14:29:48Z")

</div>

I’m trying to adapt the preprocessing pattern showcased by  
[https://github.com/jeremiedb](https://github.com/jeremiedb)  
here [https://www.youtube.com/watch?v=dzChtn9WWT8](https://www.youtube.com/watch?v=dzChtn9WWT8)  
He basically creates a structure to store transformation functions and a functor that returns the transformation itself:

```julia
struct Preproc <: Function
    layers::Vector
end

function (p::Preproc)(df::DataFrame, ids=nothing)
    df = copy(df)
    ids = isnothing(ids) ? range(1, length(p.layers), step=1) : ids
    if length(ids) == 1
        transform!(df, p.layers[ids[1]])
    else
        for layer in p.layers[ids]
            transform!(df, layer)
        end
    end
    return df
end

```

I wish to boxcox transform my target variable while storing its power parameter to apply later to novel observatios, so I’ve used BoxCoxTrans and defined the following:

```julia
struct BoxCox{T} <: Function
    λ::T
end

BoxCox(x::AbstractVector) = BoxCox(BoxCoxTrans.lambda(x).value)

function (m::BoxCox)(x::Union{Real,Missing})
    return BoxCoxTrans.transform(x, m.λ)
end

```

However, when I push and “fit” with

```julia
function build_preproc(df)

    df_fit = copy(df) 
    preproc = Preproc([])

    push!(preproc.layers, :wait => BoxCox => :wait)
    df_fit = preproc(df_fit)

    return preproc
end

```

df I get this on every cell  
 ![image](https://global.discourse-cdn.com/julialang/original/3X/4/5/45d8316617beea6859b7da8e12f23814c727b90f.png)  
instead of the transformed values and I just cannot, for the life of me, get it to return the transformed variable.

Here are the links jeremie’s work

> <https://github.com/JuliaComputing/ScoringEngineDemo.jl/blob/main/src/preproc-utils.jl>

> <https://github.com/JuliaComputing/ScoringEngineDemo.jl/blob/main/src/preproc.jl>

Thank you in advance.

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [August 23, 2022, 5:56pm UTC](https://discourse.julialang.org/t/functor-preprocessing-pattern/86200/2 "2022-08-23T17:56:35Z")

</div>

I don’t really understand the math(?) being done here, but let me ask if I’m reading this right. By transformed value, you mean the expression `BoxCoxTrans.transform(x, m.λ)` returned by the `(m::BoxCox)` method right? So the issue is that you’re reaching the callable `BoxCox{Float64}` instances and need that extra step of calling them, given some input for `x::Union{Real,Missing}`?

---

<div class="post-metadata">

**Author:** ![ctrebbau](https://avatars.discourse-cdn.com/v4/letter/c/5f9b8f/32.png) [@ctrebbau](https://discourse.julialang.org/u/ctrebbau)\
**Post date:** [August 23, 2022, 6:19pm UTC](https://discourse.julialang.org/t/functor-preprocessing-pattern/86200/4 "2022-08-23T18:19:48Z")

</div>

I’m no expert on cats either but I think you got it right, it’s that last step that eludes me

---

<div class="post-metadata">

**Author:** ![Benny](https://avatars.discourse-cdn.com/v4/letter/b/49beb7/32.png) [@Benny](https://discourse.julialang.org/u/Benny)\
**Post date:** [August 23, 2022, 7:10pm UTC](https://discourse.julialang.org/t/functor-preprocessing-pattern/86200/5 "2022-08-23T19:10:34Z")

</div>

My hunch is that the fix has to be in the `:wait => BoxCox => :wait` part. I’ve only used DataFrames rarely to limited degrees, so hopefully someone who knows it better can help you out. Maybe it would help them if you told them what the `:wait` data is (or a simulated example data), and what inputs you expect `BoxCox` and `BoxCox{Float64}` to be called on.

---

<div class="post-metadata">

**Author:** ![ctrebbau](https://avatars.discourse-cdn.com/v4/letter/c/5f9b8f/32.png) [@ctrebbau](https://discourse.julialang.org/u/ctrebbau)\
**Post date:** [August 24, 2022, 7:21am UTC](https://discourse.julialang.org/t/functor-preprocessing-pattern/86200/6 "2022-08-24T07:21:31Z")

</div>

Indeed, I’ve tried variations of `:wait => BoxCox => :wait` such as `:wait => ByRow(BoxCox) => :wait` which I think maybe got me a step closer since this is what I’m now getting  
 ![image](https://global.discourse-cdn.com/julialang/original/3X/f/8/f8cc0ad3c4c0756db4364f2a5d864dd906c5e5a7.png)  
while before the `0.25...` coincides with the BoxCox lambda.

Also, `:wait => (w -> BoxCox) => :wait` gets me  
 ![image](https://global.discourse-cdn.com/julialang/original/3X/d/f/df1f961b4af0eeb8e91e9db4f0786ecd422ca3bb.png)

On the other hand `:wait => (w -> BoxCox(w)) => :wait` returns the same as `:wait => BoxCox => :wait`

As for the interpretation of `:wait` it’s a slightly skewed normally distributed waiting time represented as a `<::Number`. Hope that helps.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [August 25, 2022, 11:06am UTC](https://discourse.julialang.org/t/functor-preprocessing-pattern/86200/7 "2022-08-25T11:06:57Z")

</div>

Looks like you don’t apply your functor after creating it, and end up with the same object in the result. I never use DataFrames and not familiar with how functors play with their custom transformation syntax, but for regular arrays the transformation would look like this:

```julia
wait = ... # your vector of numbers
b = BoxCox(wait)
wait = b.(wait) # vector of transformed values

```

---

<div class="post-metadata">

**Author:** ![ctrebbau](https://avatars.discourse-cdn.com/v4/letter/c/5f9b8f/32.png) [@ctrebbau](https://discourse.julialang.org/u/ctrebbau)\
**Post date:** [August 25, 2022, 12:41pm UTC](https://discourse.julialang.org/t/functor-preprocessing-pattern/86200/8 "2022-08-25T12:41:44Z")

</div>

Trying to broadcast with `:wait => (w -> BoxCox.(w)) => :wait) ` returns same as `ByRow(BoxCox)` version above

---

<div class="post-metadata">

**Author:** ![ctrebbau](https://avatars.discourse-cdn.com/v4/letter/c/5f9b8f/32.png) [@ctrebbau](https://discourse.julialang.org/u/ctrebbau)\
**Post date:** [September 8, 2022, 1:18pm UTC](https://discourse.julialang.org/t/functor-preprocessing-pattern/86200/9 "2022-09-08T13:18:26Z")

</div>

`:wait => BoxCox(df_fit[:, :wait]) => :wait`