Is it possible to write this with Accessors.Recursive?
using Accessors
modifyrec(f, x) =
if hasproperty(x, :bs)
(@modify(x.bs |> Elements()) do b
modifyrec(f, b)
end) |> f
else
f(x)
end
let data = (;a=1, bs=[(;a=2, bs=[(;a=3, bs=[(;a=4)])])])
modifyrec(data) do x
@set x.a = 100 * x.a
end
end
# (a = 100, bs = [(a = 200, bs = [(a = 300, bs = [(a = 400,)])])])
Imo Accessors.Recursive isnât the most generic or straightforward to useâŚ
Iâd suggest this:
julia> using AccessorsExtra
julia> data = (;a=1, bs=[(;a=2, bs=[(;a=3, bs=[(;a=4)])])])
julia> @modify(a -> 100*a, data |> RecursiveOfType(NamedTuple, order=:pre) |> _.a)
(a = 100, bs = [(a = 200, bs = [(a = 300, bs = [(a = 400,)])])])
I would still like to specify that recursion happens on _.bs |> Elements() if present. Is that possible? Ie, the âdescend conditionâ should be hasproperty(_, :bs) and the way to descend is examining the elements of _.bs.
Use either RecursiveOfType(NamedTuple) |> If(x -> hasproperty(x, :bs)) or RecursivePred(x -> hasproperty(x, :bs)).
Use RecursivePred(x -> hasproperty(x, :bs), Elements() â (@maybe _.bs), order=:pre) or analogously with RecursiveOfType.
Btw, I never â not even once â needed to explicitly specify the way to descend in practice. Maybe itâs not necessary in your case as well?
Note: inference may start suffering with complex queries at some pointâŚ
How should a case like this be treated?
data = (;a=1, bs=[(;a=2, bs=[(;a=3, bs=[(;a=4)])])],cs=(;a=5,bs=(;a=6)))
Right, good question.
modifyrec(data) do x
@set x.a = 100 * x.a
end
(
a = 100,
bs = [(a = 200, bs = [(a = 300, bs = [(a = 400,)])])],
cs = (a = 5, bs = (a = 6,)),
)
modify(f, data, RecursivePred(Returns(true), Elements()â(@maybe _.bs); order=:post)) # good
# (a = 100, bs = [(a = 200, bs = [(a = 300, bs = [(a = 400,)])])], cs = (a = 5, bs = (a = 6,)))
I think in my cases the key or propertyname is how I know the semantics of the value inside. The typeof a value does not capture much about its interpretation; itâs the interface and more importantly its relation to its context â namely its access path optic _.bs â that tells me what the data means and hence what modifying it would mean.
For example if itâs some big json I can follow a path of .children[*].children[*].children[*]... but I wouldnât want to apply a transformation all over a whole tree *[*].*[*].* lest it accidentally grab some other field .children[100].stepchildren[*].children[] either in the present document or added in a later edition of the data.
By the way whatâs happening with the order=nothing case here? I was guessing it would default to one or the other but it seems to do something else.
julia> let data = (;a=1, bs=[(;a=2, bs=[(;a=3, bs=[(;a=4)])])],cs=(;a=5,bs=(;a=6)))
f(x) = @set x.a = 100 * x.a
modifyrec(f, data)
[
modify(f, data, RecursivePred(x -> x.a ⤠3, Elements()â(@maybe _.bs); order=:pre)) ,
modify(f, data, RecursivePred(x -> x.a ⤠3, Elements()â(@maybe _.bs); order=:post)) ,
modify(f, data, RecursivePred(x -> x.a ⤠3, Elements()â(@maybe _.bs); order=nothing))
]
end
3-element Vector{@NamedTuple{a::Int64, bs::Vector{@NamedTuple{a::Int64, bs::Vector{@NamedTuple{a::Int64, bs::Vector{@NamedTuple{a::Int64}}}}}}, cs::@NamedTuple{a::Int64, bs::@NamedTuple{a::Int64}}}}:
(a = 100, bs = [(a = 200, bs = [(a = 300, bs = [(a = 4,)])])], cs = (a = 5, bs = (a = 6,)))
(a = 100, bs = [(a = 200, bs = [(a = 300, bs = [(a = 4,)])])], cs = (a = 5, bs = (a = 6,)))
(a = 100, bs = [(a = 2, bs = [(a = 3, bs = [(a = 4,)])])], cs = (a = 5, bs = (a = 6,)))
is it possible to apply (in a âsimplyâ way with the Accessor syntax) the function f() to every key :a that has :bs as a parent?
order=nothing doesnât descend into matching values at all. This is arguably the most natural and intuitive default: you generally want modify(x -> 2*x, data, RecursiveOfType(Number)) to multiply numbers by two only once, not several times for stuff like complex(10u"m").
I havenât documented this flag originally because wasnât sure that order=:pre/:post/nothing is the right/useful set of choices. Over time it seems quite natural, probably it should finally be documented⌠And ideally, when the semantics of recursive optics is fleshed out and clear, they can be upstreamed to Accessors proper.
I like the idea of specifying traversal order and the predicate specification, as youâre showing there.
That said I was worried about .stepchildren and @rocco_sprmnt21âs .cs above and I think complex(10m) is another example. Early stopping with order=nothing is one way to solve it but it still makes me nervous to specify âwhat youâll find thereâ versus âhow to get thereâ because there might be false positives.
My instinct might be something like
RecursivePre(p, optic) = ...
RecursivePost(p, optic) = ...
RecursivePre(optic) = RecursivePre(Returns(true), optic)
RecursivePost(optic) = RecursivePost(Returns(true), optic)
because
nothing isnât really an orderSure, probably best to call it :nodescent or something along those lines 
Idk, why having three types that dispatch to the same code is better than having a kwarg with three possible values?
Could you please clarify and motivate the differences from the current state?
I catch two differences:
Anything I missed?
Iâm not against changes here, but would like to understand both the actual suggestions and the underlying reasoning.
Yeah I think thatâs all Iâm saying there, though the âyoloâ-style recursion I was knocking above makes me think we must be coming from totally different planets, and I should hold my tongue till I understand your position more. Iâm curious what kind of data and applications justify that approach.
If I understand, the current RecursivePred is more or less
children(x::NamedTuple) = collect(x)
children(x::Vector) = copy(x)
children(x::Int) = Int[]
Accessors.set(x::NamedTuple, ::typeof(children), val::Array) = NamedTuple{propertynames(x)}(Tuple(val))
Accessors.set(x::Array, ::typeof(children), val::Array) = copy(val)
Accessors.set(x::Int, ::typeof(children), val::Array) = x
rec_nodescend(f, p, x) = let
if p(x)
f(x)
else
@set children(x) = rec_nodescend.(f, p, children(x))
end
end
rec_pre(f, p, x) = let
x = @set children(x) = rec_pre.(f, p, children(x))
p(x) ? f(x) : x
end
let
data = (;a=1, bs=[(;a=2, bs=[(;a=3, bs=[(;a=4)])])],cs=(;a=5,bs=(;a=6)))
[
rec_nodescend((x->100x), (x -> x isa Number), data),
rec_pre((x->100x), (x -> x isa Number), data)
]
end
2-element Vector{@NamedTuple{a::Int64, bs::Vector{@NamedTuple{a::Int64, bs::Vector{@NamedTuple{a::Int64, bs::Vector{@NamedTuple{a::Int64}}}}}}, cs::@NamedTuple{a::Int64, bs::@NamedTuple{a::Int64}}}}:
(a = 100, bs = [(a = 200, bs = [(a = 300, bs = [(a = 400,)])])], cs = (a = 500, bs = (a = 600,)))
(a = 100, bs = [(a = 200, bs = [(a = 300, bs = [(a = 400,)])])], cs = (a = 500, bs = (a = 600,)))
My general tendency is to avoid flag kwargs for somewhat âtasteâ-type reasons. Part of the motivation for this practice is to let the type checker help, and because I think âstringlyâ-typed stuff is just a little ugly.
But also for âinversion of controlâ reasonsâdo we have a proof that there are only two (or three) âordersâ for these operations such that users wonât want to supply their own?:
âOrderâ is currently being used for both (1) deciding whether f applies to x or y where y = @set children(x) = f.(children(x)), and (2) whether to apply f to the children of x, in the case of :nodescend. For example does p need to be evaluated before f is applied to x's children or could we decide whether to apply f to the current value based on the result of apply f to xâs children?: p(y) ? f(y) : y. Do we also want :noascend?
If this isnât a closed enum of recursion strategies, extensible external control might be warranted, versus internal branching on a symbol?
Iâm more or less just going from my gut here, so please take all these comments with a large grain of salt.
Correct, for the modify(f, x, RecursivePred) part. As an optic, it also supports getall() and setall(), and composes with others.
Indeed, I think I remember your opinion that itâs âillegalâ to access arbitrary properties in Julia This question may be debatable in principle, but here I look from a more pragmatic PoV.
Just searched for RecursiveOfType occurrences in my code, and all the constructor calls are single-argument â never (literally not even once!) I had to specify the second parameter, customizing the descent rule. Feels like a good argument for this being the default
Some examples include:
# conversion:
modify(Float32, x, RecursiveOfType(Float64))
modify(String, x, RecursiveOfType(AbstractString))
# replace distributions/uncertainties with some representative value
modify(mean, x, RecursiveOfType(Distribution))
modify(value, x, RecursiveOfType(Uncertain.Value))
# in https://github.com/baggepinnen/MonteCarloMeasurements.jl/pull/173/files
@getall x |> RecursiveOfType(Particles) |> length(_.particles)
modify(x, RecursiveOfType(Particles)) do p
p.particles[i]
end
# for quick ad-hoc optimization:
rawvals = getall(x, RecursiveOfType(AbstractFloat))
... rawvals is a vector/tuple of floats ...
setall(x, RecursiveOfType(AbstractFloat), rawvals)
# AST manipulation
# get all function calls:
@getall x |> RecursivePred(e -> Base.isexpr(e, :call); order=:pre) |> _.args[1]
# wrap all function calls as myfunc(f, args...):
modify(x, RecursivePred(e -> Base.isexpr(e, :call); order=:pre)) do call
:(myfunc($(call.args...)))
end
# JSON3 to Dictionary conversion
modify(Dictionary, JSON3.read(...), RecursiveOfType(JSON3.Object; order=:pre))
Itâs just so convenient to tell what kind of object you want to extract/modify, no matter how exactly it can be reached within the structure!
I wonder what your potential usecases are, and why you think automatic descent wonât work there.
I guess order isnât really the best name here (suggestions welcome!).
It defines one thing: what to do when at some step the âcurrentâ object x matches the predicate. Options are (for modify):
x with f(x)x with f(x); then descend into f(x) and replace it with the resultx and replace it with the result x'; then replace it with f(x')I think I only used #1 (nothing) and #3 (:pre).
These three seem to be reasonably generic and tightly related, so they are set with a single argument now.
Thanks for digging these up. I think these snippets show the kind of values sought, rather than the context theyâre in, and itâs the context that Iâm mostly worried about. Iâm inferring that in most of those cases, youâre confident its safe to match on types because
propertynames in the future, orObject to Dictionary)Is that basically your correctness argument, or is there something else?
I donât want to bore you to death by rehashing my concerns, but since you ask⌠:
How can I be confident that in all the data that this code will run on, there wonât be any numbers other than the ones I want?
.children and I donât want to read the docs for all the other potentially many other fields and subfields that optionally appear in this object. A hidden embedded .stepchildren field spoils my .children genealogy:data = (;age=1, children=[(;age=2, children=[(;age=3, children=[(;age=4, children=[(;children=[(;age=5, children=[], stepchildren=[(;age=100, children=[])])])])])])])
mean(getall(data, RecursiveOfType(Int))) # secret stepchild
Dictionary has plenty of internal Numbers like .indices.hashes etc and I wouldnât want to accidentally getall those. This returns all sorts of internal numbers:let d = dictionary([:a => dictionary([:x => [100]]), :b => dictionary([:x => [101]])])
getall(d, RecursiveOfType(Int))
end
Data-value types: If Iâm querying a JSON for RecursiveOfType(Integer) and then JSON3.read decides to parse json "3.0" as Float64 instead of Int64, Iâll miss it and get the wrong value.
Library changes: How can I be confident that the values I want will always have type JSON3.Object rather than having been replaced by a JSON3.NewObject when JSON3.jl introduced a new type, and then theyâll be missed and Iâll get the wrong result?
Thanks for listening.
Isnât #3 a post-order traversal, since it applies f on x' instead of on x?
Would BreadthFirst() be another acceptable order?
Indeed, that was my point Looking forward to a real/realistic examples when this automatic descent doesnât work!
Preferably, concrete usecases one may encounter in practice.
Not really⌠The main reason to use recursive descent here is because the type and exact structure of the data may change, and I donât want to manually write and update all relevant paths.
Types (2) and propertynames (3) may easily change: from the most basic case of namedtuples with arbitrary properties, to more involved changes to structs in question.
I guess this was the case in my JSON example indeedâŚ
Also remember that itâs not just modify: one could write a reasonable query like
@getall JSON3.read(...) |> RecursiveOfType(JSON3.Object) |> If(x -> x[:kind] == :mykind) |> _[:mykey]
Well, if something unexpected can change in your data or library, you need to check and potentially update your code. Doesnât seem specific to optics or recursive descent at all!
Then donât use catch-all recursive descent on these types Remember that using Children() to descend is just the default, one is free to provide any optic there.
For containers specifically, it makes sense to define children as elements instead of properties. This is easy to add and already done for arrays (AccessorsExtra.jl/src/recursive.jl at master ¡ JuliaAPlavin/AccessorsExtra.jl ¡ GitHub) as they donât have properties at all.
Sure, you are right, thanks for catching!
Idk, what would be the difference in the result? And do you have some potential applications in mind?