How do I define eltype for custom composite type?

This is what you want, I guess:

julia> const Forest = Union{AbstractVector{Tree},AbstractSet{Tree}} where {Tree}
Union{AbstractSet{Tree}, AbstractVector{Tree}} where Tree

julia> Base.eltype(::Type{<:Forest{T}}) where {T} = T

julia> eltype(Forest{Int})
Int64

However, the eltype method definition above is what is termed type piracy, meaning that it’s not OK for production code, because neither the function (eltype), nor any of the type signature elements are owned by you, so such a definition wouldn’t behave well in the package ecosystem.

To avoid type piracy, perhaps define Forest as a struct instead of as a type alias.