Abstract Types and StructTypes - error reading AbstractTypes into concrete with JSON3

djholiver · May 10, 2021, 3:16pm

Hi,

Believe I am running into a bit of a bug / unexpected behaviour that I can’t seem to work around when extracting an interface into a concrete type having defined more than one in a module.

e.g.: this works:


using StructTypes, Parameters, JSON3
abstract type ICompute end
StructTypes.StructType(::Type{ICompute}) = StructTypes.AbstractType()
StructTypes.subtypekey(::Type{ICompute}) = :computeName
function StructTypes.subtypes(::Type{ICompute})
(
    percentile = Percentile
)
end

struct Percentile <: ICompute
    computeName
    id
    percentiles :: Vector{Float64}
end
StructTypes.StructType(::Type{Percentile}) = StructTypes.Struct()

percentile = JSON3.read("""
{
    "computeName": "percentile",
    "id": "1",
    "percentiles": [10,20,30]
    }""", ICompute)

works, but trying to extend it to include more abstract types:

using StructTypes, Parameters, JSON3
abstract type ICompute end
StructTypes.StructType(::Type{ICompute}) = StructTypes.AbstractType()
StructTypes.subtypekey(::Type{ICompute}) = :computeName
function StructTypes.subtypes(::Type{ICompute})
(
    percentile = Percentile
)
end

struct Percentile <: ICompute
    computeName
    id
    percentiles :: Vector{Float64}
end
StructTypes.StructType(::Type{Percentile}) = StructTypes.Struct()


abstract type IQuery end
StructTypes.StructType(::Type{IQuery}) = StructTypes.AbstractType()
StructTypes.subtypekey(::Type{IQuery}) = :queryName
function StructTypes.subtypes(::Type{IQuery})
(
    dataDominance = DataDominance
)
end


struct DataDominance <: IQuery
    queryName
    id
    percentiles :: Vector{Float64}
end
StructTypes.StructType(::Type{DataDominance}) = StructTypes.Struct()

percentile = JSON3.read("""
{
    "computeName": "percentile",
    "id": "1",
    "percentiles": [10,20,30]
    }""", ICompute)

datadominance = JSON3.read("""
{
    "queryName": "dataDominance",
    "id": "1",
    "percentiles": [10,20,30]
    }""", IQuery)

fails with error:

ERROR: LoadError: MethodError: no method matching length(::Type{Main.GetMetricModels.DataDominance})
Closest candidates are:
  length(!Matched::Union{Base.KeySet, Base.ValueIterator}) at abstractdict.jl:58
  length(!Matched::Union{LinearAlgebra.Adjoint{T, S}, LinearAlgebra.Transpose{T, S}} where {T, S}) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\LinearAlgebra\src\adjtrans.jl:195
  length(!Matched::Union{DataStructures.OrderedRobinDict, DataStructures.RobinDict}) at C:\Users\doliver\.julia\packages\DataStructures\ixwFs\src\ordered_robin_dict.jl:86

version info:

Packages info:

Would someone mind trying to run the above and feeding back - I’ve tried myriad combinations and just not been successful.

Regards,

lmiq · May 10, 2021, 5:35pm

I could not run your code (Average not defined). It is a little bit odd to me the concept of writing and reading abstract types from a JSON file (how to know what the structure of the data written?) but I may be not aware of how that works.

Anyway, if you could provide a MWE, that will help.

djholiver · May 10, 2021, 6:34pm

Hi - thanks for responding - updated the above - I hadn’t removed the other subtypes from the listing.

As to the usage - it’s supported by Structypes.jl and I make use of it to facilitate multiple dispatch of computations (and filters) provided by users to an http endpoint on top of a calculation library.

i.e. Percentiles require percentile arguments but Average / Sum wouldn’t but they are both of type “ICompute”. The above snippet was / is working fine until I introduced “IQuery”, resulting in the error thrown.

Incidentally - I saw your contribution here and this StructTypes + Interface approach would potentially be applicable (I would post it if it worked…), since the OP is essentially making use of properties of a type rather than the type, with a default required for the generic: implying a conversion step on raw data (JSON?) to a type (or not) and an implementation of that method for the converted type to support multiple dispatch.

Like this:

#support calcs that allow streaming:
#default - fall through
streamingUpdate!(compute:: ICompute, value) = ()

function streamingUpdate!(ave:: Average, value) 
    
    if(ismissing(value)) return end
    ave.runningTotal += value
    ave.runningCount += 1

end

function streamingUpdate!(sum:: Sum, value) 
    
    if(ismissing(value)) return end
    sum.runningTotal += value

end

Regards,

Jeff_Emanuel · May 10, 2021, 7:06pm

Maybe you need

StructTypes.subtypekey that indicates which input key/field name should be used for identifying the appropriate concrete subtype.

defining “queryName” and “computeName”?

djholiver · May 10, 2021, 9:27pm

Thanks - unfortunately not as simple - I already define this in the script above

Regards

Jeff_Emanuel · May 10, 2021, 9:29pm

Sorry, I didn’t notice it.

djholiver · May 11, 2021, 5:20pm

No problem at all.

would you be able to see if the original code snippets run for you? Still at a loss whats going on.

@lmiq did the edits allow you to execute the snippets?

Regards

Jeff_Emanuel · May 11, 2021, 5:32pm

Both snippets fail for me with:

ERROR: LoadError: MethodError: no method matching length(::Type{Percentile})
Closest candidates are:
  length(::Union{Base.KeySet, Base.ValueIterator}) at abstractdict.jl:58
  length(::Union{LinearAlgebra.Adjoint{T, S}, LinearAlgebra.Transpose{T, S}} where {T, S}) at C:\buildbot\worker\package_win64\build\usr\share\julia\stdlib\v1.6\LinearAlgebra\src\adjtrans.jl:195
  length(::Base.LogicalIndex) at multidimensional.jl:732
  ...

My versioninfo() is the same as your except minor hardware details.

lmiq · May 11, 2021, 5:51pm

Yes, but I don’t know how to solve the problem

djholiver · May 11, 2021, 6:44pm

Many thanks to you both for trying - very odd that it works for @lmiq across both snippets but not for @Jeff_Emanuel at all…

Must be down to package and / or Julia versions - I can still get one scenario to work!

I’ll revert if I find the solution / issue.

Regards,

lmiq · May 11, 2021, 6:47pm

Oh, no, I meant that I can run the code, not that it works. I doesn’t work (I get the same error as you).

Skoffer · May 11, 2021, 7:41pm

It should be

function StructTypes.subtypes(::Type{ICompute})
(
    percentile = Percentile,
)
end

djholiver · May 11, 2021, 8:07pm

Thanks @Skoffer ! I was hoping it was something simple - very counter intuitive - is it something to do with Tuple definitions?

Thanks all for contributing.

Regards

lhnguyen-vn · May 11, 2021, 10:58pm

Parentheses are also used to specify order of operations, so an explicit comma is required to disambiguate that you’re constructing a tuple.

Skoffer · May 12, 2021, 9:30am

The way how it is written is slightly confusing, since tabs do not correspond to usual notations. If you rewrite your original code as

function StructTypes.subtypes(::Type{ICompute}) 
    (percentile = Percentile)
end

Or even better

StructTypes.subtypes(::Type{ICompute})  = (percentile = Percentile)

then issue is more obvious. In expression (percentile = Percentile) brackets are ignored, so it is equivalent to percentile = Percentile, i.e. it creates throaway variable percentile and returns the result of assignment operator which is equals to Percentile. So, instead of returning NamedTuple function was equivalent to

StructTypes.subtypes(::Type{ICompute})  = Percentile

And of course, since type is not the same as NamedTuple StructTypes was unable to process it.

If you want to create named tuple with single field, you can either use comma after expression or semicolon at the beginning. So both will do

StructTypes.subtypes(::Type{ICompute})  = (percentile = Percentile, )
# or
StructTypes.subtypes(::Type{ICompute})  = (; percentile = Percentile)