I would use multiple dispatch passing both model types as variables. Example:
abstract type LetterModel end
abstract type NumberModel end
struct A <: LetterModel
struct Num1 <: NumberModel
f(letter::LetterModel, number::NumberModel) # generic implementation
f(letter::A, number::NumberModel) # uses A, but ignores Number
f(letter::B, number::Num2) # Specialized method on both B and 2