The way I look at it is that if you want your components to be numbers, you should just be using numbers instead of components. Components are useful for things like dimension names, but if you only want numbers, just use those.
You could do it this way, although the inner-arrays-of-components thing is not quite as performant or flexible as I’d like it to be. The performance is more tuned for looping over inner arrays of components rather than indexing into them. And as for the limitations on flexibility: each component in the array must be the same size/type/layout. In this case it looks like you’re fine there, though.