[ANN] FieldFlags.jl: Bitfield-like structs

Package Announcements
,
1

Hi everyone! I’m happy to announce FieldFlags.jl, a small package without dependencies for creating bitfield-like structs, packing integers of various bitsizes into the smallest possible space. The current stable release, after some private experimentation & testing, is v0.3.8.

This package is mostly useful if you want to pack e.g. 2 3-bit numbers into one struct, and only have it take up as much space as necessary instead of (at minimum) two bytes if the two numbers are stored as an UInt8 each.

Features include:

  • Explicit marking of padding bits of various sizes
  • Optional mutability
  • Subtypability of the created structs
  • Good discoverable errors reported by JET.jl
  • Dot-access, just like in regular structs
  • Efficiency
    • in particular, elimination of error branches and good optimization by the compiler; I can get the compiler to emit the efficient bextract instruction on my machine.

For more information, check out the documentation and give the examples a look! :slight_smile:

15 Likes
2

Very cool package!

I wonder why padding bits are undefined. Guaranteeing that they are zero makes a lot of things easier: Structs can be compared more cheaply (instead of masking before compare, mask when constructing) and bit-for-bit serialized and deserialized, and more predictably cast to another bitstype. I would also suspect (though I am not sure) that it would be very cheap to do so, perhaps even zero-cost.

2 Likes
3

It’s a tradeoff. I have to mask incoming data on setting fields either way, because I mustn’t accidentally overwrite other fields! I want to leave padding undefined because I’m not sure about the exact/best semantics for them yet; for julia itself, these are of course not padding bits at all, but rather “well defined” bits that are part of the internal object. This is part of an internal inplementation detail though - the way this currently works doesn’t allow things like SROA (because it’s “one object” from the POV of both Julia and LLVM), leaving them semantically undefined leaves me the option for swapping out the internal implementation later on, should something better come along (perhaps true LLVM level structs?). For the same reason, conversion from a @bitfield to some other bitstype is unsupported - that would observe padding; only the conversion from some known integer bitstype like Bool or Int to a @bitfield is supported.

In practice of course, they are going to be zeroed, it’s just not something that I want to expose as guaranteed API, because it locks me into the current design :slight_smile: There’s also this related issue about equality and hashing, which currently does exactly what you describe.

2 Likes
4

Looks super useful!
A couple of question/remarks:

  1. Is there a particular reason for not merging the two main macros, @bitfield and @bitflags (I could see “clarity” but wondered whether there was something else)
  2. I can see the appeal of the x:3 syntax, but i would personally prefer something like x::3 and/or x::UInt32::3 (to also specify the type returned by getproperty), as :: is already used to specify types (and specifying the bit width can be seen as an extension of that)
  3. (more of a feature request) i use more and more @kwargs, and it would be lovely if @bitfield supported it, and was printed as e.g. MyBits(a= 0x1, b= 0x2, c= true) instead of MyBits(a: 0x1, b: 0x2, c: true), such that show output can be also used as input.
5

They actually use the same code under the hood already; the reason for not merging them further is that (in a potential future of multiple abstract subtyping) we could have the “every field is a single bit” invariant encoded in the type of the object as well. It’s also a syntactic reminder that flags don’t have a width other than the single bit!

This issue will be of interest to you then :slight_smile: The a:3 syntax is borrowed from C & C++, which this package is very much inspired by. I personally also prefer not to pun on type assertion/annotation syntax; Core language syntax like that should, in my opinion, be left alone in macros (unless you’re dealing with types).

That’s a nice feature request - can you open an issue so I don’t forget about it? I originally wanted to print it with : to remind people that it’s not a regular struct and that putting arguments into the constructor will truncate them if necessary. I can see the appeal for a kwargs constructor though (defaulting to 0 for unspecified fields, presumably).

1 Like
6

Actually, that’s also a bug - this currently doesn’t error, although it should:

julia> @bitflags struct Foo
           a:2
           _
           b
       end

julia> Foo |> methods
# 3 methods for type constructor:
 [1] Foo()
     @ none:0
 [2] Foo(t::Foo_fields)
     @ none:0
 [3] Foo(a::Union{Bool, Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8}, b::Union{Bool, Int128, Int16, Int32, Int64, Int8, UInt128, UInt16, UInt32, UInt64, UInt8})
     @ none:0

Note the last method, taking two arguments a and b. I’ll add an issue & put a fix up soon-ish.

EDIT: Issue is here

7

Really awesome!

Could you explain for a noob what the purpose/benefit of adding padding to a struct is? Like when would I benefit from putting padding between field A and B?

Would you ever consider allowing @bitfields to accept other FieldFlag types as fields? It would be nice in Agent-Based Models to store flag attributes of an agent in a nested struct without losing the tight packing.

For example

@bitflags mutable struct BasicNeeds
       hungry
       tired
       injured
end

@bitfields mutable struct Health
    basic_needs::BasicNeeds
    disease_status:2
ebd

struct Person
    health::Health
end

person.health.basic_needs.hungry = true

It would be awesome for Health to only take 8 bits, not 16.

8

My motivation for this package is some (not yet revealed - you’ll have to wait until juliacon :wink: ) microcontroller shenanigans - suffice it to say that being able to place padding bits in exact bit-precise locations is very useful there (and saves a tremendous amount of code when constructing these objects…).

Hm, I know of that problem, yes. The issue is that that would more or less require knowledge that some type annotation already is a @bitfield or @bitflags type, which I currently don’t track - and getting that information in a macro is difficult, because all I see in there is a Symbol. I could just assume that is a type, but that would require eval in the macro (or some form of “bitflags/field” registry object in FieldFlags.jl itself, which would need to be guarded against collisions somehow).

It sounds like a good & doable feature, but it would require some engineering & special handling; probably best to do after Custom field type annotations · Issue #9 · Seelengrab/FieldFlags.jl · GitHub already works. Please open an issue with this though, so I don’t forget and you get a notification when it’s implemented!

2 Likes