[ANN] PackedStructs.jl

[PatrickHaecker]

I am happy to announce that we open-sourced PackedStructs.jl.

Shamelessly copying from its README:

PackedStructs.jl provides the @packed macro to annotate structs. These structs will pack together types which do not have a native size, i.e. a power-of-two byte size. This is especially useful when using BitIntegers.jl or EmulatedBitIntegers.jl. Accesses to these packed types take some additional CPU cycles in general. Therefore, speed is effectively traded for space. However, the space savings can lead to a better caching behavior. In some situations, packed structs can be both smaller and faster than regular structs.

Usage

To create a packed struct, do, e.g.:

julia> using EmulatedBitIntegers, PackedStructs
julia> @emulate Int4
julia> @packed struct Foo
           first::Int4
           second::Int4
       end

This struct is smaller than it would be without being @packed.

julia> struct RegularFoo
           first::Int4
           second::Int4
       end
julia> Foo |> sizeof
1
julia> RegularFoo |> sizeof
2

This type can be used like a regular struct, so you can create values by

julia> foo = Foo(2, -1)
Foo(2, -1)

and you can access values by

julia> foo.first
2

julia> foo.second
-1

They have the correct type

julia> foo.first |> typeof
Int4

Internally, the bits of the different fields immediately follow each other

julia> (reinterpret(UInt8, foo), foo.first, foo.second) .|> bitstring
("00101111", "0010", "1111")

Overall, I think the package gets as close to a native implementation as you can get in a package. If you use the internal low-level functions (like getfield instead of the (implicit) getproperty), you can see the wrappers leaking, but otherwise usage should feel totally native (as seen above).

Thanks to the nice people in JuliaData, who kindly allowed me to move and maintain the code there.

This package has some similarities to FieldFlags.jl. However, FieldFlags.jl focuses more on logical bits (fields), whereas PackedStructs.jl with EmulatedBitIntegers.jl focuses more on integers.

[Sukera]

Hi, nice package and thanks for the mention of FieldFlags.jl! How does your package compare to the @bitfield macro from FieldFlags.jl? The intention of that macro is precisely this kind of packing There’s an open issue for type annotations on fields, which I haven’t had the time to tackle, but it should be fairly straightforward to implement

[PatrickHaecker]

@Sukera

How does your package compare to the @bitfield macro from FieldFlags.jl?

I wondered the same. When we created the package around 2 years ago, we didn’t know about FieldFlags.jl. I only learned later about it and wanted to mention it.

There is definitely the difference in the mental model. A bit field and packed integers are semantically something completely different, although the implementation can be identical and it can be the same just looked from different angles. But how, for example, would you transport the difference between and unsigned and a signed integer in a bitfield?

This might be the reason why I haven’t found the package back then. When I look for packing small integers in structs, “flags” is not really what I would look for.

I wanted to go into some details (bitfields are bound to the struct and can’t (currently) live outside, whereas packed structs are just a collection of types, the amount of work being put into PackedStructs.jl to find an optimum grouping of fields efficiently, …) and let Claude Opus 4.8 do a comparison, too, to not forget important points. However, I think Claude’s answer is so good that I want to share it in the following. That being said, if you see possibilities to cooperate or even merge the packages, I would be interested in this.

PackedStructs.jl vs. FieldFlags.jl (the following is from Claude)

Both are Julia packages for tightly bit-packing data into structs to avoid padding, with getproperty/setproperty! overloads making them feel like ordinary structs. They differ substantially in philosophy and what you put in the fields.

Core abstraction

	PackedStructs.jl	FieldFlags.jl
Macro(s)	@packed	@bitflags, @bitfield
Field unit	Types (Int4, UInt24, Float32, wrapper/nested structs)	Bit counts (a:3), or single bits for flags
Mental model	“A normal struct, but fields with non-power-of-two types share storage”	“C-style bitfield: declare how many bits each field occupies”
Field type back out	The original declared type (e.g. foo.first isa Int4)	Bool for width-1, otherwise UInt (custom return types are a planned feature)

# PackedStructs — types drive the layout
@packed struct Foo
    first::Int4
    second::Int4
end

# FieldFlags — bit widths drive the layout
@bitfield struct Foo
    a:1
    b:2
    _:7      # explicit padding
    c:3
end

Where they overlap

• Both overload getproperty/setproperty! (true getfield won’t work; introspection sees internal storage slots).
• Both support mutable struct with in-place field assignment.
• Both round/collapse layout at macro-expansion time, so type parameters are rejected in both (the bit widths must be known statically to compile away the shifts/masks).
• Both target negligible overhead via constant-folded bitmask operations.

Key differences

1. Padding & sizing. FieldFlags makes padding a first-class concept (_ and _:n) and always rounds the whole struct up to a multiple of 8 bits (a compiler limitation it documents). PackedStructs derives padding implicitly from the types and packs into Pack<B> groups sized to native widths (8/16/32/64).

2. Field richness. This is the biggest divergence:

• PackedStructs handles arbitrary isbits types — primitives, BitIntegers/EmulatedBitIntegers, wrapper structs for type safety, and nested immutable structs that compose by logical bits. It’s extensible via PackedStructs.pack (e.g. to pack Float32/Char).
• FieldFlags stores raw integer bitfields and boolean flags only; custom field types are a documented future plan (#9). Assignments silently truncate leading bits.

3. Ecosystem integration. PackedStructs ships a ConstructionBase extension, so Accessors.@set/@reset (and Setfield, BangBang, StructArrays) work — with a documented load-order requirement. FieldFlags has zero dependencies and emphasizes simplicity, JET debuggability, and subtyping (<: MyAbstract).

4. Dependencies. PackedStructs depends on EmulatedBitIntegers, ExproniconLite, PrecompileTools (+ optional ConstructionBase). FieldFlags is dependency-free.

When to pick which

• FieldFlags.jl — C-style bitfields/flag registers, hardware/protocol bit layouts, where you think in bit counts, want zero deps and a minimal surface.
• PackedStructs.jl — you think in types (especially EmulatedBitIntegers/BitIntegers), want fields to round-trip as their declared types, need nested structs or float/char packing, or want Accessors.jl support.

PackedStructs’ own README already nods to this: “FieldFlags.jl uses a similar approach, but focusing more on logical bits (fields) than on integers.” The inverse framing also holds — PackedStructs focuses on typed values, FieldFlags on bit widths.

[Sukera]

Well.. Claude certainly has a way of interpreting things in such a way that they don’t at all align with my own mental model of the package The summary by the LLM seems more like a dissection based on surface level words than what’s already being tracked in the issue tracker Either way, thanks for the reply!

[PatrickHaecker]

Hmm, Claude’s summary certainly matched my mental model of both packages. However, I never went through the complete code of FieldFlags.jl, so my mental modal seems to be biased, too. How would you phrase it? Claude might not learn from it, but I probably would.

[Sukera]

Well, the codebase was originally designed with the eventual extension to isbits typed fields in mind, that’s why the issue was created not long after the initial commit was made It’s just that I prefer to have documentation about what works right now, without explicitly writing every possible extension down into the docs. At the time I indeed needed it for interacting with registers, and adding typed fields to enforce invariants would be the logical next step, I just didn’t want to make a decision about how to handle padding in isbits structs. I’m actually curious how you are handling it for things like that, where a type has internal padding?

[matthias314]

The combination EmulatedBitIntegers.jl + PackedStructs.jl is somewhat related to the type PackedVector from SmallCollections.jl. See my post in the thread for EmulatedBitIntegers.jl.

[PatrickHaecker]

@Sukera

I’m actually curious how you are handling it for things like that, where a type has internal padding?

We haven’t used it for use cases where an external bit layout with specified internal padding needs to be used. We are using it to increase performance and save memory in optimized data structures without compromising on readability.

However, I agree that it is useful and just pushed it as v0.3.1. You can now write

@packed struct Frame
    a::UInt1
    b::UInt1
    _::Pad{5}
    c::UInt1
    d::Int8
end

Thanks for the trigger, @Sukera!

[Sukera]

Ah, it seems you’ve misunderstood me - I was referring to type annotations like this:

struct Foo
    A::Int8
    B::Int64
end

@packed struct Bar
    C::Int7
    D::Foo
end

Is this also supported, and how are you dealing with the internal padding in Foo if it is? This is the main reason why I haven’t added a generic type annotation feature to FieldFlags.jl

[PatrickHaecker]

@Sukera

Is this also supported, and how are you dealing with the internal padding in Foo if it is?

I deliberately let this unspecified. Currently, there is no reordering or merging done in these cases. Instead padding is always added at the end of one “group” as it is done natively in Julia (which you can observe in Foo). So the memory layout for both types is the following:

If you want to have a guaranteed layout, you can use Pad. Otherwise, the defaults and therefore the layout might change in the future. If you want to optimize the layout across data types, you currently need to do this yourself. I am not yet sure how the best tradeoff looks like here. On the one hand, Julia promises a C compatible memory layout. On the other hand a Rust-like approach of reordering types for best performance while semantically not changing anything for the user also seems to be attractive.

[Sukera]

@PatrickHaecker

Currently, there is no reordering or merging done in these cases. Instead padding is always added at the end of one “group” as it is done natively in Julia (which you can observe in Foo)

What do you mean by this, what is a group? I thought @packed would remove padding, unless explicitly specified - or does it only pack the special types from BitIntegers.jl?

[PatrickHaecker]

@Sukera

What do you mean by this, what is a group?

Internally, PackedStructs.jl groups fields together to fit into a native type. This is using an optimization algorithm, which I think always finds an optimum solution (but I lack the proof). The reason is that only then there is hardly any performance difference. You can check the source to learn about the optimization problem if you are interested in the details.

@Sukera

I thought @packed would remove padding, unless explicitly specified - or does it only pack the special types from BitIntegers.jl?

It only packs what agrees to the pack interface. The pack interface is already defined for some types (EmulatedBitIntegers, regular integer, isbits structs), however, Foo is not a @packed struct and does not override the default pack interface and therefore sticks to a C-compatible memory layout. This means that Foo must have 7 bytes of padding between the two fields and it must be aligned on a memory address dividable by 8.
Therefore, Bar, although being a packed struct can’t pack anything in this case without reordering fields.

As written, it might make sense to have something like @crammed struct which does not stick to a C-compatible memory layout giving the optimization much more degrees of freedom by reordering fields.