[ANN] Ark.jl: archetype-based entity component system (ECS) for games and simulations

Yes, here “agents” has a different meaning: it’s a way to specify the entities of your system. Applications can be of many types, e.g. many games use this to handle the objects (which are the entities in this case) and their interaction. One possible direct use case in Julia is to produce similar simulations which could be alternatively modeled with Agents.jl. But being a general methodology it can be used in many settings. In this thread it was discussed and coded an individual-level SIR model, this is e.g. a simple concrete example.

More generally, an ECS is an architectural pattern which allows to separate data and behaviour more than other approaches while still mantaining high performance, unlike using a single/multiple arrays of structs directly for example.

I guess a comparison of Ark.jl (or another Julia ECS package) vs non-ECS solution of the same simple problem would help?.. If I understand you correctly, it helps with performance in some scenarios?

Sorry if the questions sound trivial, you are talking to someone who never encountered this kind of agent-based modeling (hope that’s the right term for this approach) :slight_smile:

That‘s definitely an option. But I wonder whether making the stateful iterator handle this wouldn‘t be preferable to making this a user problem. Stateful iterators are quite uncommon in the ecosystem, so I don‘t know about how others deal with this issue of exceptions leaving the iterator or other captured objects in a broken state. But from a user perspective, reducing the number of things to keep track of when using the API is very helpful.

I‘m doing simulations with many different types of entities with many types of interactions. ECS allows me to disentangle the individual behaviors and associated data (states). The way this simplifies simulation code may be hard to convey with simple examples, it really starts to make a difference when you scale up complexity. I find that inheritance based architectures become unmanageable rather quickly.

So the primary use case for me is architectural and pays off even for smaller systems in terms of the sheer number of entities, and using a good library over a naive implementation makes using this pattern feasible from a performance perspective when the number of entities becomes larger.

1 Like

Maybe we can just add a unlock!(world) method

I think this is not a good solution, as I see the risk that users will use it to try to get out of an eligibly locked state, or to get rid of queries that just forgot to close. Instead, we can add is_closed for queries.

Regarding handling by the query iterator, I do not really see how this would be possible. The iterator does not “see” what is going on in the loop, so catching in iterate would have no effect if I am right.

You‘re completely right, I confused myself there.

I think I‘ll wrap this API with a functional do block convenience wrapper on the application side.

Sorry for the noise.

Sorry for the noise.

No problem, really happy do discuss anything here!

Thank you @aplavin for pointing this out. As ECS it not yet used a lot in modelling, this is definitely a valid point.

I added a short README section hoping to convince more modellers to give ECS a try.

Hi, it seems a really interesting and powerful package. I wonder if it is possible to use it to implement the Gillespie algorithm - Wikipedia when the number of reaction species is huge?

If the algorithm can be expressed as reactions between individual molecules, it should be possible.

I would also be interested to compare ECS to the approach in Agent-based modeling via graph rewriting – AlgebraicJulia blog . The overlap with ECS seems significant. Maybe one could use the wiring diagrams produced by such algebraic techniques as systems, or in other words, use Ark.jl as a backend execution engine for running algebraically constructed models.

3 Likes

If Ark.jl could implement the abstract ACSet interface (ACSets.jl/src/ACSetInterface.jl at main · AlgebraicJulia/ACSets.jl · GitHub) it could be possible. I’m not clear right now what are the morphisms in an ECS. I agree that there is a strong overlap which is worth looking into, I have been investigating Ark.jl with similar thoughts.

1 Like

Here is the full example of using Ark.jl to implement the SIR model which I referred to earlier in the thread sir-julia/markdown/markov_ark/markov_ark.md at master · epirecipes/sir-julia · GitHub. A reasonable comparison might be to compare to an OOP language, it is common to see C++ or Python agent-based simulation code implementing some “Person” class, maybe with subclasses, etc.

1 Like

Thanks for the example! I’m just trying to understand what benefits ECS brings here? Compared to a trivial world = (S=123, I=456, R=789) (namedtuple of counts) that should be super performant.

Or something like world = Vector{Symbol} if one really needs individual entries in the world for each entity.

1 Like

To see the improvement from a performance and maybe also architectural perspective, you need to think more…heterogeneously. Let’s take for granted that individual entries are useful. Now let’s say you introduce in the SIR model some other entities having a different set of components, e.g. agents which represent multiple other species not susceptible to the infection and having characteristics which are not shared from one species to the another (but among individuals of that species). Also consider that old individuals could die, and new individual could be born, also this is supported well with this pattern. All in all, ECS are better than alternatives because they are built to work with deep heterogeneity. But also with low heterogeneity models like this one they are very fast.

1 Like

Even on this simple example Ark.jl surpasses a Vector{Symbol} by 2-3x.

utils:

using Random

function rate_to_probability(r::T, t::T) where {T<:AbstractFloat}
    1 - exp(-r * t)
end

const rng = Xoshiro(42)

# Time domain parameters
const δt = 0.1
const nsteps = 400
const tf = nsteps * δt
const t = 0:δt:tf;

# System parameters
const β = 0.05
const c = 10.0
const γ = rate_to_probability(0.25, δt);

# Initial conditions
const N = 1_000_000
const I0 = 5;

Ark.jl version (on dev):

using Ark

abstract type HealthState end
struct S <: HealthState end
struct I <: HealthState end
struct R <: HealthState end;

function get_count(world, ::Type{T}) where {T<:HealthState}
    count = 0
    for (entities, ) in Query(world, (), with=(T, ))
        count += length(entities)
    end
    return count
end

function run_sir()

    # set up world, add initial entities
    world = World(S,I,R)
    new_entities!(world, N-I0, (S(),))
    new_entities!(world, I0, (I(),))

    # output
    trajectory = zeros(Int, nsteps+1, length(subtypes(HealthState)))
    trajectory[1, :] = [get_count(world, T) for T in (S, I, R)]

    # vectors to contain Entities making state transitions on this step
    i_to_r = Entity[]
    s_to_i = Entity[]

    # vector to contain uniform random numbers for state transitions
    rands = Float64[]

    # run sim
    for t in 1:nsteps
        # S->I
        i = get_count(world, I)
        foi = β * c * i/N
        prob = rate_to_probability(foi, δt)
        for (entities, ) in Query(world, (), with=(S, ))
        	resize!(rands, length(entities)); rand!(rng, rands)
            @inbounds for i in eachindex(entities)
                if rands[i] <= prob
                    push!(s_to_i, entities[i])
                end
            end
        end
        # I->R
        for (entities, ) in Query(world, (), with=(I, ))
        	resize!(rands, length(entities)); rand!(rng, rands)
            @inbounds for i in eachindex(entities)
                if rands[i] <= γ
                    push!(i_to_r, entities[i])
                end
            end
        end
        # apply transitions
        for entity in s_to_i
            exchange_components!(world, entity, add = (I(),), remove = (S,))
        end
        for entity in i_to_r
            exchange_components!(world, entity, add = (R(),), remove = (I,))
        end
        # record state
        trajectory[t+1, :] = [get_count(world, T) for T in (S, I, R)]
        # reuse vectors to avoid allocations
        resize!(i_to_r, 0)
        resize!(s_to_i, 0)
    end
    
    return trajectory
end

@time run_sir(); # 0.439895 seconds

Vector{Symbol}:

function get_counts(model)
    count_S, count_I, count_R = 0, 0, 0
    for agent in model
    	if agent == :S
        	count_S += 1
        elseif agent == :I
        	count_I += 1
        elseif agent == :R
        	count_R += 1
        end
    end
    return [count_S, count_I, count_R]
end

function run_sir()

    model = Symbol[]
    for _ in 1:N-I0
    	push!(model, :S)
    end
    for _ in 1:I0
    	push!(model, :I)
    end

    trajectory = zeros(Int, nsteps+1, 3)
    trajectory[1, :] = get_counts(model)
    for t in 1:nsteps
        i = get_counts(model)[2]
        foi = β * c * i/N
        prob = rate_to_probability(foi, δt)
        @inbounds for i in eachindex(model)
            if model[i] == :S && rand(rng) <= prob
                model[i] = :I
            elseif model[i] == :I && rand(rng) <= γ
            	model[i] = :R
            end
        end
        trajectory[t+1, :] = get_counts(model)
    end
    
    return trajectory
end

@time run_sir(); # 1.317357 seconds

Optimized Vector{Symbol} → going more unreadable:

function get_counts(model)
    count_S, count_I, count_R = 0, 0, 0
    for agent in model
    	if agent == :S
        	count_S += 1
        elseif agent == :I
        	count_I += 1
        elseif agent == :R
        	count_R += 1
        end
    end
    return [count_S, count_I, count_R]
end

function run_sir()

    model = Symbol[]
    for _ in 1:N-I0
    	push!(model, :S)
    end
    for _ in 1:I0
    	push!(model, :I)
    end

    trajectory = zeros(Int, nsteps+1, 3)
    trajectory[1, :] = get_counts(model)
    for t in 1:nsteps
        i = sum(1 for agent in model if agent==:I)
        foi = β * c * i/N
        prob = rate_to_probability(foi, δt)
        count_S, count_I, count_R = 0, 0, 0
        @inbounds for i in eachindex(model)
            if model[i] == :S
            	count_S += 1
            	if rand(rng) <= prob
                	model[i] = :I
                end
            elseif model[i] == :I
            	count_I += 1
            	if rand(rng) <= γ
            		model[i] = :R
            	end
            else
            	count_R += 1
            end
        end
        trajectory[t+1, :] = [count_S, count_I, count_R]
    end
    
    return trajectory
end

@time run_sir(); # 0.884714 seconds
1 Like

This specific example is just a straw man, of course for a SIR model interpreted as a simple discrete time Markov chain, we would prefer to store state as a length 3 vector. But ECS, and really any framework that lets the modeler think less about intricacies of how state is stored and how to implement efficient queries and state updates, and instead focus on the system under study is a big win when working with large models with intricate state (floating point values for individuals, spatial structure) and complex transition rules.

I’m no longer active in academic research in this field anymore so these investigations are somewhat of a hobby, but the complexity level of a model like GitHub - SwissTPH/openmalaria: A simulator of malaria epidemiology and control is a good example of the amount of complexity these things can accrue.

1 Like

Here are two examples of models that I developed in the past. As you will see, “agents” have tens of state variables, there is hierarchical structure, and the model descriptions are dozens of pages. It is impossible to express these models in a few equations.

A model for BVD in the cattle population of Ireland. Millions of animals, more than 100k herds. The animals are of different age classes and designation with individual variability, complex disease dynamics, 16 different herd types with complex management of their livestock, different surveillance and eradication strategies, …

Paper: Redirecting
Full text PDF: UFZ - Publication Index - Helmholtz-Centre for Environmental Research
Complete model description: https://ecoepi.eu/ecoepi/supl/Brock-2021-PVM_ODD.pdf

A model for African swine fever in wild boar, on a regional to continental scale. Dozens of state variables per entity, hierarchical structure with social groups and individualy, a multitude of possible management measures, … This one is used for decision support on the EU level over many years.

Latest paper: https://doi.org/10.1080/22221751.2022.2146537
Complete model description: swifco-rs documentation — swifco-rs 0.1 documentation

7 Likes

Thank you for preparing this. This is really great to see, and I would not have expected Ark.jl to provide a significant performance advantage already for such a simple model. :rocket:

3 Likes

@mlange-42 do you have any plans (shorter or longer term) for GPU-support in Ark.jl?