Brainstorming about a game engine's architecture

New to Julia
1

Hello everyone, I have been thinking about making a game engine, so I did a little brainstorming about his architecture but since I new to Julia and programming in general (I started just 1.5 years ago as a self taught), every feedback is welcomed
Also, my brainstorming are sometimes messy😅 so I use an AI to structure it, so if there is something you don’t understand, don’t hesitate to ask
The original brainstorm is in the comments

Architecture Analysis for the Cruise Engine

Structure Overview

Utilities  
└── Main System  
    ├── OutDoor  
    │   └── Horizons  
    └── Interaction

Is an Event-Driven Architecture (EDA) a Good Choice?

We need to determine the optimal architecture for the engine’s subsystems. Each module (e.g., physics engine, graphics engine) must remain independent but share data efficiently. Below, we analyze potential solutions.

Key Requirements:

  • Decoupling: Subsystems must not directly depend on each other.
  • Data Sharing: Efficient communication between subsystems.
  • Scalability: Adding new modules should not disrupt existing workflows.

Proposed Architectures

1. Linear Process Order

  • Approach: Define a fixed execution sequence where each subsystem passes data to the next.
  • Pro:
    • easy to implement
  • Issues:
    • Inflexible: Adding/modifying subsystems requires reordering the entire pipeline.
    • Tight coupling: Changes risk cascading errors.

2. Serialization-Based Architecture

  • Approach: Subsystems write results to files/directories for others to read.
  • Pros:
    • Decentralized: Subsystems operate independently
  • Cons:
    • Latency: File I/O is slow.
    • Platform-Specific files management
    • Synchronization risks: Subsystems might read stale data.
    • Requires strict read/write ordering or waiting mechanisms.

Optimizations:

  • Use I/O buffers instead of files for faster access.
  • Define a shared directory structure with clear read/write rules.

3. Asynchronous Message Queue (Event-Driven Architecture)

  • Approach: Subsystems publish results to a shared message queue/bulletin board. Others consume data as needed.
  • Pros:
    • Decoupling: Subsystems interact via events, not direct calls.
    • Scalability: New modules subscribe to relevant queues.
    • Concurrency: Asynchronous processing improves performance.
  • Cons:
    • Complexity: Debugging event flows is challenging.
    • Ordering: Requires careful handling of event priorities.

Example: The Godot Engine uses a notification system where subsystems emit signals (e.g., physics updates) to modify a central object’s state.

Proposed Solution: Hybrid Event-Driven Architecture with Observers

Design Overview

  1. Central State Management:

    • A main system tracks the global state of objects.
    • Subsystems (e.g., physics, graphics) act as observers that modify this state.

  2. Event Streams:

    • Each subsystem publishes updates to dedicated streams (e.g., PhysicsUpdates, RenderQueue).
    • Subsystems subscribe to streams relevant to their tasks.

  3. Workflow:

    • Subsystems process data and publish results to their stream.
    • The main system updates the global state and propagates changes to dependent streams.
    • Example:
      • Physics engine → Publishes position updates → Main system → Graphics engine renders new positions.

Advantages:

  • Loose Coupling: Subsystems only interact via streams.
  • Error Handling: Wrap subsystems in tasks; failures don’t block the entire pipeline.
  • Extensibility: Add subsystems by connecting them to existing streams.

Implementation Steps:

  1. Define subsystem types (e.g., Physics, Graphics).
  2. Implement a ListenToStream method for subsystems to receive updates.
  3. Create a function to publish the results of your subsystem

Conclusion: A Multi-Stream Architecture (MSA)

The final design combines event-driven principles with observer patterns:

  • Event Streams: Replace direct dependencies with asynchronous queues.
  • Central State: A single source of truth for object states.
  • Caching: Optional caching for fault tolerance (e.g., fallback to last valid state).

This approach avoids the pitfalls of file I/O and rigid process ordering while retaining the flexibility of EDA.

Notes on Terminology

  • Multi-Stream Architecture (MSA): A custom term emphasizing parallel, decoupled event streams.
  • Observables: Inspired by reactive programming, where state changes propagate automatically.

This architecture balances performance, scalability, and maintainability for the Cruise Engine.

1 Like
2

If you are interested in my original brainstorm, here it is

Structure

The structure is as follow
Utilities||____ Main System — OutDoor | | | |_ Horizons | Interaction

Is an Event driven Architecture a good choice ?

Here we will analyze which architecture should be used for the engines.Each part or module of the engine is independent from each other. The physics engine should not depend on the graphics engine, etc. but we need them to pass data to each other. So, how do we achieve this ?We have choice :

  • We can create a process order for the different functionality and then every step will pass his data to the next step. The problems is that if we want to introduce a new module we will need to change the order which will lead to great problems.
  • We can serialize data and make them available for whoever need them. It can be a good approach if we can make the serialization keep up with the needed performance and the available resource. We could have the different sub sustem doing their things and just fetching data from a file when needed. problem, files are slow but we will keep this in mind.
  • We can use an asynchronous system. Each sub system will be connected to an event bus or message queue and when they need a piece of data, they just take them cause it’s exposed on the event bus. This approach is relatively quick and seem good but is harder to understand since the order in which features interact with each other is complicated and debugging became more difficult.For me, I want something like a bulletin board, after processing for the current stage, object put the relevant on the bulletin board so every other sub system can be informed. But how will it be done ? Synchronously ? Asynchronously ? With Serialization ? We need the be careful on this step 'cause it will impact the interaction of all sub system and will be a pain to change. Let’s explore the different fields to see how everything will be globally done in each of them.## Serialization Solution ##Here is how the architecture with serialization will be like.A subsystem process data, once finished, they write the result into a file and then put them on a repertory for it to be use by other sub system. when another substem need data from the previous substem, it just read the data.One problem may be if a substem read old data before the new one have been written. We can just just add an order for writting and reading. if a sub system need a data from another, we make that sub system execute first and write data then the other can pass. Or we can make it wait but that would be difficult to define when a sub system should wait for new data.Also, there may be some plateform dependency or some authrization things, depending on where the software is. But we can easily avoid this by defining a clear repertory for data to be written. And then we can make different releases for different platform so that everything will be done for that platform.Lastly, files are slow. this architecture imply to constantly work with file, which mean that their access(reading and writting) should be optimized to fit the needs. Maybe IOBuffers may be the solution. They are decoupled from the platform and requires less than files but we have then the problem of passing these IOBuffers around.We may consider a sub system buffer as a sheet of paper, the sub system put the relevant data in it and then post it on the bulletin board so that if another sub system is interested, it just have to read it.But how do we publish this on the bulletin board ? We may do it like in pokemon dungeon mystere. at the start of the process every sub system start their things. Went one finish a task, he publish the result on the bulletin board. Other task waiting for these result take them and do calculation on it and them post it again (maybe with it’s result.). To avoid concurrency, when a piece of information has already been taken, the other’s are forced to wait for it to come back. All this take us to one thing. We need a way for sub system to wait, publish task, process data, read data all this at the same time, We need an asynchronous system.

Asynchronous Solution

Here we propose that a subsystem do a job and when he finish, he post his result on a message queue so that another process can take it. For that, we need this to be asynchronous so that subsystem can do all this at the same time. Also this architecture make it possible to send request or signal to other system. The Godot Engine use some sort of notification system, when the state of an object is modified, we just send a notification that modify his state in the main object. Maybe we can do the same, it will save us the need to directly pass data. For example, if the physics engine has done some physic thing on an object and have modified his position, then he will send a notification to the main system so that the position will be updated. But we talk about sending data. If we directly send data to the main system then why publishing data ? Maybe a sub system will need it but all the information a system need for a given object is contained in that object himself.If it is the case then we only need subsystem that update the state of a main system and then request come in handy. We can just do some processing in a subsystem and then send the request to the main object to make the change. If the main object do the change it’s good if not, the subsystem just continues doing his thing. So we will still have waiting object but not for the same reason. Object will wait for the main object to reach a certain state before doing thing with it. For example, the main object can be in an initial state, then the physics engine process his data , send a notification to the main object for update. The main system will then update the object and then post a reference to the object for example in another message queue, the graphic engine then take that reference and do some processing with it and once done notify the necessary update to the system. Then if necessary, the main system will then post the new data to another message queue, etc.The thing is that every sub system can access any of these message queue(At the condition that they reput! it one done.). If a system need a data from a queue, he just wait for it in that queue. Now if we wanna add a sub system, how will we do it ? We will just plug it on the main system so that it can send notification to the main system. Notification are like a way for sub system to interact with the main system. There is no message sending but don't know yet for who it is.. Sub system only interact with their boss(professionalism it is) not with their mates.

In the end

So, does this fit an Event Driven Architecture(EDA) ?For this we need a little description of an EDA.An EDA is an architecture in which multiple structure interact with each other via a message queue, they don’t directly pass data to avoid coupling. So each services is independent from another.So does our brainstorming has shown us if an EDA was suited for this ? I think not fully. We saw that we need a system of notification on which each system can have his own stream for communication(Notification), that each sub system to actually interact with the world need to use notification and the main system will only need to execute these notifications. We saw that the use of state may be useful so that subsystem know were to deploy themselves also we can use different message queue for example corresponding to every state of an object. Since substem mainly operate on object, we can just put these object on the correct message flow.So this system is more like a Multi-Stream Achitecture(MSA, my creation tho) than an EDA but we will need some features from the EDA for notification 'cause the main object is a not big central tower but more likely a set of structure that have a main goal, keep track of the state. so when a notification is sended, we put it on a message queue and then the part of the main object to which the notification refer will act in consequency (or we can use [Observables] but exploring how they are made is not bad though) So that is it for the architecture of the Cruise Engine, A Multiple Stream Architecture where object wait for a specific message stream to work. if a previous step fail then we have a problem that is the fact that if for example the physics engine fail then no notification will be send and the graphic engine will also display nothing. One solution may be cache so that when an subsystem receive something it’s just to update some cache but that would be expensive(and if the cache is never created it’s a problem.). We can use task so when it’s the turn of sub system to execute, we wrap it in a task and run it. if it failed we continue the process.So if a sub system wanna be connected to the flow he need to :

  1. Maybe define a type for this sub system.
  2. Define a priority for this sub system so that we know on which part of the workflow it will be triggered.
  3. A ListenToFlow function so that he can receive data from the main object.
  4. Process the object.
  5. Then notify that the object is ready to pass to the next stage.

The Big Solution

I think finally the solution to our problem is event, yes but with Observers.We will have a flow of event for different objects. Then if a modification is done on an objectthen that modification is published to a stream and then other subsystem can take it.So we will have many type of subsystem, those who modify objects properties and those who applies the changes. So we can plug Systems at the right place and he will just do his thing.

1 Like