JuliaActors: a road to an Actors ecosystem

[pbayer]

Following the discussion in another thread now we have JuliaActors, a yet small Julia GitHub group of enthusiasts dedicated to build an actors ecosystem in Julia. Since this is not anymore a discussion about one library, I open a new topic.

We have concluded that we should build a Julian approach on the classical Actor Model:

The Classic Actor Model formalises the Actor Model in terms of three primitives: create, send and become. The sequential subset of actor systems that implement this model is typically functional. Changes to the state of an actor are aggregated in a single become statement. Actors have a flexible interface that can similarly be changed by switching the behaviour of that actor. (from 43 Years of Actors)

We will start as a 1st step with a minimal interface (spawn, send and become) satisfying the above definition and allowing actors from different implementations to communicate. This therefore will provide a 4th primitive, a link (as Agha called it) over which actors can communicate.

With that (building on Julia’s primitives) we will be able to run Agha’s examples.

@fborda then pointed out that there is more to actors and that a Julian actor library should be able to compete with languages/ecosystems like Erlang/Elixir and Scala/Akka:

@fborda

you’d probably need a signaling interface, … a way to signal a shutdown command, … a way to link one actor … all of these are necessary …

… it ends up converging on a a standard implementation

Therefore in a 2nd step we have to complement the four elements above with some more like:

5. exit and shutdown,
6. self(),
7. onmessage and an internal messaging protocol which would allow to plugin an API,
8. a registry, which could be a service of a core actor library to other actor libraries using it,
9. …

Other actor libraries reimplementing the first four primitives should be able to plug this in too. This then becomes looking more like a “standard implementation”.

@fborda

I might not be as interested in creating my own implementation of Actors as in the infra-structure around it

Then in a 3rd step we can start to develop it further and to build the infrastructure around it. That is where the real fun starts.

Since we have some work done and some experience what works, I hope that we can pull things together and progress quickly. Keep tuned!

@oschulz contributed his old Actors.jl library . Now that we have kind of a roadmap, I think we can organize work around that one.

[richiejp]

Thanks for organising this. At some point I will rename my Actors.jl back to Luvvy and put it in JuliaActors and rename Luvvy to LuvvyWeb. I’m not sure there is any point in me continuing to work on those with circo and YAActl around, but it’s nice to keep them visible.

[pbayer]

Actors, Luvvy, YAActL, Circo …, those were/are the pioneers. I hope that with a renewed Actors.jl we can soon converge into something bigger. And if we succeed with the common actor interface the pioneer libraries can keep their special place, e.g. for web services, state machines, simulations … and use a modern API and infrastructure. We will see.

Now it is good to put the libraries together and after all our experiences made in developing them. I hope with time more people will join. I’m really looking forward to see how that evolves! Concurrent programming is fun and challenging!

[pbayer]

Good news: Actors.jl has been rewritten and is available as v0.1.1 from the Julia registry. It provides

• the basic functionality of the classical Actor Model and
• a minimal interface which allows other actor libraries to plugin the Actors API and their actors to communicate with each other.

@tisztamo

Defining a minimalistic interface for the classic model seems like a good start.

This proposal of @tisztamo turned out to work very well. In order to demonstrate how that works, I wrote a completely different toy actor library SlowActors.jl with about 100 lines of code. It uses the Actors API, can run the same examples as Actors and actors from both libraries can communicate with each other:

julia> using SlowActors

julia> raise(a, b) = a^b
raise (generic function with 1 method)

julia> query(act, a, b) = request!(act, a, b)
query (generic function with 1 method)

julia> A = Actors.spawn(Func(raise))
Link{Channel{Any}}(Channel{Any}(sz_max:32,sz_curr:0), 1, :local)

julia> B = spawn(Func(query, A))
Link{SlowActors.Mailbox}(SlowActors.Mailbox(Deque [Any[]], ...

julia> request!(B, 2, 2)
4

In that example “slow” actor B queries actor A to raise 2 by 2, takes back the result and delivers it. This works in either direction.

That means that we can start to connect our actor libraries around the Actors interface. How that works is described on the integration page of SlowActors.

Now we can progress to step 2 above to provide Actors with a modern API, a registry, a supervision tree …

[tisztamo]

Thanks, it looks great at first sight! I don’t have much time this week, but I will get back to this soon and try to implement it in Circo.

For now only a single point: Moving the interface to its own package would be very helpful in my opinion, for the following reasons:

• It would be trivial to see what is implementation detail that is possible to replace, and what is interface that needs to be supported. E.g.: The status of Func, Call, Request and Response is not clear at first sight.
• It would allow Actors.jl to grow without hurting other implementations with unnecessarily loaded code.

[pbayer]

For sure it can yet be improved.

@tisztamo

Moving the interface to its own package would be very helpful …

But I don’t think this to be a very good idea for several reasons:

• The biggest part in the library will be the messaging protocol if we come to the API functions, supervision … This stuff must remain together.
• The core implementation of actor primitives (newLink, spawn, and the actor loop) is actually quite small and will not grow much.
• This is also needed internally if we want Actors to provide a registry for all libraries using it. A registry is part of a modern actor API. It must be kept at a central place. Actors will therefore contain a registry server.
• As far as I can see Actors will remain a small library: registry, error handling, grouping actors, monitors and the supervision tree are not very much code.
• Sockets, TCP, … protocols with foreign actors can all be implemented by ecosystem libraries.

@fborda

Stuff like Akka HTTP, Akka Streams, Erlang Term Storage/Mnesia, Phoenix PubSub are all tools that I really think would really complement Julia’s current strengths.

As @fborda mentioned infrastructure libraries will become more important than Actors itself. Actors will be a stepping stone for those.

But yes, even if the interface is already small, we should try to simplify it further. Thank you for keeping me thinking .

BTW:

@tisztamo

The status of Func , Call , Request and Response is not clear at first sight

Those are all part of the messaging protocol. Explanation of the messaging protocol is still missing in the documentation. Func is needed to send and store functions and arguments.

[datnamer]

Pitch #2: Protocol-based Actor Isolation - Pitches - Swift Forums second iteration

[pbayer]

Thank you for sharing this:

a key question is: “when and how do we allow data to be transferred between actors?”

Their approach is:

The ActorSendable protocol models types that are allowed to be safely passed across actor boundaries by copying the value. This includes value semantic types, references to immutable reference types, internally synchronized reference types, and potentially other future type system extensions for unique ownership etc.

In a dynamically typed language we won’t declare types as ActorSendable. Furthermore if we restrict strongly the types users can send to an actor without being deeply copied, we constrain users too much in building functionality. But I agree with the Swift people that we must give them more control of shared mutable resources. Since in the classical Actor Model resources can only be shared between actors by

1. passing them as arguments when spawning behavior functions and
2. sending them explicitly to actors,

users have some control of what variables are shared (Here I assume strongly that a user avoids global mutable variables!). I don’t consider the first method as a problem since a programmer can take care that an actor is not created with a shared mutable variable, but which variables are sent between actors should be made more controllable.

3. A protocol could be implemented into the send! function. If a Link is marked :safe, it could mean that mutable variables are copied by default. A programmer could then explicitly override this mechanism with a keyword argument.
4. Next as pointed out by Gul Agha (in Ch. 6 Concurrency Issues) a “resource can still be modelled as a system of actors so that there may be concurrency in the use of the resource”. Therefore an Actor library can provide a mechanism to wrap a shared variable into a messaging protocol.
5. Finally we still can follow the advice in the Julia manual to wrap mutable variables into a lock.

[tisztamo]

I had hard time mapping the representations I use in Circo to the interface in Actors.jl. This was partly because the interface is a bit too restrictive (e.g. Link is not an abstract type but a UnionAll struct; pid is the concrete Int), but more importantly because the model of Actors.jl is an extension of the model currently implemented in Circo (e.g. a Link - although I am not sure I understand its distributed semantics - is more than an address, because it has some state).

To solve the issue I have created a proposal formalisation of the actor model, with the hope that it can be the base model not just for Actors.jl and Circo, but hopefully for future implementations.

The goal is to create an API specification that is independent from any implementation and can be used to write actor programs that run on potentially many actor system. I think this is not exactly the same goal as what Actors.jl wants to achieve, e.g. running actor schedulers from different libraries in the same Julia process is not a goal here. The goal is to not naming any implementation in actor code, thus being able to run the code on several implementations (Similarly to Java EE, POSIX, etc.).

As there is not a single “the” actor model, but it seems that the “Classic” Agha version is the base of the others, ActorInterfaces.jl defines the model as the classic one and a set of extensions. I have worked only on the classic model, it is usable now for primitive examples, and really minimalistic with the hope that it can be compatible with any implementation. I created the “Armstrong” extension as a stub, it is yet to be filled. Maybe another extension will be also needed for Actors.jl.

I have also created a minimalistic reference implementation of the model, called QuickActors.jl A sample program written in the model:

using ActorInterfaces.Classic

struct Spawner end  # Actor behavior (and empty state)

struct SpawnTree # Command to spawn a tree of actors
    childcount::UInt8
    depth::UInt8
end
Classic.SendStyle(::Type{SpawnTree}) = Sendable() # CopySendable and Racing is also defined

function Classic.onmessage(me::Actor{Spawner}, msg::SpawnTree)
    if msg.depth > 0
        for i = 1:msg.childcount
            child = spawn(me, Spawner())
            send(me, child, SpawnTree(msg.childcount, msg.depth - 1))
        end
    end
    return nothing
end

To run this program with QuickActors.jl:

using Test
using QuickActors
import ActorInterfaces.Implementation.Tick # Just a test to see if scheduler lifecycle can also be standardized

const TREE_HEIGHT = 19
const TREE_SIZE = 2^(TREE_HEIGHT + 1) - 1

s = QuickScheduler()
rootaddr = spawn!(s, Spawner())
send!(s, rootaddr, SpawnTree(2, TREE_HEIGHT))
println("Building a tree of $TREE_SIZE actors and delivering the same amount of messages")
@time while Tick.tick!(s) end
@test length(s.actorcache) == TREE_SIZE

Building a tree of 1048575 actors and delivering the same amount of messages
 0.734399 seconds (4.19 M allocations: 170.170 MiB, 36.28% gc time)

QuickActors.jl, of course is not based on @async, so it does not handle blocking. But I defined the classic model to leave open this and other questions so that extensions can fill it. I think that abstracting out the interface stuff from Actors.jl is possible, and it will open the possibility to write a bridge between Circo and Actors.jl - either as a separate package or as part of Circo.

It would be nice to get some feedback on the Classic model, as defined in

https://github.com/JuliaActors/ActorInterfaces.jl/blob/main/src/Classic.jl

I am especially interested in your thoughts on the SendStyle trait, which is meant to solve the sharing issue julianly, without prohibiting shared state but also with help to write safe code. There are further examples in the QuickActors.jl tests.

EDIT: The JuliaActors logo is great!

[pbayer]

Great, this is definitely a step forward. There are many ideas in your proposal and I have not yet digested them all. Therefore now I can only give you my preliminary thoughts on it.

YES, I think we can put the primitives send, spawn, become and also onmessage and addr into a separate library as in your ActorInterfaces.Classic. (BTW: we should still call send! in a Julian way)

NOT SURE yet about behavior and Actor{TBehavior}:

• Why should the behavior be “of any type” and not a function and its (maybe partial) arguments and why should it hold actor state?
• The same goes for TBehavior. Why is it necessary?

As you may have noticed, I have come recently to the conclusion that I need an actor mode, a method to express its behavior on a more abstract level as GenServer, Agent or Supervisor … Do you mean something similar?

BUT: I don’t yet buy into the SendStyle mainly for the same reasons, I don’t hail the Swift proposal linked above. Please help me with some of my reservations:

• In order to realize actor protocols we will wrap sent variables into Call, Cast, Exec, Update … message bodies (This is extremely useful). We would have to define all of them as Sendable and then still we can send any dangerous stuff with them. Problem not solved!
• Even without an actor protocol, a programmer may want to send, say a Dict to an actor to make it serve that. She is pretty sure that other actors don’t use it, so it is safe to send it. Should she then declare all Dicts as sendable? Not really! Since programmers are inventive she will circumvent the problem by defining a SendableDict wrapping her Dict or declaring a Ref{Dict} as sendable and then send that one. Better! But I suggest that a programmer should be able to decide on a concrete basis to unlock sending of unsafe data like send!(servy, Sendable(mydict)) and I cannot see how to do that with a type annotation.
• Then the actor should not be stressed with choosing which messages are safe as in your onmessage. It should be sure that any arriving messages are safe. So our Maxwell’s demon has to sit in the send!.

I am not even sure that always copying mutable data is the best approach to the problem. I’m referring to section 6.1.3 in Gul Agha, Actors where he wrote:

The mutual exclusion problem arises when two processes should never simultaneously access a shared resource. … Although, a single receptionist may control access to a resource, the resource itself can still be modeled as a system of actors so that there may be concurrency in the use of the resource. … In general, a programmer using an actor language need not be concerned with the mutual exclusion problem.

So the Actor Model way to tackle the problem may be to write an actor library, say Guards, and then we can do using Guards. If someone then tries to send! a mutable variable, it will spawn a guard actor serving the variable and send! its link to the consumer. The consumer then can do get, or call the guard actor to do getfield, +=, setfield! and everything allowed for the variable’s type. I don’t know if we can abstract even that away so that one could do those operations directly on the guard actors link. What do you think?

@tisztamo

The JuliaActors logo is great!

I took Agha‘s figure 3.2, but it can still be improved.

Edit, addendum: If we send a Guard{VariableType} containing the link of the guard actor, we could probably abstract away the link and the guard actor by enhancing Base.+ and so on on Guard{T} types. Can this be a solution also for common race conditions in parallel programs? We will see.

[tisztamo]

Thanks for working through the proposal! There is a lot to discuss and settle, let’s start!

@pbayer

NOT SURE yet about behavior and Actor{TBehavior} :

How we represent the behavior of an actor is definitely the most important question, and unfortunately there is already two “schools of thought” in this regard. Actor{TBehavior} tries to give a common representation that can handle this two (maybe also future ones).

To be sure that we speak about the same, here is my understanding of the two schools:

• In Circo, in richiejp/Actors.jl (and in QuickActors.jl) the behavior of the actor and its state are essentially the same thing: a typed value, where the type is used by the lib to dispatch messages and the value holds the state. A single multiple-dispatched function is used to deliver messages to actors (e.g. onmessage(actor, msg) ). I call this the OO approach, although it is less rigid than traditional object orientation: The type of the behavior can be changed, and it will change the type of the actor itself. There are significant implementation differences, but the essence is the same IMO.

• In YAActL.jl and in Actors.jl the behavior is a function. The lib dispatches messages to this function, providing also the state of the actor, which is stored somewhat separately, and which is not a single object but an argument list. I call the the YAActL approach. (Not sure but the fact that the function is wrapped in a Func struct seems like implementation detail to me that can be hidden from the user.)

Trying find the common model

The main difference seems that the YAActL approach allows to change the type of the state separately from the behavior function, and that the state is not a single object. To be honest, I do not see this as a real advantage. I feel that it is easier to reason about an actor if at any given time its behavior is represented by a named struct. So I think that the model should support the OO approach. We should find a bridge that makes the two compatible.

I hope we can do that on the basis of Actor{TBehavior}, but I am not exactly sure. I came to this form while searching for an ideal of the classic model: An actor is described by its behavior, hence TBehavior, which is a type provided by the user. The Actor part is for libraries to have different actor implementations. The user ideally never names a concrete Actor type, they are created internally.

Note that Circo uses simply Actor to mark state objects. I do not insist on the parametrized form, just have the feeling that this is the essence of what an actor is, and it allows different implementations. (I will have to rewrite all my actor code to support this, and the current Circo interface was chosen for maxing out performance, and I only hope to get the same when implementing this)

If you could find a way to use something like Func as TBehavior, that would be the solution!

A big problem seems like the become! call, which has two different forms in Actors.jl: One for changing state, another for changing behavior and state. In the OO approach, it seems impossible to provide those separately, because they are encoded in the same type. I do not yet see a solution here but I feel it exists. I hope you will find it!

I think we should settle this before discussing the other issues (naming, SendStyle etc), because this one is the basis for everything, and thinking a bit more on the others before discussing may also be beneficial. I plan to read/reread some relevant papers and check Ponys approach.

[pbayer]

@tisztamo

If you could find a way to use something like Func as TBehavior , that would be the solution!

I think I can go with defining

1. mutable struct _ACT{TBehavior} <: Actor{TBehavior} .... end, then
2. A = _ACT{Func}(...), then

julia> A isa Actor
true

if it helps. Still I don’t understand very much its benefit other than characterizing actors in a very abstract way.

@tisztamo

A big problem seems like the become! call, which has two different forms in Actors.jl

This I don’t understand. become!(lk::Link, func, args...; kwargs...) is only a more convenient form of become!(lk::Link, bhv::Func). Both are essentially the same.

[pbayer]

I had to make Func a non-parametric struct for it to work. This is – I think – less efficient and should be done only for practical reasons. So my question: is it worth it?

PS: this is on https://github.com/JuliaActors/Actors.jl/tree/dev now!

Edit, addendum:

Now I have done measurements:

using BenchmarkTools

f(a, b; c=1, d=1) = a + b + c + d

struct F1{X,Y,Z}
    f::X
    args::Y
    kwargs::Z

    F1(f, args...; kwargs...) = new{typeof(f),typeof(args),typeof(kwargs)}(f,args,kwargs)
end

struct F2
    f
    args::Tuple
    kwargs::Base.Iterators.Pairs

    F2(f, args...; kwargs...) =new(f, args, kwargs)
end

julia> @btime f1 = F1(f, 2, 2, c=2, d=2);
  2.030 ns (0 allocations: 0 bytes)

julia> @btime f2 = F2(f, 2, 2, c=2, d=2);
  7.547 ns (1 allocation: 48 bytes)

julia> @btime f1.f(f1.args...; f1.kwargs...)
  311.602 ns (9 allocations: 432 bytes)
8

julia> @btime f2.f(f2.args...; f2.kwargs...)
  266.178 ns (7 allocations: 224 bytes)
8

Creation of the nonparametric struct is slightly slower but execution is significantly faster. So I will go with it. I will update my PR.

[pbayer]

I would like to comment also on your two “schools of thought” and I think we should differentiate between what a behavior is in the classical model and how we represent it.

I hope, we can agree that the behavior is a function φ(c) of the incoming communication c (see Agha Actors, pp. 24, 30). If we can agree about that, we can reason about on how to represent it. Yet more concisely Agha has in his appendix B:

The behavior of an actor maps the incoming communication to a three tuple of tasks created, new actors created, and the replacement behavior.

(This is also the essence of the JuliaActors logo). Missing in that definition is that an actor may do a computation c→d, such that d becomes an acquaintance, it gets into a acquaintance list while c also can be seen as a list, the communication list.

An actor executes commands in its script in the environment defined by the bindings of the identifiers in the acquaintance and communication lists. (Agha, p. 30)

So our behavior is φ_{i}(d_{i}, c_{i}) → \{d_{i+1},\{τ_{a},τ_{b},...\},\{\alpha_{x},α_{y},...\},φ_{i+1}\}.

Now i find it quite natural to represent that with a Julia function φ(d, c).

1. The actor holds its behavior function φ_{i} and its acquaintance list d_{i}. This can also be called a partial function. In Actors and also in YAActL this is stored in Func (currently a struct but could be also a closure).
2. When a communication c arrives, the actor gets the remaining parameters and it then dispatches φ_{i}(d_{i}, c_{i}).

This is a functional approach and I have yet to see how this can be expressed more elegantly with an “OO” approach. IMHO a functional approach is a better fit to the classical actor model.

Yet a remark about state. In the above view the actor state is represented by the partial function φ_{i}(d_{i})(c), or better yet by the tuple \{φ_{i}, d_{i}\}. But since in a modern implementation an actor should support also generic servers, supervision and so on, it will have also state variables holding links of connected actors, served modules, init and exit functions …, that is, stuff going beyond the classical actor model.

[tisztamo]

After a long discussion with @pbayer we agreed in the first version of the Classic interface, ActorInterfaces.jl is now registered!

I am very happy with the new version, it is cleaner, leaner, but it still captures the essence of the actor model.

Main changes from the previous version:

• SendStyle got removed: Breaking the model by sharing state is possible, help for the programmer to avoid doing it accidentally will be added as an extension interface.
• The Actor type was removed, anything can be spawned as an actor. I think this will allow easier interoperability with non-actor code.
• The “actor context” - which allows the runtime to identify the actor without maintaining global state - is explicitly passed to the onmessage method and must also be explicitly passed to the send, spawn, etc. primitives. There is a convenience macro @ctx that does it automatically.

The mass spawn example now looks like this:

using ActorInterfaces.Classic

struct Spawner end

struct SpawnTree
    childcount::UInt8
    depth::UInt8
end

@ctx function Classic.onmessage(me::Spawner, msg::SpawnTree)
    if msg.depth > 0
        for i = 1:msg.childcount
            child = spawn(Spawner())
            send(child, SpawnTree(msg.childcount, msg.depth - 1))
        end
    end
    return nothing
end

Running it with QuickActors.jl

using QuickActors, Test
import ActorInterfaces.Implementation.Tick

const TREE_HEIGHT = 19
const TREE_SIZE = 2^(TREE_HEIGHT + 1) - 1

s = QuickScheduler()
root = Spawner()
rootaddr = spawn!(s, root)
send!(s, rootaddr, SpawnTree(2, TREE_HEIGHT))
println("Building a tree of $TREE_SIZE actors and delivering the same amount of messages")
@time while Tick.tick!(s) end
@test length(s.actorcache) == TREE_SIZE

Building a tree of 1048575 actors and delivering the same amount of messages
  0.532118 seconds (4.19 M allocations: 170.170 MiB)

Agha’s stack example is also available (see the README), but more lifelike examples are still lacking. The only implementation is QuickActors.jl at the time, but hopefully both Actors.jl and Circo will support the interface soon.

Of course this is only a small subset of the APIs that should be standardized if we want to allow real projects to be written using ActorInterfaces.jl instead of a specific implementation, but I think that having the base model more or less settled is a huge step towards this goal.

[datnamer]

How do you see the Actor stuff interacting with other types of concurrent and parallel code like tasks, distributed, async, GPU, dagger.jl etc?

[pbayer]

Well, it is another concurrency paradigm, but Actors builds on Julia’s concurrency and distributed primitives and thus is basically compatible with other non-actor code in Julia.

With the next major release of Actors v0.2 there will the first actors framework modules available: GenServers and Guards. Those abstract away the actor machinery to an interface which can be seamlessly used in any multithreaded and distributed code. Consider the following code snippets:

julia> using .Threads

julia> for _ in 1:1000    # push asynchronously
           Threads.@spawn begin
               push(st, threadid())
           end
       end

julia> length(info(st))
1000

doesn’t look like actor code, right? Now st is a :genserver stack actor. It helps you to avoid race conditions.

The following is an example with a :guard actor:

julia> incr(arr, index, by) = arr[index] += by
incr (generic function with 1 method)

julia> gd = guard(zeros(Int, 10))
Guard{Array{Int64,1}}(Link{Channel{Any}}(Channel{Any}(sz_max:32,sz_curr:0), 1, :guard))

julia> for i in 1:10
           @threads for _ in 1:nthreads()
               @grd incr(gd, i, 1)
           end
       end

julia> @grd gd
10-element Array{Int64,1}:
 8
 8
 8
 8
 8
 8
 8
 8
 8
 8

and with Distributed:

julia> using Distributed

julia> addprocs(1);

julia> @everywhere using Guards

julia> gd = guard([1,2,3], remote=true)  # a guard with a remote link
Guard{Array{Int64,1}}(Link{RemoteChannel{Channel{Any}}}(RemoteChannel{Channel{Any}}(1, 1, 13), 1, :guard))

julia> fetch(@spawnat 2 @grd gd)         # show it on pid 2
3-element Array{Int64,1}:
 1
 2
 3

julia> @fetchfrom 2 InteractiveUtils.varinfo()
  name               size summary
  ––––––––––– ––––––––––– –––––––––––––––––––––
  Base                    Module
  Core                    Module
  Distributed 918.170 KiB Module
  Main                    Module
  gd             56 bytes Guard{Array{Int64,1}}

julia> @grd push!(gd, 4)                 # push! on pid 1

4-element Array{Int64,1}:
 1
 2
 3
 4

julia> @spawnat 2 @grd push!(gd, 5)      # push on pid 2
Future(2, 1, 20, nothing)

julia> @grd gd                           # it is everywhere up to date
5-element Array{Int64,1}:
 1
 2
 3
 4
 5

here the :guard actor makes a variable on pid 1 available everywhere.

I hope this helps to illustrate that there is practical value in an Actors ecosystem even long before we arrive at a fully developed system like Erlang/Elixir/OTP or Scala/Akka.

[datnamer]

That is extremely slick. Looking forward to giving it a go