[ANN] YAActL 0.2, Yet Another Actor Library (built in Julia)

Package Announcements
, ,
21

Say, it enables building fault tolerant systems:

  • an abnormal termination of an actor can be detected,
  • if an actor A is supervised by B (you can have a supervision hierarchy), B can restart A (maybe on another node)

This has been implemented in Erlang with success years ago.

I think that all modern implementations of the actor model must build on that and enable fault tolerant programming. This will make it highly interesting for distributed systems.

Chapter 5 “Programming Fault-tolerant Systems” of the Armstrong thesis describes that in detail.

22

I suppose that Actors and Actor tasks do not need to match 1-to-1 with Julia’s Tasks. We may have a message processing scheduler which assigns messages to native tasks or threads who’s lifetimes are not tied to Actors or Actor Tasks. In fact many seem to think this is preferable and use some work stealing dequeue algorithm to schedule (locally atleast) the processing of actor messages on each CPU (with each thread being assigned to a CPU and having its own work queue). This decouples Actors from OS threads or Julia’s task scheduler. I haven’t tried this yet myself, but it seems like the right way to go although it has its challenges. At any rate it does mean that we don’t have to worry too much about being walled in by something like structured concurrency.

“Soft realtime”, looks like a bit of nightmare to implement in Julia though, unless you use OS processes which can be killed at any point by the kernel instead of threads. Even with the proposals for cancellation points, this won’t deal with infinite loops and such. It is possible to make threads cancellable at any point, but it is probably never going to be safe in Julia (or anything for that matter). Still creating a Julia process for each work queue is not really such a big issue, it just makes performance worse :slight_smile:.

1 Like
23

As someone who programs in Elixir for work, I don’t think the structured concurrency would restrict the actors, and it’s basically similar to how things are done in practice to the point that I had to think about what’s the behavior within the Erlang VM. For example if I want a process to run a task that could outlast itself (or if I just want to run async), I’d usually set a Task.Supervisor as a child of the Application Supervisor (a process that lives as long as the application runs, so anything tied to it will definitely never outlive it), and from the current process I can attach a new actor/task to that supervisor, which you can optionally keep linked (and if that process has an exception it will be raised in the current process as well, otherwise it will only announce to it’s supervisor).

In fact I think it’s even expected that if a parent process goes down, it will take all his children with itself and always avoid any kind of process leakage (especially if the parent is a supervisor). The important thing here is that you must still be able to set the automatic kill behavior, for example each children will get a Kill Exception with a configurable timeout to perform the necessary work to safely shut down in case their parent is killed.

When I have some free time I’ll try to play with the library and think about it since I have an interest in an actor model for Julia since you can’t program in Elixir and not be amazed by it, and because it maps really well to a class of applications that I think Julia should really target (like all the libraries within the Akka framework).

3 Likes
24

YES, and

If actors don’t run as Julia Tasks but build on another concurrency primitive, you are right. For example in Elixir tasks are built on actors (called “processes” in Erlang/Elixir) and not vice versa:

# Elixir example ...
worker = Task.async(Fib, :of, [20])
result = Task.await(worker)
IO.puts "The result is #{result}"

Tasks are implemented as OTP servers, which means we can add them to our application’s supervision tree. (Programming Elixir, p. 294)

It makes absolutely sense to introduce something similar in a Julian actor library as it solves the problem better than the structured concurrency approach. Just we cannot call it Task since that is in Core.

2 Likes
25

Tasks in elixir were just an example (and are indeed different from Julia’s tasks/@spawn which would be closer to Elixir’s primitive ‘spawn’), but it’s also done for example for database connection pool (Repo), GenServers and even libraries (added with Mix). You always add them in a supervision tree path in which you know the parents won’t die until the usefulness of it’s children ends.

I didn’t try to use the library yet (nor tried to understand the machinery), but I’ve read the documentation. My experience is very biased (I only used actors in Elixir and Scala/Akka) so I took a little time to understand (only when I went to the API listing) as it takes a different approach. Actors are these fluid things that you pass behavior after they are already running, with a currying-like strategy in terms of pre-defining arguments (which is a very functional approach). That is quite different from those two languages that give a template and you complete it to have a full behavior (Elixir uses macros mixins, and Akka you subtype some AbstractBehavior and override methods like onMessage and onSignal), both very OOP-like.

Of course I have to get experience with this approach, but in general I like rigid actors like Elixir and Akka since if I want another behavior for the actor I can just kill it and create a new one, and I always know the behavior of the actor since they are basically objects in message passing terms (like Ruby or Smalltalk objects). I think I’ll see if I can implement this kind of behavior as a thin abstraction layer over the library, for example building a GenServer, something based on interfaces like:

mutable struct Stack{T}
  stack::Vector{T} # state
 end

struct Push{T} <: Message
  elem::T
end

struct Pop <: Message end

Stack(initial::Vector{T}) where T  = Stack{T}(initial) #init

# execute something without returning anything (might need another argument with the Link, not sure now)
handle_cast(st::Stack{T}, msg::Push{T}) where T = push!(st.stack, msg.elem)

#return the result
handle_call(st::Stack{T}, msg::Pop) where T = pop!(st.stack)

#Which is similar but not quite how the library does, while the rest is more similar but tries to abstract the machinery a little more:

mystack = Actor(Stack([1,2,3]) #mystack would wrap the Link so you don't need to explicitly define it
send!(mystack, Push(4))
response = call!(mystack, Pop) # (or the send! + receive!)

Which is more direct than https://pbayer.github.io/YAActL.jl/stable/examples/stack/
I think making it OOP-like, fill in the blank (interface) and wrapping the internals unless you need to deal with it makes it immediately obvious for the usual programmer with an OOP background, even if less flexible (which Elixir appeals as well even being functional). But of course, I really have to go deep to understand the motivations and trade-offs in the library, so sorry if I’m being too premature with this comment.

1 Like
26

Thank you! Yes, I know :slightly_smiling_face:. The stack and also the factorial examples are from the earliest days of YAActL and demonstrate the basic machinery and consistency with Agha’s original examples.

With the (GenServer-like) API-functions call! and cast! it is easier and more convenient to implement them and the documentation should mention it and give examples.

But I think it makes sense to keep the lower (fluid) Message API open for system programming stuff or extensions.

I think this is possible and supported and I’m looking forward to see what you come up with. :grinning:

1 Like
27

Well after taking a closer look, I have to say that I am a big fan of YAActL’s interface. It might be more alien to most people, but if it can be wrapped in a faimiliar skin then you can have your cake and eat it. I am thinking about some things I’d like to try with the library, but please bear with me because I’m always over contended xD

BTW perhaps it would be a good idea to create a JuliaActors GitHub group and move our projects there @pbayer @oschulz?

FYI, the following is how I implemented the Agha’s stack in my test suite and actuallly this is more like the OOP way, but perhaps still somewhat alien:

# Special Actor which represents the Actor system
struct StackPlay end

# An Actor's state
mutable struct Stack{T}
    # an item on the stack
    content::Union{T, Nothing}
    # next item on the stack, note that Id's are not like Links in YActl AFAICT
    link::Union{Id{Stack{T}}, Nothing}
end

Stack{T}() where T = Stack{T}(nothing, nothing)

# message handler, the Scene is a context object passed to all handlers, avoids task local variables.
# I was playing with symbols here, but you could separate this into multiple handlers and dispatch on type instead
function hear(s::Scene{Stack{T}}, msg::Tuple{Symbol, Union{Id, T}}) where T
    (type, m) = msg

    if type === :push!
        # We have various accessor functions like my which always take the scene object
        if !isnothing(my(s).content)
            my(s).link = invite!(s, Stack(my(s).content, nothing))
        end

        my(s).content = m
    elseif type === :pop! && !isnothing(my(s).content)
        say(s, m, my(s).content)
        my(s).content = nothing

        isnothing(my(s).link) || forward!(s, my(s).link)
    else
        error("Can't handle $type")
    end
end

# Genesis! is the first message sent to the actor system
function hear(s::Scene{StackPlay}, ::Genesis!)
    stack = invite!(s, Stack{Symbol}())

    say(s, stack, (:push!, :a))
    @test ask(s, stack, (:pop!, me(s)), Symbol) == :a

    say(s, stack, (:push!, :b))
    @test ask(s, stack, (:pop!, me(s)), Symbol) == :b


    stack = invite!(s, Stack{Int}())

    for i in 1:5
        say(s, stack, (:push!, i))
    end

    @test ask(s, stack, (:pop!, me(s)), Int) == 5

    say(s, stack, (:push!, 6))
    @test ask(s, stack, (:pop!, me(s)), Int) == 6
    @test ask(s, stack, (:pop!, me(s)), Int) == 4

    leave!(s)
end

# Actor based test set!
@testset LuvvyTestSet "Actors Stack" begin
    play!(StackPlay())
end

Unlike YAActl in my library you must start the actor system (instead of simply creating the first actor with a link?) which is represented by a special actor called the ‘play’. This was probably a mistake although it helps with implementing actor addresses in the way which I did that. Each actor has one or more addresses which are resolved to a mailbox when a message is sent. I’m not sure how that compares to links in YAActl, but it (somewhat) obviates the need for an actor registry as you can simply give some actor’s a static address (address can be reused in case of failure).

TBH, my interface results in hear being way overloaded, this is not good for multiple dispatch performance and just looks bad IMO. YAActl’s way is better although I can imagine it easily being abused and leading to confusing dynamical behavior, but I also think contracts and protocols can help with that. However my library also allows the user to override the default message loop which is:

function listen!(s::Scene)
    @debug "$s listening"

    for msg in inbox(s)
        @debug "$s recv" msg

        hear(s, msg)
    end
end

You can just provide a more specific method for listen! like:

listen!(s::Scene{Stack{T}}) where T = for msg in inbox(s)
    do_something(s, msg)
end

This is very useful if an Actor also needs to listen for some I/O events, for example from my experimental web server:

function Actors.listen!(s::Scene{Server})
    dir = my(s).dir
    my(s).srv = srv = HTTPServer(nothing, listen(8888), "localhost", "8888")

    say(s, dir, Listening!())
    # start an asynchronous julia Task which plays nice with the Actor we are launching it from
    async(s) do s
        try
            for sk in srv.server
                say(s, dir, Connected!(sk, srv.hostname, srv.hostport))
            end
        finally
            leave!(s)
            close(srv)
        end
    end

    for msg in inbox(s)
        hear(s, msg)
    end
end

I guess in YAActl this might be done with init!, but it might also be necessary to allow the user to override the message loop in _act. Either way I/O operations may not play nice with scheduling message execution. Perhaps I could try writing some sort of web service in YAActl and see what problems arise…

1 Like
28

Thank you very much! YAActL has some innovation in basic actor design, e.g.

  • actors are designed as function servers,
  • the loop is realized by the actor and has not to be implemented by behavior functions,
  • actors are finite state machines,
  • YAActL exploits Julia’s multiple dispatch.

But credits for most of the API should go to the designers of Erlang/OTP who basically developed it long time ago (Elixir adopted it too). I don’t want to reinvent the wheel and therefore try to reimplement its ideas in YAActL’s API. It is surprisingly simple to implement on the basis of a state machine.

definitely! Good idea to put our things together. I hope this would attract more interest, ideas, participation, collaboration. And we would have the ideas and discussion in one place. :smile:

BTW, today I registered YAActL 0.2.1 with an actor registry. It makes a lot of sense since it works transparently across distributed processes.

I plan to continue with the development of the API, see

With v0.3 it should become clear that we can have in Julia a fully fledged actor library.

This would be great! I learned from @fborda above that the examples should reflect more practical applications than agreement with theory. This must definitely be the case with v0.3.

29

@pbayer, if you like I’ll give you full control of Actors.jl, so you can move it. I can’t do anything with actors, currently (too many other projects), and I have a feeling you’re thought long and deep about this. So if you’d like to take Actors.jl (which never really went very far) and turn it inside out and into a modern Julia actors framework, please do!

30

I created the GitHub JuliaActors group and moved YAActL.jl there. You and @richiejp, @fborda, @tisztamo are invited to join and to move or fork your relevant repositories there.

Thank you for that. We may use this name in the future if we have something more mature. using Actors suggests something on a par with using Threads or Distributed and integrated to work well with them. This is not the case for now but hopefully not too far away if more people collaborate.

To become …

at least two more levels must yet be developed:

  1. error handling with supervision, monitors, grouping of actors … and
  2. interfaces to other actor systems, languages and network services.

This will take some effort and time, but not too long since Julia provides a mature basis to do it. If more people join in, work can be done concurrently. :wink:

2 Likes
31

Thanks for the inviation! (I am also working on an actor system called Circo, yet to be announced)

I agree that this is not the time to name a specific implementation as “the” Julia actor library. On the other hand, it would be very nice to have an interface defined that allows actor-level compatibility between implementations. Not sure if that interface should be called Actors.jl, AbstractActors.jl, ActorInterface.jl (see ArrayInterface.jl ) or something else.

It is not easy to define that interface, because the actor model has variations with different semantics. E.g. as I understand, in YAActL.jl actors are a bit like processes, e.g. YAActL.receive! is a blocking operation, which is convenient, but implementing it in this way seems to be very tricky when the underlying implementation does not map actors and Julia tasks one-to-one.

The paper 43 Years of Actors: A Taxonomy of Actor Models and Their Key Properties seems like a good start to better understand the history and scale of actor models. It defines four classes: Classic Actors, Active Objects, Asynchronously Communicating Processes (like Erlang, not the CSP model of concurrency) and Communicating Event-Loops. I think that the other three are more or less extensions of the classic model, minus the behaviour change.

Defining a minimalistic interface for the classic model seems like a good start. What do you think?

3 Likes
32

Thank you very much for the paper. It gives a good overview. It defines:

The Classic Actor Model formalises the Actor Model in terms of three primitives: create, send and become. The sequential subset of actor systems that implement this model is typically functional. Changes to the state of an actor are aggregated in a single become statement. Actors have a flexible interface that can similarly be changed by switching the behaviour of that actor.

well not quite:

  1. YAActL actors are classic actors in the above sense.
  2. they don’t need receive!. But this is a convenient function for taking a message from an actor.

I very much agree for several reasons:

  1. The classical model is a particularly good fit for Julia since a behavior is a function f(c) of the received communication c. This sounds like a match made in heaven with Julia’s multiple dispatch.
  2. It allows to follow several roads in parallel in developing a JuliaActors ecosystem, while being able to interoperate.
  3. I think it is fairly easy to do. (I know it since I started like that)

Currently in YAActL 0.2.1 an actor link is defined as

struct Link{C}
    chn::C
    pid::Int
    type::Symbol
end

We could move that e.g. to Actors.jl and define (following roughly Circo):

struct Actor{M}
    mod::M     # module
    func
    args
    kwargs
end
Actor(func, args...; kwargs) = Actor(Actor, func, args, kwargs)

Actors.jl would have functions (based on Julia primitives) like

send!(lk::Link{Channel}, msg) ....
send!(lk::Link{RemoteChannel}, msg) ....
spawn(a::Actor{Actor}, ....) ....

Circo, YAActL and others then do using Actors and reimplement Actors.send! and Actors.spawn with their concrete Link and Actor types. A Circo actor can be started with spawn(Actor(Circo, f, args)), a YAActL actor with spawn(Actor(YAActL, f, args)). Both return concrete link types and can communicate via send!. If we implement protocols they can even understand each other. :slightly_smiling_face:

Should we try that as our first joint project over at JuliaActors?

33

Yes, it would be great to see your proposal for a general interface!

I am not sure I fully understand your outlined idea, but I am surprised you call this task easy, because I have the feeling that there is a hard-to-solve semantic difference between YAActL on the one side versus richiejp/Actors.jl and Circo on the other (oschulz/Actors.jl seems not using the classic model). The interface of Circo and Actors.jl are very similar, but they differ in the number of arguments used in message handlers, which itself seems hard to consolidate (Actors.jl uses composite types for things that are separated in Circo).

But I may be wrong, so If you could provide a proposal with an implementation in YAActL, I will try to bend Circo to also implement it, and report concrete problems I find.

Additionally, I think that the interface should be more abstract than in your outline. For example, Circo avoids using channels for performance reasons, but what it calls an Addr is conceptually very close to Link. Maybe the name Ref, as used by Akka would be good for an abstract type here (if needed). Regarding your Actor struct above I have the feeling that a factory function without a memory layout would be less restricting.

I think in general that an interface should mostly operate with functions and traits, maybe abstract types. I am not a big fan of macros, but it seems that they can help here hiding implementation details. Maybe the common interface should be a DSL.

Looking forward to see your proposal!

ps.: It seems that we should move this conversation to somewhere. What are the possibilities? I would prefer to stay on Discourse if possible without generating too much noise that can disturb others. Maybe a new “domain” for actors?

1 Like
34

I am referring to your proposal to implement a minimalistic interface to the classic actor model. Basically it must

  • allow to spawn (create) an actor, which returns a Link (or Ref) to it,
  • hide completely how the actor machinery works, but
  • allow to send it something and
  • allow to set its behavior function e.g. with become.

Then each implementation of the classical actor model should be able to implement those four elements.

We must not even have a common type for Actor. Maybe it would make things more clear. But for a minimal interface it is not strictly necessary. Yes, I will make a proposal on JuliaActors.

yes, this thread is becoming long and maybe difficult to follow. But I am happy that it brought some actor people together. Can we use a new topic and tags?

2 Likes
35

I’m not sure yet if a minimalist interface will end up sufficient in practice, and not a complex enough interface that it could very well end up dictating the implementation (especially when performance considerations enter into play). For example, you’d probably need a signaling interface, so you can inquire the state of the actor (if it’s alive), you’d need a way to signal a shutdown command, you’d need a way to link one actor to another so an exception can trigger a reaction on another actor (and not only within the supervision tree which is insufficient), all of these are necessary so it will play nice in an actor tree/graph (which might lead to: do we also need a supervision tree interface?). If we used Elixir reference, it would be similar to the methods defined in Process

But once an interface is defined (even if it ends up converging on a a standard implementation), JuliaActors group can very well become something like JuliaArrays, with libraries that build on the Actor interface to provide all kind of specialized actors for each purpose, like DynamicSupervisors, Tasks, GenServers, GenStage (consumer-producer abstraction).

That said I might not be as interested in creating my own implementation of Actors as in the infra-structure around it, Actors for me are more of a means to an end. Stuff like Akka HTTP, Akka Streams, Erlang Term Storage/Mnesia, Phoenix PubSub are all tools that I really think would really complement Julia’s current strengths. I’ve been toying with implementing a Kafka consumer group similar to KafkaEx, which while only supporting very basic features of Kafka (below 1.0), it can provide a lot of reliability through the let-it-crash strategy. And I’m really interested in how a heavily ccall/threadcall/IO-based actor would behave (for example if you’d be forced to lock the primary Julia thread for it to work).

1 Like
36

I don’t know if this is entirely relevant, but have y’all seen GitHub - JuliaParallel/Dagger.jl: A framework for out-of-core and parallel execution? Uses a DAG to keep track of )execute tasks and dependencies across threads, processes and GPUS etc

Also see the Julia folds ecosystem.

May be opportunity for integration, or maybe the models are too different

1 Like
37

I don’t know about interop, but I suppose the scheduling and dequeue code in Circo would be a nice thing to share somehow (which looks very nice btw!). Also a lockless message box would be nice and this is not a trivial thing to implement safely (I don’t think Circo is lockless?). However I think the way to do this is to copy and paste it into YAActl, make it work, then think about how to deduplicate.

This will clash with a type in the Julia base library IIRC.

38

I opened another topic:

I will pull in some of the good stuff of YAActL into Actors.jl in providing the first two steps described in the roadmap but not all. Actors.jl should be more minimalist than YAActL, e.g.

  • it does not need two APIs,
  • Dispatch modes are not necessary, I will go with “full dispatch”

But foremost it must provide a sleek interface. I can use Circo’s onmessage for plugging in a message protocol and an API similar to YAActL’s. This then can be used by other actor libraries.

After that I will decide how to proceed with YAActL.

1 Like
39

Thanks! The scheduler has its separated package partly for better reusability, and we already chatted with @pbayer about trying to make Circo a “backend” of YAActL. It seems to me that the hard part of this is that Circo only implements the classic model without blocking (at least for now) while YAActL heavily uses blocking operations on channels. I fear that we will get no performance gain if we simply implement a channel-like interface over Circo, and use it in YAActL, because the slowness of standard Julia async is mostly coming from task switches, and that is where we have to fall back for blocking. But this is just my current state of thought, my understanding of both YAActL and the inner workings of async is far from complete.

On locks: Yes, a lockless mailbox would be nice, I thought a bit on that but never tried to implement. Circo does not have per-actor mailboxes, it uses “hand delivery” instead: per-thread message queues and actor-to-thread pinning to minimize locking - no locking is needed when the source and target actor live on the same thread. Actor placement is more or less automatic and runtime-optimized for “actor locality” to minimize inter-thread communication.

40

FYI in case anyone is interested there’s a Julia multithreading meeting every other week: Julia #multithreading BoF

The next one is today at 1:30 est. Might be a good place to swap ideas, get insight into roadmap for planning etc with @jameson who works on parallelism and compiler stuff.

2 Likes