[ANN] Rembus.jl

Package Announcements
21

It looks I must have maybe misunderstood this part. For my system Redpanda is the major building block allowing exchange of messages between various applications. Even Fluvio is supporting such a compatibility. Without Redpanda I will have problems with integration further down the road as Kafka is a kind of a standard. As I see your advice currently, with simple tasks I would probably be able to use just Julia packages at their basic form but even then the amount of code to maintain by one person will be increasing almost exponentially. I will be also a kind of limited with vertical and horizontal scaling. Thus, as for now, I would prefer to offload background infrastructure to specialized packages or containers and to focus on non-standard analytical work in Julia. I will be observing further developments of your package. I wish you best.

22

Hi, amazing package.

I have some questions:

  1. What is benefit of using Visor vs kubernetes or docker compose with restart policy? Can it help to avoid compilation on process startup?

  2. What is size limit of single payload if any? I see in example you use DataFrame that is 15MB serialized. For instance in NATS limit is 1MB.

  3. What are message delivery guarantess (at most once, at least once, exactly once)?

23

Thanks, my answers/observations:

What is benefit of using Visor vs kubernetes or docker compose with restart policy? Can it help to avoid compilation on process startup?

I never thought of Visor as an alternative to container/pods schedulers such as docker compose
or kubernetes because they supervise different types of objects.

But, yes, this is a good point, because with Visor you could potentially move from an architecture consisting of multiple containers to a single container with multiple Julia tasks managed by Visor.

There could probably be some compile-time benefits, but in my opinion the main point
could be improved performance at runtime by eliminating inter-container communications.

What is size limit of single payload if any? I see in example you use DataFrame that is 15MB serialized. For instance in NATS limit is 1MB.

A part the system memory availability, there is not limit on message size.

Julia uses HTTP.jl and the WebSocket constructor uses as default maxframesize=typemax(Int), the maximum value permitted by WS specification.

What are message delivery guarantess (at most once, at least once, exactly once)?

The rpc case uses a unique message id to match the request with the response, so I would say that the case is “exactly once”.

For the pub/sub case the mechanism is fire-forget, so the delivery guarantee is “at most once”.

There is, at the moment an undocumented and not much tested feature, that does message acknowledgement. You will have obviously some performance degradation but you will gain “at least once” delivery guarantee.

2 Likes