Can pluto handle real time data from python code

General Usage
,
1

I am working on my first julia program and using an existing python script I wrote to give me a data stream. To develop I am using Pluto and PyCall and trying to implement a asynch callback that returns a stream of data. So the calling cell is posting an asynchronous request to a broker api which starts to send back a stream of stock price change EVENTS which triggers a series of python print statements. So each time a price changes the callback gets triggered.

In python it’s working properly and I get a stream of data that looks like this

640898070.0 ^  V  ^  49462172  ^  217.87
1640898070.0 ^  VLO  ^  13497  ^  74.28
1640898070.0 ^  VWO  ^  27684033  ^  49.59
1640898070.0 ^  VZ  ^  4901  ^  52.25
1640898070.0 ^  WBA  ^  178634687  ^  51.99
1640898070.0 ^  WDAY  ^  115324104  ^  276.5
1640898075.0 ^  ESH2  ^  461318816  ^  4773.25

I don’t know if I can do this in Pluto?

I have been working my way ( with a LOT of help from discourse) down the python code using the journey to learn.

begin
py"""
def onBarUpdate(bars, hasNewBar):
    print( datetime.timestamp(bars[-1].time),"^ ", bars.contract.localSymbol," ^ ",bars.contract.conId," ^ ",bars[-1].close )

"""
onBarUpdate_py = py"onBarUpdate"
end

where onbarupdate is an event that emits values to connected listeners.

calling cell which posts a request for 5 second bars for the related stock.

stock_bars.updateEvent += onBarUpdate_py

Right now the PyCall is throwing an error that has caused me to ask the bigger question, does Pluto handle this sort of event.

PyError ($(Expr(:escape, :(ccall(#= /home/dave/.julia/packages/PyCall/L0fLP/src/pyoperators.jl:12 =# @pysym(:PyNumber_Add), PyPtr, (PyPtr, PyPtr), a, b))))) <class 'TypeError'>

TypeError("unsupported operand type(s) for +: 'Event' and 'function'")

    pyerr_check@exception.jl:62[inlined]
    pyerr_check@exception.jl:66[inlined]
    _handle_error(::String)@exception.jl:83
    macro expansion@exception.jl:97[inlined]
    +@pyoperators.jl:11[inlined]
    top-level scope@Local: 1[inlined]

I’m going to hit the PyCall docs related to the TypeError but I’m wondering if Pluto can’t handle these types of events. If it can would someone point me at a worked example of a Pluto cell ( or series of cells) that can handle asynchronous events. I have tried a few google searches for clues like this one

Real-time updating of plot data #30
thank you

2

I do not find adding a function to an event relevant. Do you mean a queue of events? It helps if you provide some working python code to clarify your intention.

1 Like
3

You make a good point. I have changed the original question to clarify my question.

The code works well in python and PyCall does an EXCELLENT job of exposing python functionality BUT my question is does Pluto.

In my python code I import a wonderful set of tools called ib_insync which has an event loop, adding additional multiprocessing or other external asynchronous methods can be a challenge thus causing my question.

In my case a series of events ( changes in a stock price) triggers callback which prints the price to the screen as in the example output. To post the working python code would ask you, and anyone else, to go down the rabbit hole as to how the brokerage interface works. I feel that’s not playing with a straight bat so I didn’t do it.

Also, I didn’t want to post the python code as I only put in the PyCall error to show what caused me to asked the bigger question about Pluto.

I thought Pluto would be the ideal IDE for handling real time data as it could be hooked up to IOT devices in labs to process the data feeds. What has caused me to ask the question is that I don’t seem to be finding many examples of that approach and I would have expected them to be prevalent.

thanks for helping me to clarify my question and happy new year.

1 Like
4

pluto can handle this type of workload fine, but you may want to look at PlutoHooks.jl since it’s such a asynchronous data handling workflow:
https://juliapluto.github.io/PlutoHooks.jl/src/notebook.html

that looks like a pycall error to me fyi. I’m guessing it would work if you used the lower level pycall API.

1 Like
5

Hi there
Great to hear that. thanks for your observation. I knew it was a PyCall error. The clue was in this line

PyError ...

just kidding :slight_smile: working my way through pyoperators.jl right now. Seems that I need to get to

# .+= etcetera map to in-place Python operations
for (op,py) in ((:+,:PyNumber_InPlaceAdd), (:-,:PyNumber_InPlaceSubtract), (:*,:PyNumber_InPlaceMultiply),
                (:/,:PyNumber_InPlaceTrueDivide), (:%,:PyNumber_InPlaceRemainder),
                (:&,:PyNumber_InPlaceAnd), (:|,:PyNumber_InPlaceOr),
                (:<<,:PyNumber_InPlaceLshift), (:>>,:PyNumber_InPlaceRshift), (:⊻,:PyNumber_InPlaceXor))
    qpy = QuoteNode(py)
    @eval function Base.broadcast!(::typeof($op), a::PyObject, a′::PyObject, b)
        a ≛ a′ || throw(MethodError(broadcast!, ($op, a, a', b)))
        PyObject(@pycheckn ccall((@pysym $qpy), PyPtr, (PyPtr, PyPtr), a,PyObject(b)))
    end
end

have a great New year’s

1 Like
6

I LOVE the look of PlutoHooks.jl but it doesn’t seem to be released yet. I think it’s EXACTLY what I need. I was getting a little concerned because of this thread Non-blocking asyncio calls with PyCall which didn’t even add the extra fun of the brokerage python asynch loop and Pluto.

When I had a brain I used to program in a fault tolerant environment and sometimes the different asynch event loops got a little hairy, that’s why I asked the question. I’m putting the PyCall approach on hold and headed down the apache route. I am REALLY looking forward to PlutoHooks.jl and thank you for pointing it out to me. I have marked your answer as the solution. Happy and safe 2022 to you and yours

7

I went back and read the PyCall documentation ( properly this time) and saw that @stevengj had added

so I cranked up the REPL ( not Pluto waiting for Plutohooks.jl) the code below worked out of the box. Really helped me out. Thanks @stevengj great work.

@pyinclude("test.py")