Julia script using zmq works run from cli but NOT from cronjob

General Usage
, ,
1

Hi there

not sure if this is a linux ( I’m running linux mint 20.3) and have set up 3 cronjobs.

the first starts a ZMQ PULL server that gets messages and println’s the messages to stdout which I pipe out to a text file thus

45 5 * * 1,2,3,4,5 julia ZMQ_PULL_SERVER_BASIC_5555.jl >  "$(date +"\%Y_\%m_\%d_\%I_\%M_\%p")_daily_zmq.txt"

the PULL server will be terminated by an “END” message.

then I start a script that opens the ZMQ PULL port PUSHES messages to the PULL server and closes the port but does NOT send an “END” message so the ZMQ is still running.

I then start another julia script that opens the ZMQ PULL port and PUSHES messages to the PULL port and when finished sends “END” which closes the ZMQ PULL server down.

the issue is that the TXT file exists but has nothing in it when the scripts are run from the cronjobs

BUT
IF I run the scripts by cut an pasting the cron entries the TXT file has been created and the data written correctly.

does julia handle the piping differently in the two situations?

thank you

2

imho:

  • try to use full path - everywhere (juliafile, output ) !
    cronjob has a different user / workspace !

like
julia /my/project/ZMQ_PULL_SERVER_BASIC_5555.jl > /my/project/output.txt

or write a wrapper bash job - to cd to the project directory
01 00 * * * cd /company/juliaproject003 && ./daily_zmq_job.sh

EDIT:
tips for debugging: How to Test Cron Job?(Cron Job Testing) - DEV Community
0/1 * * * * /usr/bin/julia /home/my_username/hello.jl > /home/my_username/log.txt

3

thank you for the reply. I “think” I am using all the full paths. I trimmed the layout in the OP to simplify things.

Does this look right to you? It’s the FULL crontab layout that’s causing the issue.

45 5 * * 1,2,3,4,5 julia /home/dave/tontine_2022/test/4_15_22_ZMQ_PULL_SERVER_BASIC_5555.jl > /home/dave/tontine_2022/data/zmq/"$(date +"\%Y_\%m_\%d_\%I_\%M_\%p")_daily_zmq.txt"

00 6 * * 1,2,3,4,5 julia /home/dave/tontine_2022/test/5_3_22_live_STK_HV20_10_5_price+percent_CLOSE.jl 

# 00 7 * * 1,2,3,4,5 julia /home/dave/tontine_2022/test/4_12_22_live_STK_IV_percentile_close.jl

I’m pretty sure it’s something to do with ZMQ not liking the PULL server being opened by one script, written to, the script closing the port and another script opening it and writing to it, finally sending “END” to tell the PULL server to shut down. For some reason the PULL server is not closing the TXT file correctly.
thank you for the debugging link very handy but I’d already worked through most of the steps before posting this question. The cronjobs run on time, the text file gets created, the scripts are NOT running ( I use htop to check ) at least they didn’t this morning when it ran.

4

OK; So your code is running with root user ; from the / (root) dir …

sudo su
cd / 
julia /home/dave/tontine_2022/test/5_3_22_live_STK_HV20_10_5_price+percent_CLOSE.jl

but from the crontab this code is not working … :thinking:

5

Hi there
sorry I don’t understand what you mean when you post. I didn’t mean to be rude by saying that the link to the cron debugging was not helpful, it was. I’m still working through it.

sudo su
cd / 
julia /home/dave/tontine_2022/test/5_3_22_live_STK_HV20_10_5_price+percent_CLOSE.jl

when I run from the CLI I’m logged in as user dave, if that helps. dave is an administrator account type
Am I doing something wrong? Linux is NOT my strong suit. Thank you for bearing with me on this.

6

ok so you are running “cronjob” with the dave user

  • and you can see your task list with : crontab -u dave -l

:thinking:

ideas:
1.) maybe you have to “redirect standard error” to file - and check the output …
like in this example ( 2>&1 in the end of the command line )

2.) You have to test the “environment” variables

* * * * * env > /tmp/env.output

any path missing ? ( source )

7

ok…

check this post

and add a project path :
julia --project=/path/to/your/project/folder /path/to/your/script.jl > /path/to/your/output.txt

1 Like
8

first of all thank you for doing something I should have! I clearly didn’t search for this on discourse and that’s very poor of me, sorry about that.

I’ll make the modifications as below

45 5 * * 1,2,3,4,5 julia --project=/home/dave/tontine_2022/test/4_15_22_ZMQ_PULL_SERVER_BASIC_5555.jl > /home/dave/tontine_2022/data/zmq/"$(date +"\%Y_\%m_\%d_\%I_\%M_\%p")_daily_zmq.txt"

00 6 * * 1,2,3,4,5 julia --project=/home/dave/tontine_2022/test/5_3_22_live_STK_HV20_10_5_price+percent_CLOSE.jl 

00 7 * * 1,2,3,4,5 julia  --project=/home/dave/tontine_2022/test/4_12_22_live_STK_IV_percentile_close.jl

and see what happens. AGAIN so sorry for not searching before bothering you.

1 Like
9

julia --project=/home/dave/tontine_2022/test/4_12_22_live_STK_IV_percentile_close.jl

imho:

based on my past experiences with crontab:

I am suggesting

  • adding a cd \<projdir> before calling julia;
    • for me it is important if I use some relative path; inside my code!
    • so it is “safe” method …
  • && ( chaning operator )
  • julia command with 2 parameter !
    • --project=/<projdir>
    • /<projdir>/juliaprog.jl ( julia program with full path ! )

so your full command look like this:

  • 00 7 * * 1,2,3,4,5 cd /home/dave/projdir && julia --project=/home/dave/projdir /home/dave/projdir/juliaprog.jl > /home/dave/projdir/output.txt
2 Likes
10

hi there
WOW thanks again. I have taken your suggestion and made the modifications so the crontab now looks like this, as the scripts are in the /home/dave/tontine_2022/test folder.

45 5 * * 1,2,3,4,5 cd /home/dave/tontine_2022/test && julia --project=/home/dave/tontine_2022/test /home/dave/tontine_2022/test/4_15_22_ZMQ_PULL_SERVER_BASIC_5555.jl > /home/dave/tontine_2022/data/zmq/"$(date +"\%Y_\%m_\%d_\%I_\%M_\%p")_daily_zmq.txt"

00 6 * * 1,2,3,4,5 cd /home/dave/tontine_2022/test && julia --project=/home/dave/tontine_2022/test /home/dave/tontine_2022/test/5_3_22_live_STK_HV20_10_5_price+percent_CLOSE.jl

00 7 * * 1,2,3,4,5 cd /home/dave/tontine_2022/test &&  julia  --project=/home/dave/tontine_2022/test /home/dave/tontine_2022/test/4_12_22_live_STK_IV_percentile_close.jl

fingers crossed for tomorrow :slight_smile: thanks again for all your GREAT help.

1 Like
11

hi there @ImreSamu so it didn’t work. The text file was created but nothing put into it. I’m going to move on from this and approach the goal from a different route. I thank you so much for all your efforts and they have NOT been wasted. I have learnt a lot from your replies and it’s given me a new view on cron in relation to julia. I can’t mark this as solved but I CAN mark the link with the specific julia environment definition in cron as the most probable solution. Thank you again for all your help

1 Like