Is it possible to shutdown the server after N minutes of idle an start a new one when a new runargs is called?
This would give a trade-off between compiletime and running tasks. Tasks would be automatically handled as batch tasks if they are provided in a given timeframe.
I write to announce a new version, V0.1.1, of my package DaemonMode. The code has been rewritten with the apportation of XeCycle, Páll Haraldsson and Gautier Soleilhac. Thank you, people!
The main changes are:
I hope you receive the new version, and your experience using it should be better.
Sorry, I did not read that message. Yes, it could be possible, I will use that idea when I implement the remove version. Sorry again for not replying you.
I write you to inform that I have updated a new version v0.1.2 that improve a lot the Exception stack. For instance, with the following file bad2.jl:
function fun2(a)
println(a+b)
end
function fun1()
fun2(4)
end
fun1()
Directly with julia:
$ julia bad2.jl
ERROR: LoadError: UndefVarError: b not defined
Stacktrace:
[1] fun2(::Int64) at /mnt/home/daniel/working/DaemonMode/test/bad2.jl:2
[2] fun1() at /mnt/home/daniel/working/DaemonMode/test/bad2.jl:6
[3] top-level scope at /mnt/home/daniel/working/DaemonMode/test/bad2.jl:9
[4] include(::Function, ::Module, ::String) at ./Base.jl:380
[5] include(::Module, ::String) at ./Base.jl:368
[6] exec_options(::Base.JLOptions) at ./client.jl:296
[7] _start() at ./client.jl:506
in expression starting at /mnt/home/daniel/working/DaemonMode/test/bad2.jl:9
with DaemonMode v0.1.1 it gave:
$ julia -e 'using DaemonMode; runargs()' bad2.jl
LoadError: UndefVarError: b not defined
Stacktrace:
[1] fun2(::Int64) at ./string:2
[2] fun1() at ./string:6
[3] top-level scope at string:9
[4] include_string(::Function, ::Module, ::String, ::String) at ./loading.jl:1088
[5] include_string at ./loading.jl:1096 [inlined] (repeats 2 times)
[6] #7 at /mnt/home/daniel/.julia/packages/DaemonMode/lrn5P/src/DaemonMode.jl:141 [inlined]
[7] (::DaemonMode.var"#3#5"{DaemonMode.var"#7#9"{String},Sockets.TCPSocket,Bool,Bool})() at /mnt/home/daniel/.julia/packages/DaemonMode/lrn5P/src/DaemonMode.jl:98
[8] redirect_stderr(::DaemonMode.var"#3#5"{DaemonMode.var"#7#9"{String},Sockets.TCPSocket,Bool,Bool}, ::Sockets.TCPSocket) at ./stream.jl:1150
[9] #2 at /mnt/home/daniel/.julia/packages/DaemonMode/lrn5P/src/DaemonMode.jl:89 [inlined]
[10] redirect_stdout(::DaemonMode.var"#2#4"{DaemonMode.var"#7#9"{String},Sockets.TCPSocket,Bool,Bool}, ::Sockets.TCPSocket) at ./stream.jl:1150
...
However, with DaemonMode v0.1.2, it gives (using colors like in Julia):
$ julia -e 'using DaemonMode; runargs()' bad2.jl
ERROR: LoadError: UndefVarError: b not defined
Stacktrace:
[1] fun2 at /mnt/home/daniel/working/DaemonMode/test/bad2.jl:2
[2] fun1 at /mnt/home/daniel/working/DaemonMode/test/bad2.jl:6
[3] top-level scope at /mnt/home/daniel/working/DaemonMode/test/bad2.jl:9
It is easier to read, in my opinion :-).
I write you to inform that version v0.1.3, with the following changes:
The script can use Logging. There are two situations:
The messages are written to a external file.
The messages are written to console.
Both situations are working nicely. For instance, for the file test_log1.jl:
using Logging, LoggingExtras
function msg()
@warn "warning 1\nanother line\nlast one"
@error "error 1"
@info "info 1"
@debug "debug 1"
end
msg()
running directly with julia:
$ julia test_log1.jl
┌ Warning: warning 1
│ another line
│ last one
└ @ Main ~/working/DaemonMode/test/test_log1.jl:4
┌ Error: error 1
└ @ Main ~/working/DaemonMode/test/test_log1.jl:5
[ Info: info 1
while in color:
running with client:
$ juliaclient test_log1.jl
Warning: warning 1
│ another line
│ last one
└ @ Main /mnt/home/daniel/working/DaemonMode/test/test_log1.jl: 4
┌ Error: error 1
└ @ Main /mnt/home/daniel/working/DaemonMode/test/test_log1.jl: 5
┌ Info: info 1
└ @ Main /mnt/home/daniel/working/DaemonMode/test/test_log1.jl: 6
┌ Debug: debug 1
└ @ Main /mnt/home/daniel/working/DaemonMode/test/test_log1.jl: 7
or in color:
runargs() returns
By example:
$ jclient hello.jl
Hello, World!
$ echo $?
0
$ jclient bad.jl
ERROR: LoadError: UndefVarError: b not defined
Stacktrace:
[1] fun2 at /mnt/home/daniel/working/DaemonMode/test/bad.jl:2
[2] fun1 at /mnt/home/daniel/working/DaemonMode/test/bad.jl:6
[3] top-level scope at /mnt/home/daniel/working/DaemonMode/test/bad.jl:9
$ echo $?
1
julia client only send the information by sockets, so it could be implemented in any compiled language. I have used Nim for that (I first tried Rust but it is was not easy to do that, in nim it was surprising simple), and it is available in (GitHub - dmolina/juliaclient_nim: Julia client binary using DaemonMode.jl package).
You can compile it, or downloading directly the binary version (only for Linux and 64bits) at Release v0.1 · dmolina/juliaclient_nim · GitHub.
For instance, with a script that load packages CSV and DataFrame:
$ time jclient_julia test.jl 7days.csv
...
real 0m1.389s
user 0m0.489s
sys 0m0.333s
While using jclient_julia it takes only (with juliaserver loaded):
$ time jclient_julia test.jl 7days.csv
...
real 0m0.443s
user 0m0.511s
sys 0m0.292s
The half of second is due to running the julia interpreter.
Using the binary version the time is greatly reduced:
$ time jclient test.jl 7days.csv
...
real 0m0.019s
user 0m0.004s
sys 0m0.007s
To summarise, using the binary client it is faster because there is not penalty due to the load of the Julia interpreter.
After a busy weekend, I write you to inform of a new version v0.1.5, with multi-task running of clients.
In previous versions, if you run a a program with take some time, and you try run another one before the first had finished, the second one did not start until the first one was finished.
In version v0.1.5, all programs are run as tasks in parallel, so the second one can be started before the first one is finished.
Because I have consider that the new behaviour is a lot better than previous one, I have set the async mode active by default. However, you can run the server function with the parameter async=false to have the previous behaviour.
# Async mode
$ julia -e 'using DaemonMode; serve(async=true)'
# Sync mode (previous behaviour)
$ julia -e 'using DaemonMode; serve(async=false)'
This have several advantages:
You can run any new client without waiting the previous ones.
If one process ask for close the Daemon, it will wait until all clients have finished.
Normal output (and standard error) are shown always to the corresponding program.
Disvantage:
@info is shown by the last one. Because @info is usually only used for debugging, I think it is not a big problem.The main problem in the development was the fact that redirect_stdout and redirect_stderr did not work right with tasks, because they change a global variable are not supported in a multi-tasking environment. Thus, the output (normal and error) was always sent to the last program to be run, and not to the program responsible of the output. For fixing that, I have defined print and stderr to redirect that info manually to the socket (using async code to be able to redirect the output in real-time). That approach cannot be replicated with macros, that is the reasons of the previous disvantages.
I hope you consider it very useful.
This is great, thanks for the wonderful work!
I’m curious about how does the server know which environment to be remembered for which client. Does the server essentially run all scripts from different submissions in one environment?
Thank you for your interest @jling.
Yes, all scripts are run in one environment for avoid loading the libraries more than once, but each one script is run in a different module (created dynamically), to avoid conflict of names.
For print the output and not confusing them between clients, I have also created for each module small functions to define local stdout and stderr.
Fantastic news!
Just a bit of bikeshedding, in julia, we normally distinguish between single thread / single process concurrency with multicore parallelism, and withing multicore parallelism we distinguish between multithreading and multiprocessing.
Looking at the code, it seems that what you’re during is single core concurrency. If you swapped @async for Threads.@spawn it would be parallel.
Thank you very much for your explanation. You are right, I thought the code used different threads but it just made them asynchronous.
I have made the suggested change in a new version v0.1.6, to actually be able to run the clients in parallel.
With the optional parameter async=true to server, the server run each client in a new thread.
$ julia -e 'using DaemonMode; serve(async=true)'
That command will allow to run different clients parallel, but it will use only one CPU.
If you want to use several threads, you can do:
$ julia -t auto -e 'using DaemonMode; serve(async=true)'
Auto allows DaemonMode to use all processors of the computer, but you can put -t 1, -t 2, …
With several threads (indicated with -t), you can run several clients in different CPUs, without increasing the time for each client. If there is only one CPU the processing time will be divided between the different clients.
I often use @info for other purposes (as there is @debug for debugging). The logging framework should be able to handle this, and include information about which process the log comes from, though I’m by no means an expert here.
I’d say that in a julia context at least, using the term async to refer to multithreading might be a bit confusing to some (It’d certainly confuse me). Could I suggest serve(threaded=true) or serve(spawn=true)? Then you can offer that in addition to serve(async=true) because there are workflows that benefit from just being asynchronous but not necessarily multithreaded, especially if people don’t want to spin up all their cores.
Yes, you are right, I wanted to say @show, not @info. @info as other options of logging, can be easily redirect to a file by the script. This is the reason I have already send a new version, because it is now perfectly usable.
In a near future I have consider to include information to send to the client (with LoggingExtras), and then discriminate (also to send to the stdout, or the stderr, depending of the criteria).
Yes, maybe it could be confusing. Actually I was not confused because I did not know the difference between them 
. I actually am not sure about maintaining both async and threaded: When server is run in only one thread (without the -t parameter, or using JULIA_NUM_THREAD), it is the same (but neither program can be run in parallel). I only consider that it could be useful to distinguish them, to allow have async and not threaded, in the case you are running a complicate program that use several threads, and you want all threads for yourself, but in that case maybe you do not want async at all. What do you think? Do you can think in another workflow?
@Mason Finally I have presented a new version V0.1.7 that have two options: async and threaded:
serve(async=true, threaded=false).Also, now you can use the function exit in the client without any problem (in previous versions, that implies that the Daemon closed also, not any more in that and following version).
In addition, several errors have been fixed.
I hope you consider useful. I expect to give a talk in JuliaCon talking a little about the package and its possibilities.
I write to inform that:
I am going to talk about the package and its advantages in the JuliaCon 2021 in the poster session Talk about DaemonMode.jl in JuliaCon 2021, do not miss it! 
There is a new version 0.1.9 of DaemonMode. The main changes are:
DaemonMode would execute the codes that are directly passed to the server, so each time the codes are updated, you would get the up-to-date results. However, sometimes you may also be developing some packages in the same time, and want they got reloaded when modified. You can use Revise together with DaemonMode for this purpose. You only need to add using Revise, before starting the DaemonMode server:
julia --startup-file=no -e 'using Revise; using DaemonMode; serve()'
It is a very useful improvement when you are developing a package and using scripts using it. Thanks a lot to @Shuenhoy for this great feature!
A part of that, in between there have been a version 0.1.8 that remove several disturbing messages.
More in details:
Hi @dmolina
What happened with your Talk?
I wish you are fine.
Hi @Dictino,
Sorry for the delay, I am currently on holiday (and watching the JuliaCon 2021 talks).
About my talk, it was available as a virtual poster (but there was a problem with the noise that takes time to detect it, fortunately, when I reported it, it was quickly fixed). Today it is available at Faster scripts in Julia with DaemonMode.jl | Daniel Molina | JuliaCon2021 - YouTube for watching it. In that talk I didn’t mention the Revise feature because when it was not ready when it was recorded.
Me too 
Thank you for the link, and have a nice holiday!
Thank you!
Is there an easy way to terminate the Daemon server? I can do it through the console quite manually…
In my project it would be useful to activate and terminate the server within a Julia file.
Cheers