What I am missing is comparison with what shell programming using bash and the command line applications coming with a Linux system for arbitrary precision arithmetics like bc is capable of. Which of the in the bash shell available array operations and via bc available arithmetic functionalities are duplicated in Julia and how? What does Julia provide which can’t be achieved with comparable speed using shell scripting and it numerous command line “friends” like for example bc ?
Shell scripting languages are usually interpreted and therefore for many tasks orders of magnitude slower than compiled languages like Julia. They also lack a proper type system, exceptions etc.
BASH and friends can be useful for programs with less than 20 lines of length. They become a nightmare to read, maintain, test and debug when they become longer.
I use bash scripts to launch Julia and to create Julia system images.
Sorry … but I can’t see any advantage of Julia in the range of numbers which are mostly used in practice, where other cases are extremely rare and need anyway other tools as Julia will fail to cover such cases:
~ $ time python -c "print( 3^100000000000 % 102 )"
91
real 0m0.030s
user 0m0.026s
sys 0m0.004s
~ $ time julia -e "println( big(3)^100000000000 % 102 )"
ERROR: OutOfMemoryError()
Stacktrace:
[1] pow_ui!(x::BigInt, a::BigInt, b::UInt64)
@ Base.GMP.MPZ ./gmp.jl:179
[2] pow_ui
@ ./gmp.jl:180 [inlined]
[3] ^
@ ./gmp.jl:628 [inlined]
[4] bigint_pow(x::BigInt, y::Int64)
@ Base.GMP ./gmp.jl:649
[5] ^
@ ./gmp.jl:654 [inlined]
[6] literal_pow(f::typeof(^), x::BigInt, ::Val{100000000000})
@ Base ./intfuncs.jl:351
[7] top-level scope
@ none:1
real 0m0.582s
user 0m0.530s
sys 0m0.174s
Here the timings I consider as more representative as the edge case mentioned:
~ $ time julia -e "println( big(3)^100 % 102 )"
81
real 0m0.242s
user 0m0.206s
sys 0m0.088s
~ $ time echo "3^100 % 102" | bc
81
real 0m0.003s
user 0m0.003s
sys 0m0.000s
3^100 is already a very big number. In practice you won’t go beyond this, but even if see the timings with exponent of 10000 :
~ $ time echo "3^10000 % 102" | bc
69
real 0m0.003s
user 0m0.004s
sys 0m0.000s
~ $ time julia -e "println( big(3)^10000 % 102 )"
69
real 0m0.244s
user 0m0.222s
sys 0m0.073s
I mean if what you need is exactly covered by bc, then there might not be an advantage to using Julia? there’s no need to use the same tool for everything.
I sometimes use my phone calculator — unfortunately Julia becomes much slower when I have to take the train home to grab my laptop first.
Well, normally you are not doing one calculation, but many in a loop, or you work with vectors and arrays.
For just doing one or two scalar calculations bc is fine, you don’t need Julia for that. We could give a better answer if you would explain your use case.
The question is how does Julia compare to shell scripting and the available executable “friends”.
Do I understand it right that Julia startup time is too large so it does not make sense to use it just for some simple calculations even with arbitrary precision exceeding int64 or float128 for single numbers?
Does it make sense to use it for an array of 1080x1920x3 uint8 values where all of the RGB values need to be set to for example the mean value of them?
Julia has some startup time which other things don’t:
$ time julia --startup=no -O0 -e 'println("hello world")'
hello world
real 0m0,149s
$ time echo hello world
hello world
real 0m0,000s
$ time awk 'END {print "hello world"}' < /dev/null
hello world
real 0m0,004s
$ time perl -e 'print "hello world\n"'
hello world
real 0m0,004s
So in many cases the startup latency of julia exceeds the run time of various other linux utilities. For quick scripting, other languages and utilities are often better suited.
^ is bitwise exclusive-or in Python, not exponentiation. Try it with ** instead, but be ready to Ctrl-C.
You also increased the exponent by 5 orders of magnitude compared to Oscar_Smith’s example. log2(3^100000000000)=1.585*10^11 bits is 19.8 gigabytes, the software is trying to protect you from crashing the computer with a few integers.
Your timing is including loading the REPL, compiling first calls, and println, those are not representative steps of any number crunching. bash has nice functionality, and if it’s fast enough for you, you shouldn’t bother loading an interactive session in any other language, especially Julia’s relatively hefty runtime. If you need a longer-running number crunching program, bash is not the best tool.
I am trying to profit from the experience of others which went the way of details in many scenarios down and can tell something generally valid about such comparisons. It is hard to believe that one “language” can be orders of magnitude faster on the same hardware.
What I would like to find out is if shell scripting can cover what other languages provide, so that there is no need for the other languages and proficiency in shell scripting and knowledge about appropriate executable files available on the system would be enough to cover most of the general purpose use cases.
If you go enough into detail … you will loose the chance to arrive at some general valid guidelines … With requesting enough details you can’t compare at all … because at the very detail things are not comparable …
It can provide a small subset, good enough in some use cases as long as the script stays very short. Long shell scripts are not maintainable. And for many tasks like optimization problems, differential equations, machine learning etc shell scripts are no option at all.
This is a universally acknowledged fact. The performance difference between languages like C/C++/Fortran and languages like R/MATLAB/Python in particular is a premise for Julia’s initial development. Many development perks sacrifice performance, which is why people use the latter languages at all. Many languages, including Julia, find their own middle ground and styles.
What is the reason for this? What in detail makes the difference? At the bottom its anyway one CPU-instruction after another … and the file system. So assembly language should be enough for everything in hands of a skilled programmer which orchestrates executable files in a way they provide the result. I suppose that 95% of each programming language is nothing else than duplicating what is already there, but it still requires re-learning these 95% to become able to use it. Orchestrating specialized executable files reading/writing to files (on a RAM-disk) will be then the better way of arriving at any result than usage of some programming language. Shell scripting is orchestrating executable files … in best way possible … so it should be enough to cover all needs, don’t it?
You’re trying to argue that programming languages are redundant because you can’t see their purposes. Instead of playing out logical fallacies, I would suggest that you learn these languages before making erroneous comparisons and challenges. I can’t possibly speak for all programming languages, but I can broadly respond to a few of your statements.
bash is single-pass interpreted. There is no opportunity for compiler optimizations, which will be present in compiled languages with any interest in performance. It’s even more restricted than CPython, which at least compiles to bytecode before interpretation.
Assembly is not portable, which is why a high level language C is the most accessible programming language. It’s by far the most popular choice for implementing other programming languages.
Yes it all ends up at machine code eventually, but that does not imply it’s just as easy or even feasible to do the same thing in every language. For example, some languages need a runtime to handle things like memory in addition to the code you might find in minimal executables, and the runtime overhead can easily be too much to run on small embedded systems. MicroPython was developed because the CPython implementation was too big for microcontrollers. Julia’s runtime is even bigger in part due to the compiler.
To eliminate the previous factors, let’s limit the conversation to languages compiled to minimal executables. Languages will vary a lot in style and rules, and this affects how people think about programming and the ease of features. Functional programmers enjoy refactorability and reflection benefits from referential transparency. Rust made rules to provide memory safety without the need for a garbage collector. There is no one best language because different people value different tradeoffs.