Do not rely on people reading what you wrote in a particular way you want, you have the responsibility to write respectfully. Someone offered you those general practices in this case, and there is an etiquette to how you criticize someone’s practices. More than 1 person have patiently asked you to adjust your tone, I suggest you take heed instead of thinking about their reading comprehension. I have said enough, so I’ll drop this subject now.
Me too! Thank you all for your suggestions!
I fully agree with this. Julia is protecting from some things, but introduces other pitfalls which are not present in other languages. Much of it has to do with the dynamic nature of julia. The chaos around OffsetArrays some years ago springs to mind (introducing arrays which start at 0 or any other integer, not everyone was prepared for that, to say the least).
When first starting with julia one gets the feeling that it babysits you, protecting you from, like … out of bounds errors. Lets you mix e.g. different numbers like integers and floats, everything just works. Until you one day hit pure weirdness, and understand nothing until you figure out the logic behind it.
The chaos around
OffsetArrayssome years ago springs to mind (introducing arrays which start
at 0 or any other integer, not everyone was prepared for that, to say the least).
Who in the whole world does not like such a feature? It allows painless integration with Fortran/Matlab that start from 1 by default, C,C++ that starts from 0, and ADA has this since its very beginning. It is really nice because you do not have to offset yourself and introduce errors due to wrong offsetting.
Surely, the feature is great. Now, the problem was another one. A lot of methods, even in standard libraries and in the documentation, looked like:
function f(A::AbstractVector{T}) where {T<:Real}
s = zero(T)
for idx in 1:length(A)
s += A[idx]
end
return s
end
An AbstractVector was used to accomodate views and ranges and such things, so you can do f(1:10), f(0.1:0.1:1) and f(rand(10)).
Those methods failed with an OffsetVector starting at 0. (Because A[length(A)] is out of bounds). Even worse, they were often marked with @inbounds, leading to a segfault or undefined behaviour rather than an out of bounds error. An OffsetVector is of course a subtype of AbstractVector. Likewise a view (or SubArray). That’s when things like eachindex and axes became good programming practice.
I have something similar, which is quite common in vector PDEs. There the solution vector may consist of many different variables (velocity, pressure, density, etc) y = [u_x, u_y, u_z, p, ρ]. I created a class called ShiftedArray or OffsetArray that takes the shift at the constructor and overloaded the indexing operator [ j ] so that internally v[j] returns v[offset + j]. Internally in Debug mode you can check for violation of upper/lower bounds. Then the code you wrote runs without issues and without eachindex.
The problem affecting the OffsetArrays package (or everybody else) was that it couldn’t really be tested with all possible uses. It subtyped AbstractArray which is used everywhere, not only in standard libraries, but in many packages which the author of OffsetArrays didn’t know about, or they weren’t written yet. Or other package authors couldn’t test their package with OffsetArrays, because it wasn’t there or they didn’t know about it.
It did respect @inbounds so that performance can be high. Now there is an informal interface to AbstractArray which subtypes should respect, but there are still no formal interfaces in julia which are enforced for subtypes. The same goes for iterators, there is an informal set of methods which should be defined, but it is not enforced. (like size, length, and so on).
Normally this is not a problem for small projects where one has control over things, but it is a potential problem for public packages.
Julia is a big language, with a lot for a beginner to learn. One of its core features is modules, these are namespaces used to organize code. They’re essential, and the rest of the tooling in the language is built on the premise that users will use them.
Modules are the wisdom of the forefathers, which Julia was thoughtful enough to include. Implicitly, you thought of A.jl and B.jl as though they were modules. But that isn’t how it works, you need to make modules. The language doesn’t treat files as namespaces, which I’m sure you’ll find preferable once you start to use them.
No one likes losing three days of their life while learning a new language. But this has caused you to see problems with the language which don’t actually exist. I encourage you to keep learning, and let the hostility become humility. I think you’ll see why we like Julia, in the end.
1 Like
Hi Sam,
thank you for your inputs. To be honest something inside me was telling me that modules is the way to go, but as I explained to @mkitti I was under the impression that you create a module once you are 95% there (nearly perfected your implementation).
To be even more honest, once in the past I tried to modify an installed package which would not install properly due to differences in library versions (HSL, MKL, etc) and I was really annoyed when I realized that my edits in the installed files somewhere in ~/.julia/ were not executed. I realized that once a package is installed then it gets read-only. I did not really know what to do and did not have time to look deeper into it.
Months later I decided to delve deeper and familiarize with Julia. It started with scripting just to make sure that my code gets executed and my edits are executed and not get installed as modules/packages somewhere in ~/.julia/blablabla/acdferh4890asd/. As soon as I would feel that my script does 95% of what it is supposed to do, I had in mind to convert it to something like a package just to familiarize with packages.
I was not expecting however, that 2 scripts with the same functions definitions are allowed to be included in the same master script one function overwrites the other and you get no warnings or errors for doing so.
And by the way, I am not hostile. Some other people mentioned that Julia allows functions with the same definitions, and this was really troubling for me, since even if that is the case I would expect an error or warning. They told me that there is such an option to get warnings but it is hidden and you have to activate it. I prefer safety over anything else. So from now on I will follow your advice and @mkitti advices and work with modules. But this information should be explained at the first tutorials with a big WARNING in case include scripts are including functions with identical definitions.
1 Like
While there are young people involved, there are some seasoned experts involved from the very beginning. I attended a dinner in celebration of Alan Edelman’s 60th birthday several months ago. Saman Amarasinghe and Gerry Sussman supervised Jeff Bezanson’s dissertation describing Julia. Professors Tim Holy and Steven G. Johnson are active members of the community.
In many ways, Julia is really a continuation of the Lisp family of languages originating from the 1960s.
A critical aspect to recognize is how Julia is fundamentally different from languages like C++ and Fortran and purposely so. It is a dynamic language that is also compiled to native code. Many people understandly conflate compilation with static analysis, but these are distinct phases of development.
A very important open question is when should static analysis be performed. As a dynamic language, Julia is designed to allow code to be added during runtime including the potential for redefinition. Unlike static languages there is not a clear point where we can declare code as complete for and thus ready for analysis.
As a dynamic language Julia is often conflated with a scripting language. While the language does allow for top-level scripting, it is not very well optimized for this case. Most of the optimization involves procedural or functional programing, and thus you are encouraged to aggressively put code into functions.
Here the concern is that you seem to be quite critical about function redefinition whereas this is in some ways a core feature around which Julia was designed. In the packaging and module context where code is formalized and compilation is cached, this becomes less desirable. Moreover, I would argue that precompilation might be a good place to perform some static analysis. The other development phase is unit and integration testing, which are both tightly integrated around the concept of modules and packages.
If you care about the formalisms you seek, I think the answer for you may to proceed as quickly to the package stage as possible. Above I’ve demonstrating the quickest shortcut to package implementation possible: the modification of LOAD_PATH.
6 Likes
Dear @mkitti,
thank you for the long and extensive clarifications. Lets rephrase. I do not care about “dynamic” or “static”, or whatsoever language features if they sacrifice safety, and if they cause me to waste days to understand why my code is not working as expected. This is very fundamental and I think it should be much higher in the priority list of language features than other properties (dynamic, Lisp, scripting). This is because the language is used by a human being to produce a result and the outcome should be predictable because the programmer is expecting to see what his program does. This is not a formalism. It is fundamental. If the dynamic character of the language sacrifices that, then some priorities are sorted according to wrong criteria.
Back to function redefinition, the problem would have been solved if you get a soft ERROR (in red so it is visible) explaining that you are redefining functions in your include files. Is this done in purpose? Problem solved, safety is promoted, dynamic character of Julia preserved. Another approach would be for the user to run Julia with special flags if he desires code to be added on the fly as you explained.
My hope, following the development of Julia, as a user of some packages, was that one day this new and promising Language will replace everything that we have been using in the past (C,C++,Fortran) at least for engineers who need to solve really hard computationally intensive, problems and it would become the new standard in academia and industry because it can incorporate all the best practices from existing languages, provide interfaces with legacy codes, automate GPU, OpenMP parallelization in a seamless way, prevent you from writing unsafe code etc.
I am pretty sure several engineers and scientists out there share the same dream about Julia. A language written for engineers, mathematicians, and scientists.
- Scripting is probably the worst way to use Julia for what you want. Build packages, not scripts.
If you had loaded the code as package, then you do get a soft error without having to use special flags.
- Use packages with static analysis tools. Use Aqua.jl and JET.jl for example to analyze your code.
As other have mentioned and I have pointed out above, there are several distinct ways to load and reload packages. Revise.jl will automate some reloading of packages for you. Also note that if you add packages to your environment via the Pkg.add method then you are using an immutable copy in .julia/packages. However, if you use the Pkg.develop method of adding packages to your environment, then changes you make on disk will be reflected in the code you load.
- Julia may be the wrong language for you. If safety first is your mantra, you probably want a static language like Rust that can perform very thorough checks enabled by the language design. Julia’s design makes this really difficult if not impossible to do.
Given the things you care about, you probably should care if you are using a static or dynamic language.
Static languages are easier to check. The analyzer can assume that the program is complete and can tell you that something is wrong. It’s like turning in an exam to the teacher. No more changes are allowed. The exam can be graded.
Dynamic languages can be harder to check because something can always be added or, yes, things could be redefined. This is like an interactive tutorial session. You can change your answer or elaborate further. There’s no defined point in time where you can be evaluated to produce a grade.
Julia does blur the lines between dynamic and static but it is fundamentally a dynamic language by design. People use Julia in the way you are using it to purposely redefine functions as part of an interactive REPL session. They will intentionally use include in the REPL to overwrite functions, sometimes by repeatedly reloading the same file. If you want safety, I highly suggest that you change your workflow away from this.
Modules and packages are where Julia start to take on static characteristics. At some point you end a module and in doing so you are starting to say that code contained within is a complete unit of code. Moreover packages are now compilable units of code that can be cached. Loading code via using or import is a clearer sign that the code being loaded should not be redefined or contain redundancies and that is why warnings and errors are produced when you try to do that.
To be honest, Julia is really not a good replacement for these static languages. I would argue that Julia is a better fit for some problems than Fortran, but I do not think of Julia as a wholesale replacement for it. Julia is not dependent on C and C++ extensions to achieve acceptable performance and is in some ways safer, but it is still easier to do some forms of static analysis in those languages than it is to do in Julia.
If you most care about safety above everything else, you probably want a static language, and Julia is not a static language.
1 Like
Dear @mkitti,
thank you for the extensive and clear explanations. Would you please explain me why I get this info only from you and not anyone else? Is this your understanding of Julia? Is it not common knowledge?
Anyway, I will follow your advice and start working on modules/packages whatever they are called. Thanks for the info about Aqua.jl and JET.jl. But if they are doing such a nice job, why Aqua and JET are not incorporated at the core of the language?
Would you please give me a link about the Pkg.develop method? This is sounds interesting. You mean that instead of ] add package_name I could do ] develop package_name and I get the package downloaded somewhere and I can edit and experiment etc? But should I remove the original package added in my environment with ] add package_name first or not? Currently if I want to change something I use “grep” to find the function call inside the ~/.julia directory and it seems I change a read-only file to enforce the functionality I want. The other way would be a “pull request” as some people explained which I tried and I was so confused with the whole thing being explained across several pages in github each page linked inside the other that I dropped it because what it is named “pull” request seems to be a “commit” request, because it assumes that you have already pull and forked the code and somewhere you have a branch locally. The logic seems so reversed that my brain shuts down, I cannot follow.
Something so fundamental instead of being explained in a single document with an example in a tutorial style is spread in the Github universe that you need the Avengers to help you get it back together.
I am so sorry to hear that. I had such a sweet hope in my heart when Julia started, I was under the impression the goal was to create the language that was missing for engineers and mathematicians, something like MATLAB but without its pitfalls. A great opportunity. But it seems it has ended up like a computer science project where computer scientists are trying to incorporate all fancy features without caring about the end result.
In any case with the Aqua and JET packages you described above, plus some other flags maybe it is still possible to convince the computer science people behind Julia, to allow developers to start Julia in such a mode that could be used as a Fortran replacement. Because where else can you find so many packages and so much functionality already available? All you need is a “package-protected-mode” or “protected-environment” to do your coding. Otherwise people will end up using Julia for testing ideas and algorithms and then they will have to implement the same ideas or algorithms in different languages from scratch. Yet, I know companies that use Julia for the final product, but wouldn’t it be better to protect these companies from Julia features that may destroy their package environments? We do not want Julia to end up like python with so many different packages and dependencies that if one does not compile then the whole installation is stuck and useless.
The 3rd post in thread contains information about Pkg.develop:
It’s well understood that Julia is not a static language, but you can search this forum to see that you are not the first person to ask for further static analysis of code. There’s also plenty of discussion around interfaces.
Follow the link below.
https://pkgdocs.julialang.org/v1/managing-packages/#developing
You do not need need to remove a package from your environment first. If you already added the package and then tried to develop a package with the same UUID, then the developed package will replace the added package.
You can provide a local path. If you just provide a package name, then the code will be git cloned into ~/.julia/dev by default.
You can find references to this in the help mode of the Pkg REPL where it also provides a link to https://pkgdocs.julialang.org/ .
(@v1.10) pkg> ?
Welcome to the Pkg REPL-mode. To return to the julia> prompt, either press
backspace when the input line is empty or press Ctrl+C.
Full documentation available at https://pkgdocs.julialang.org/
Synopsis
pkg> cmd [opts] [args]
Multiple commands can be given on the same line by interleaving a ; between
the commands. Some commands have an alias, indicated below.
Commands
activate: set the primary environment the package manager manipulates
add: add packages to project
build: run the build script for packages
compat: edit compat entries in the current Project and re-resolve
develop, dev: clone the full package repo locally for development
You can also see the help page for Pkg.develop.
julia> using Pkg
help?> Pkg.develop
Pkg.develop(pkg::Union{String, Vector{String}}; io::IO=stderr, preserve=PRESERVE_TIERED, installed=false)
Pkg.develop(pkgs::Union{PackageSpec, Vector{PackageSpec}}; io::IO=stderr, preserve=PRESERVE_TIERED, installed=false)
Make a package available for development by tracking it by path. If pkg is given with only a name or by a URL, the package will be
downloaded to the location specified by the environment variable JULIA_PKG_DEVDIR, with joinpath(DEPOT_PATH[1],"dev") being the
default.
If pkg is given as a local path, the package at that path will be tracked.
The preserve strategies offered by Pkg.add are also available via the preserve kwarg. See Pkg.add for more information.
Examples
≡≡≡≡≡≡≡≡
# By name
Pkg.develop("Example")
# By url
Pkg.develop(url="https://github.com/JuliaLang/Compat.jl")
# By path
Pkg.develop(path="MyJuliaPackages/Package.jl")
See also PackageSpec, Pkg.add.
There’s a link to this from the Julia manual:
https://docs.julialang.org/en/v1/stdlib/Pkg/
As I’ve tried to explain, there are some fundamental tradeoffs in language design. If you want to keep some of the dynamic style of MATLAB, then doing static analysis will be difficult. If you do not care as much about the dynamic part, you could look to Chapel or Mojo, which are actually static languages at heart.
I’m not exactly sure what you are trying to say here. If you want to use Fortran packages, we have many of them prepared to use in Julia here:
I’m also getting confused about what you are trying to say here. I’ve already explained that loading code via using and import and putting code into packages offers some additional protections.
Also note that you cannot easily redefine functions exported from a package once you have used it unless you explicitly say you are doing so.
julia> write("HelloWorld.jl", """
module HelloWorld
export helloworld
function helloworld()
println("Hello World")
# Put "script" here
end
__init__() = helloworld()
end
""")
167
# I don't recommend this method of pushing into LOAD_PATH for serious work.
# It's better to use Pkg.develop to add packages to the environment
julia> push!(LOAD_PATH, pwd());
julia> using HelloWorld
[ Info: Precompiling HelloWorld [top-level]
Hello World
julia> helloworld()
Hello World
julia> helloworld() = 5
ERROR: invalid method definition in Main: function HelloWorld.helloworld must be explicitly imported to be extended
If you run Pkg.test("YourPackage"), your package will also be loaded with check-bounds=yes.
I never use include from the REPL. The scripting workflow that you originally presented is incompatible with the kind of checking that you want.
This mostly sound like dependency management issues. You can have them in Julia as well if you try to load too many packages into the same enviroment. Fortunately, Julia’s default package manager includes a comprehensive environment system to address these kind of issues. We do not recommend add all your packages to the default environment. I usually keep my default enviornment to a small number of utility packages. I then make use of stacked environments.
It depends how you define the “core” of the language I suppose. We generally prefer to have as much code in packages as much as possible since it is easier to develop and iterate on them as indepenent packages.
2 Likes
Thank you for all the explanations and your patience. You are amazing.
I meant that in the same way you need Aqua and JET to allow some static analysis, perhaps it could be possible some day that Julia installation comes with Aqua and/or JET and/or similar functionality installed by default. Once you install Julia you get by default what JET has to offer.
An alternative would be a Julia operating mode (this could be interesting by the way even as a Computer Science project) chosen by the user. For instance at the moment you explained me you can have Pkg add and Package develop modes. Perhaps it could be possible to have a Julia static and a Julia dynamic mode. Bing’s CoPilot offers for example 3 options the last one being more Precise. Why not Julia offering these static or dynamic options to the user triggered similarly by typing
go dynamic
go static
depending on the developer specific needs?
By the way, thank you for everything I highly appreciate your patience and lengthy clarifications. I wish you all the best for whatever you are doing. You deserve it.
This is because you use a highly unusual workflow with julia. This was not clear in your initial post. Most developers follow the advice in the docs about workflows, or use a fully interactive style e.g. when profiling specific functions with all the necessary minor modifications. And, of course, spend some time to understand how julia actually works before suspecting some inexperienced “julia chief designers” have done something stupid.
Your initial posts also revealed profound confusion about how compilation in julia works. The initial part of the thread centered about what the circumstances of your problem was. When it transpired that you had accidentally defined the same method of a function twice, a perfectly normal thing to do in julia in certain interactive workflows (like it is in other dynamic languages), you insisted that this must by default be considered an error.
So, after a lot of beating about the bush, @mkitti has read the docs out loud, to explain how julia works. I suspect most others have gotten tired of your besserwisser attitude.
4 Likes
Give it time 
I don’t know that “Julia for embedded” will ever be a major use case, or even possible, but there’s a lot of work (as you know @mkitti) going in to making it tractable to compile shared-library binaries which don’t carry along the entire runtime. Enough work along those lines and Julia could end up a fine language for systems work, which would be more like a dialect than a second language.
There’s a creative tension between Julia’s dynamic nature and its rather precise type system, which hews closely to the machine in the style of other systems languages. Gradual typing, pervasive precompilation to machine code, these things blur the traditional distinction between a static and dynamic language, Julia’s approach really is unique here. Common Lisp may be the closest cousin along this dimension, but even there, Julia’s principled type system stands apart.
This would go against the current sentiment, which is that too many features live in the standard libraries, and they should be moved into their own packages if and when it’s possible. JET.jl living in the Julia repository wouldn’t change a line of its source code, after all. This would really only make sense if the core language needed some of its features, which it doesn’t, and even in the event, only the minimum code necessary should be moved. It’s easy to add code to a standard library, and difficult to move it once it’s established.
This is a fair critique. Julia is a large language solving difficult problems, and (IMHO) relies too heavily on lore spread out between packages and living in Discourse threads. The only solution to it is for “someone” (and we’re all someone in open source software) to take the time to consolidate and maintain guides and tutorials to make this process easier. The maintenance is really what gets you, there are a few projects of the “awesome Julia” variety but the language is a moving target and none of them seem particularly up to date.
My advice to you is: don’t fight the language. “Why doesn’t Julia do things the way that X does” isn’t usually the best question, especially phrased as “Julia sucks because it doesn’t do Language X thing in the Language X way”.
“How do I do this thing from Language X in Julia” is the better question, and as I hope you appreciate, people around here are willing to spend considerable time sharing their hard-won knowledge on the subject.
It remains up to you to take advantage of this resource.
I think a closer characterization would be that too many standard libraries live in Base
it is not that features need to move out of stdlibs, but that stdlibs need to move out of Base
1 Like
Fair, I wasn’t trying to draw such a sharp distinction as all that. More lumping Base in as the most standard of standard libraries, if you will. My sense is that if it were easy, which it isn’t, much of what’s in the standard library would live outside it. There are more programs which need a JSON parser than a TOML parser, I suspect. Do the REPL and Pkg need to be in the standard library? And so on.
The point was more about what isn’t in the main repo, and a salutary resistance to moving things from outside to in.
The main consideration for standard libraries was whether they were in the system image or not. Until Julia 1.9, the system image was the only ahead-of-time compiled native code.
Now we have package images with native code, so there are less performance issues to worry about ejecting them from the system image now.
2 Likes