Integrating Julia with Rust?

General Usage
21

I’ve tried some of the alternatives and I didn’t really see any improvement over pip+venv. Without venv, you’re into having a system-wide environment, which I assume we all agree is not a good idea? (eg: You are likely to get conflicts between different projects.)

I have used miniconda/conda/anaconda quite a lot too. I use it on Windows because it’s a convenient way to get a Python install. However, I don’t like it as much as it tends to fail to resolve dependencies more often. Also it works a bit different from pip. I don’t recall exactly why it is different, so I won’t risk providing incorrect information here but essentially when I was looking into it I concluded it was less general purpose. I think I’m correct to say pip has a wider range of packages than conda, and that the two don’t work well together. (If at all?)

There are other things, poetry comes to mind, but iirc that’s basically a wrapper around pip with some support for uploading your packages to the main pip repo/server. Maybe I misremember this too, and it’s something else rather than poetry.

When I looked into the different options my conclusion was that pip was the most simple and straightforward option to use and it was sufficient in what it does to be useful. It’s also widely used, which helps, because then everyone uses a standard tooling.

To answer your question here, yes it may be that I just don’t like the UX.

eg: What could be more simple than

  • cargo test
  • cargo build --release
  • cargo run
  • or cargo run --bin NAME

It works anywhere within your project repository.

By contrast, to run tests in Julia

user@linux$ julia runtest.jl

is the closest alternative. Not bad, but you have to be in the test directory. (Where runtest.jl lives.)

I’m still trying to figure out the most convenient way to run a “binary application”. You have to be somewhere in src, probably. Again, maybe I’m wrong here, I’m still new to it.

Maybe I can say something more intelligent by saying with cargo, if you want to install a package, you just run cargo install PACKAGE.

On the other hand, what should I do with Julia?

  • One option is to run the REPL, using Pkg and then Pkg.add("PACKAGE"). Personally, I don’t like it. Even with pip, one can do
user@linux$ ./.venv/bin/activate
user@linux$ (.venv) pip3 install PACKAGE

I agree, there are problems with this. How do you deploy with Docker? Actually, there are good solutions. You set your entrypoint to

/project_name/.venv/bin/python3 -m /project_name/path/to/module

or, because you are in Docker, you install packages “systemwide” and don’t bother with venv. I prefer to use venv still, because it is consistent.

To go back to Julia, another option is:

user@linux$ julia
julia> ]
(@1.11) pkg> add PACKAGE

Here’s the thing. I don’t know how you productionize this.

To go back to the first option, is it a good idea to write

using Pkg
Pkg.add("PACKAGE")

at the top of a Julia script which is the main entrypoint for an “application”? I’m not sure about this. You are mixing code with environment. No other language works this way. Is it better? Worse? Why? I don’t know. (Yet)

Well - why? It seems a lot of people here don’t like pip much. I’m not sure I understand why. In the Python world, I would suggest it is almost universally liked.

Can you tell me a problem it has that you have difficulty with? It’s possible I can suggest a slightly different way of using it which is better for the developer experience. (Something I’m very keen on being good btw.)

22

You never strictly have to restart Julia. Yes, you can’t change structs, so you need to work around that, I do a lot of my_struct2, my_struct_3 etc.

Revise.jl handles everything else (and there may be a package for structs, I do not recall).

I’m thinking since structs are a know problems, and hard to work around at full speed, could some development mode for the REPL only work? Indirection solves a lot of problems, and I think it could work around this one too, at least give the illusion you can change anything at runtime (though with speed-penalty).

I would also like a developer mode at least or non-default allowing rearranging structs, closing the (potential) speed gap with Rust, improving on C++'s defaults like Rust does. Rust and Mojo also allow eager deallocation (Julia has workarounds in Bumper.jl package), at least Mojo improves on C++ that way, Julia could also by default.

[Not sure why, a Linux problem, not Julia’s? And even if unsolvable:]

That seems similar to the (known) struct redefinition problem. Something I overlooked, as not inherent to Julia development, only for interop. I suppose you can rename to my_lib2.so, my_lib3.so etc. Maybe some package could automate, only to be used in development?

See --project=. option.

I used to run with no options cumulative massive global environment. It has huge downsides, and make Pkg look much worse than it is, seeing frequent downgrades and other problems etc. But it’s misusing Julia, all avoidable.

23

This is not true. It’s possible, but not a recommended way to run a test suite.

Pkg.test() runs the test suite for the currently active package.
And Pkg.test("foo"; julia_args=["--inline"]) does what it looks like does.

I recommend at a minimum reading the doc string for Pkg.test and test in the pkg repl. Then you could make more constructive comments.

5 Likes
24

we must live in very different Python worlds :sweat_smile:.

  • the resolver is VERY slow sometimes. even for medium size projects it might take several minutes
  • It can try to download like a billion versions of the same package trying to find one that’s compatible
  • it does said compatibility checks a lot of the time by just… fully downloading and checking rather than a smarter approach based on compat bounds
  • the resolver isn’t even correct a lot of the time. I frequently find that for the same set of core dependencies the order in which I add / install them can be the difference between a clean install and a “no compatible versions found”
  • the requirements.txt files do not disambiguate between direct dependencies and transitive dependencies (in Julia these are the Project.toml and Manifest.toml files) so pinning versions of direct dependencies while allowing transitive dependencies to freely upgrade (or add more or remove) is quite cumbersome
  • the setup scripts can run… arbitrary code. and if the author of the package did something not super robust and you start getting like PATH issues or its trying to use the wrong environment within the setup script, god help you
4 Likes
25

To clarify my point here - I was not suggesting run Julia without using environments. (Using global environments.)

I suppose the difference to Rust is Cargo knows when it is inside a project, because it sees a Project.toml somewhere in the directory stack.

With Julia it’s explicit. You have to say --project=.. I don’t see why it couldn’t be changed to be automatic, like Rust Cargo.

This is that isn’t an improvement, it’s actually much worse for two reasons:

  • you cannot productionize this because it requires a human to manually load the REPL, and run using Pkg then Pkg.test(). That means you cannot integrate this as part of CICD. The only way you could make this part of automated testing would be to write a short script which calls using Pkg; Pkg.test(), in which case you might as well just run julia runtest.jl from the command line
  • second point is minor. In the context of human interaction with tests: It requires you to type more characters. It’s a lot less convenient than just running julia runtests.jl from the command line

Regardless if you are a human or a machine, it’s worse.

26

Nobody should be manually modifying the PATH, If somebody has released a package with some PATH hackery in it then they’re just making use of bad engineering practices - not sure what else to say.

I know that lots of posts on stack overflow (etc) will contain information about doing this, but it’s the wrong solution. It seems to be a mistake on behalf of the language designers that this was allowed. I’ve never seen a legitimate use for it.

I agree with you this is a potential issue, but again I can’t honestly say I have seen it come up often, if at all.

In regards to your other points, yes I don’t disagree - these are all possibilities. But again, I haven’t personally seen them actually occur in real systems very often, if at all.

When things do go wrong, usually re-creating the requirements.txt file is the quickest solution.

Again, I agree it’s a pretty trash solution and it shouldn’t be the way things are. So yes, generally I agree with you but I think the impact of these things is often overstated, and relatively easily mitigated.

1 Like
27

It’s automatic like Cargo if you

export JULIA_PROJECT="@."

However, unfortunately this is not the default behaviour. I believe security reasons have been cited as the main reason why.

Calling julia -e 'using Pkg; Pkg.test()' works too, but I agree it’s more awkward than cargo test. Arguably this is rather mitigated by being able to “sit in” the Julia REPL while developing, but I can see how this doesn’t help the new-user experience.

2 Likes
28

This is in context of my dlclose comment, in which you always have to restart if you want your Rust changes to take affect.

Maybe this would be possible. But I think we’d also have to make some changes where the code gets loaded as well, which sounds like it would be pretty prone to error. Plus I don’t really love the idea of having a bunch of these loaded and not getting closed.

Our flight code is written in Rust and our physics simulation is written in Julia. There are a lot of things we can (and do) unit test on the Rust side, but that misses the important parts of how physics interacts with the system, especially when doing controls work.

29
4 Likes
30

A common way to do this is julia -e 'using Pkg; Pkg.test()'. I agree with you that it’s not quite as nice as Cargo.

There is a significant advantage to the REPL workflow though: you don’t have to recompile. With Revise, if you run tests in a REPL, modify your code, and rerun tests, it’ll be much, much faster – including significantly faster than Rust.

1 Like
31

What are people’s subjective experience for calling Julia from Rust (or the other way around, or even both direction in same project)? I like to hear from people that have actually used it? I know it’s easy to call C, also since C is an easy language, the package jlrs is certainly large, I think because of complexity in Rust. It seems e.g. it adds a Julia string type for interop:

No it’s not. Already very fast to or from Julia, for e.g. Rust and Python. Some limitations apply currently e.g. for MATLAB, then no zero-copy.

No, already not strictly needed. E.g. if you call Julia from Rust, you could be calling a precompiled package. You just want to make sure all the code, or for the relevant types is precompiled. It’s been a problem before, some learning curve, but see below on juliac that eliminates that. There are also other alternative AOT compilers.

In some cases JIT might be used, and in general for generic code, but that wouldn’t apply calling from Rust(?). The future (in 1.12-DEV already) is juliac where you compile apps or libraries AOT. Such is already available with many restrictions (for e.g. 1.10), juliac with almost none (e.g. none compared to Rust I believe)

Yes, why not? :slight_smile: Most of the time you can “just”, meaning only use Julia. But I was answering for those that want to use those two languages (or any two+) together. It’s not a goal per se, but valuable if you have working code in another language already.

Julia has pros and cons (well any language).

Rust has also pros (better for concurrency, than most any language; Mojo also good for, I believe, with ownership model, though parallelism more of its goal) and cons. If you do not need concurrency (as opposed to parallelism) I doubt I would use Rust.

I didn’t say that, yes if I wanted to use Rust. I think it’s just not possible to change Rust to be easier in that or many other ways.

32

I think you already know what I am about to say, but just to emphasize this point:

  • what do you do when it is time to go to production

I understand that many people who are using Julia who are data analysts of some kind never think about such things because they never need to.

I’ve worked in both fields so I’ve seen it from both sides.

Allow me to define what I mean by going to production in case it was not obvious. (It may not be depending on what kinds of firms you have worked for and in which roles.)

Going to production essentially means deploying some application to a server somewhere which is not managed by human interaction on a day-to-day basis. An automated system.

Think along the lines of the webserver for this discourse website. A version of the code which runs discourse is installed on some server somewhere, someone started it from a command line, and then logged out of the system and presumably hasn’t needed to touch it except for upgrades and maintenance.

This is what I mean. An environment where (among other things) there is no REPL.

1 Like
33

You just call the relevant command-line julia form? I feel like there’s a communication mishap going on here.

4 Likes
34

I very much appreciate all the perspective you’re bringing to this conversation. I do just want to gently note that a great number of users (including myself) here also have to “go to prod” and have also seen things from both sides. So yes, many users are notebook-only analyst types; but be careful about reducing everyone to that demo.

1 Like
35

Going to production in Julia at my company involves making a Docker image with the whole environment, which then runs something like julia our_app.jl. It’s the same workflow we use for Python.

It’s not as nice as languages that have a clean binary, like Java or Go – we end up needing pretty big Docker images in practice.

3 Likes
36

Possibly dumb suggestion #2:

Can you write integration tests and deploy those tests to a UAT system.

In other words (maybe)

  • you write a standard, repeatable, input which you feed into your Rust flight control
  • you monitor from the Rust flight control some outputs
  • you spin up a server which hosts both your Julia physics simulation and your Rust flight controller
  • you check to see if your output matches a previously saved output within certain tolerances
  • run all of this using Docker and CICD
  • this in combination with specific unit tests for your Rust and Julia codes should give some pretty strong guarantees about consistency

A possible issue I foresee with this is if you get different outputs tracking down why there is a difference could be - let’s say - at least non-trivial.

Is having to compile that much of a barrier? You can often split code into modules, and you don’t need to compile everything everytime. I would argue all you are doing is moving your “waiting for compile delay” from one place to another.

What I would agree with you on is how certain things can interrupt your thinking.

  • imaging you’re working on some complicated statistics/data science project
  • the mental overhead of playing whack-a-mole with the Rust borrow checker is probably too much of a distraction
  • so you use Julia
  • but Julia takes a bit of time to compile your code at runtime, startup and initialize etc
  • but Rust takes some time to compile your code before running
  • is Julia better than Rust because of when it spends its time compiling or is it better because you have less mental overhead to manage. Isn’t it the same with Python?
  • Python is super slow and will probably take all night to run our simulations. On the other hand, Rust runs our sims in 10 minutes, but it takes 1 minute to compile. Are we choosing Rust? No - because we don’t want the distractions of the borrow checker

This is an interesting point. Thanks for raising this.

It’s ok - I also want this.

Yes exactly - my point being you can’t spawn a REPL and type

julia> using Pkg
julia> Pkg.test()

or

julia> Pkg.add(...)

Sorry hope my point is clear now?

I think this is a good solution. I have no objections to this - makes sense to me and it works well.

1 Like
37

The communication mishap continues :rofl:

If I could interest you in taking a second look at my message:

You just call the relevant command-line julia form?

38

yeah dude I’m not disagreeing with you - somewhere previously it was suggested spinning up a REPL to run the test suite was the best approach…

Sorry for the confusion!

39

All good. I think those comments are around when you’re developing a Julia package for production. In that scenario, I am pretty happy with pkg> test.

All of the REPL stuff can be done at the command line though when you need to bundle it up/ship it, it’s just a more complex invocation than with cargo.

40

Imagine you’re doing TDD. You load up the REPL with Revise.jl, your first run of tests takes ~1 minute due to compilation. Then you modify the code to add some tests and change functionality, and rerun tests. This now takes 5 seconds, and you find some failing tests. You change something else, rerun, 5 seconds.

This isn’t just moving the compile delay, it’s changing it from ~O(n) to O(1)

In my experience Rust takes ~1 minute to compile + test every time, even if your modules are relatively small, because it frequently recompiles dependencies. But my experience is a couple of years out of date, this could have changed.

2 Likes