Is Julia suitable to analyze the results of astrophysical simulations?

New to Julia
1

Hi everyone!

I am here to ask you your opinion. I am a PhD student and I’ve started using Julia as my main programming language in my research. I am very happy about it, despite the fact that my usage of Julia is far from deep (I come from C and MATLAB, to give you an idea of my coding background).

My supervisor teaches a course in Galactic Dynamics which consists in running some simulations with a C code and analyze the results. At the moment, the first week of the course covers the basis of Python to show how to read the snapshots / outputs of the simulations (plain text) and how to draw plots.

He asked me three questions:

*Do you think Julia will replace python for this kind of analysis?
*Do you think that switching to Julia would help the students that will look for a job?
*Should we use Julia in the course?

I have my answers, but I am very interested in what you Julia experts & users think about this and what are your answers to these questions.

1 Like
2

My very breif answers are yes, no, and maybe. Julia is quite good for this use case (see https://github.com/jeff-regier/Celeste.jl), but it’s still not the established tool for the job.

7 Likes
3

Thanks for the link! I know it’s hard, but maybe you have an idea about it based on the recent growth of the community; how long will it take before Julia becomes the standard tool for these tasks?

4

It will never become the standard tool if everyone waits to adopt it until becomes the standard tool.

12 Likes
5

Probably the answer depends on what you want to analyze. If there is some package that does exactly what you need, it is better (unless for learning purposes), stick to the package.

Curiously, despite the fact that astrophysics is not my field, one of the packages I have put up was motivated by a user who wanted a practical tool to compute astrophysical properties (Pairwise computation slower than Python (Cython) code (BallTree very slow!)). The problem there was computing the histogram of pairwise velocities of galaxies, but with some specification that was not covered, afaik, by one standard package.

This led me to split one package of mine into this package: https://github.com/m3g/CellListMap.jl

Which allows you to write your own functions to compute properties of the simulation quite simply (assuming their are dependent on the relative positions of the galaxies, planets, etc), and compute them efficiently. The advantage here over other languages/implementations, is exactly the flexibility.

Now brief two cents for the other answers:

  • Do you think Julia will replace python for this kind of analysis?

I do not think that packages that work and are reasonably well implemented will be “substituted”. But alternatives exist when people need different things and/or different performance.

  • Do you think that switching to Julia would help the students that will look for a job?

Yes. If the students turn out to learn Julia properly they will become much better programmers, even in Python. And that will help them getting jobs.

  • Should we use Julia in the course?

What is the purpose of the course? Do you need the students to implement their own analysis routines, perhaps dependent on the positions of millions of particles? If so, yes. Start with my package there, and let the students have fun :wink: .

5 Likes
6

  1. “Replace”… no. But this isn’t an either-or. In my toolbox in the garage, I have regular, Phillips head and Torx screw drivers AND I have a set of Allen wrenches. You want the right tool for the job and you want a complete set of tools. I have environments where Julia hasn’t been approved yet. I have projects that must be done in MATLAB. As a result, I have to be prepared to use Julia, MATLAB and Python.

  2. What will help most is flexibility and adaptability. I’ve used <counts on fingers & toes> over 10 programming languages in my career. Students should be encouraged not to tie themselves to a single language.

  3. I imagine that’s a function of how the course is structured. Are students required to submit the analysis code as part of assignments? The basics of reading & plotting data are similar enough between MATLAB, Python & Julia that I imagine one could structure that tutorial to use examples of all 3 and allow the students the latitude to choose what they want.

7 Likes
7

Quoted for emphasis.

A university degree should not be about teaching/learning the latest tool used in a job, but the thinking and analytical skill. Programming languages have fads over a relatively short time frame. Jobs in astrophysical data analysis must be few and far between, but there are multitudes of jobs in general data analysis these days. For those jobs, an astrophysics graduate would be competing with the legions of “data science” graduates. The astrophysicist graduate candidate is unlikely rise to the top of consideration for those based on just programming language skills, let alone one particular language. When I consider candidates, skills in a particular programming language are not a primary concern. Demonstrating that you can learn and code proficiently in multiple programming languages is.

8

Speaking as someone who recently completed a PhD in astrophysics and used both Python and Julia for my work modeling the dynamics of nearby galaxies.

*Do you think Julia will replace python for this kind of analysis?

I’m not sure what replace means in this context. If you mean to ask if Julia can replace Python at performing the task, certainly yes. If you mean to ask if Julia will be soon be used by the majority of astronomers, I think likely not.

*Do you think that switching to Julia would help the students that will look for a job?

I don’t think there’s any current job advantage in using Julia over Python in the context of this particular class. It is advantageous to know the strengths and limitations of multiple languages, but academic jobs in particular tend to care more about publications than technical background.

*Should we use Julia in the course?

Possibly! If the main goal of the course is to introduce students to the current set of tools used in the field, then using Julia may not be ideal. If however the main goal of the course is to teach the underlying concepts and learn to implement these concepts in code, then using Julia seems justified.

If you look at projects like galpy, pynbody, rockstar, halotools, most of the heavy computational lifting is done in C/C++, even if most of the users of the software use an interface with Python.

For a course aimed at junior researchers, it may be more important to gain familiarity with the tools of the field and their standard interfaces. However, as a developer of your own tools, you may see the benefit of using Julia, e.g., for solving the two language problem that other projects have faced.

5 Likes
9

Just shamelessly stressing that the pure-Julia CellListMap.jl performs better than halotools for the one task I compared them, and the CellListMap version is actually a completely customizable computation, in pure Julia (this example performs the same task as the mean_radial_velocity_vs_r function of halotools, and these benchmarks compare the two in performance).

2 Likes
10

Let the students start with Python. Then let them speed up inner loops using C++. After they get tired of writing FFI code, teach them how Julia simplifies the workflow.

11

The transition in the field of astronomy from the use of IDL to Python for small scripting tasks is not quite analogous to a proposed shift from Python to Julia, but it may provide some degree of context in terms of how long adoption could take.

Python 2 was released in 2000, but even as late as 2009, the field was split between using IDL and Python. The Astropy project, which started sometime around 2011, gathered together a much of the needed functionality for doing basic astro computational tasks in Python in one place.

Julia 1.0 came out in 2018, so my wild and crazy guess (via linear extrapolation) as to when Julia might “take over” astronomy is around 2028 :slight_smile:

That said, lots of great work has already been done with Julia in the field. Looking through the repositories under the JuliaAstro GitHub group, there’s a good mix of fundamental functionality (with decent coverage compared to Astropy), as well as novel research code. So while degree of adoption would certainly be a deciding factor in an instructor designing a course, it shouldn’t necessarily be a deciding factor in determining what language you’d like to use for your own research projects.

12

Hi!

*Do you think Julia will replace python for this kind of analysis?

Have you seen the article “Parallel Supercomputing for Astronomy”? [Researchers use Julia on a NERSC supercomputer (650,000 cores) to speed astronomical image analysis 1,000x, catalog 188 million astronomical objects in 15 minutes and achieve peak performance of 1.5 petaflops per second]: Parallel Supercomputing for Astronomy - Julia Computing

On a broader basis, not only related to astrophysics, IMO, it seems to be still uncertain, however, possibly going into this direction, especially in the technical and data science field. The adoption is growing, however, in a sense, I would say that currently, the entry barriers are not at the same level as with Python. IMO, one important area that Julia is still very much behind is access to knowledge in particular disciplines. For example, if you want to learn reinforcement learning there are books on the subject with examples written in Python, if you want to learn about quantum computing there are books with examples in Python, if you want to learn … I do not know what … there are probably books on this subject with examples in Python. Even though, Julia is still a new language, I would welcome more decisive initiatives directly related to Julia in this area. (Disclaimer: I have a very limited knowledge about Julia initiatives in general.) On a positive side, please be informed that there are books, blog sites, conferences and packages that are second to none. It should also be noted that the community is a bunch of exceptional figures. I have not expected such a level of scientific knowledge in such a density.

*Do you think that switching to Julia would help the students that will look for a job?

I would say that Julia is a unique skill and growing. My understanding is that currently there are much more jobs related to Python and Python is very much supported in various fields. Python seems to look like a mature language with … various, specialized packages … coded in C. I would say that C / C++ coding is somehow regarded as a unique and a very valuable skill. I would not underestimate C and modern C++, especially if I be an astrophysicist. (Disclaimer: I am regarding astrophysicists as a very clever people). How will it look in 5 / 10 / 15 years from now? I think, that it’s still early to say. It should be noted that there were public releases with information about Series A funding at a level of USD 24m for Julia programming language parent. This implies potentially a very dynamic growth curve, however, at the same time a significant amount of risk. Usually, you are not risking this kind of money without a reason. Again, how it develops? More is to be said in about 3 years from now.

*Should we use Julia in the course?

Absolutely.

I wish you a lot of fun coding.

Edit:

I decided to post this edit as I found two interesting and extensive readings in the area of reinforcement learning that are available online and are directly related to Julia. They are not published books in a full sense. The first one I understand is scheduled to be published on paper in 2022 and the second is a material exclusively available online and is an add on to the already published book. When I was doing my research focused on published materials they were not showing on my screen. Hope it can be useful if by any chance somebody comes by this topic here.

ALGORITHMS FOR DECISION MAKING, MYKEL J. KOCHENDERFER, TIM A. WHEELER, AND KYLE H. WRAY [https://algorithmsbook.com/] - draft of a book that is scheduled to be published in 2022.

“Reinforcement Learning: An Introduction” by Richard S. Sutton and Andrew G. Barto; Second Edition, MIT Press, Cambridge, MA, 2018 [http://incompleteideas.net/book/the-book-2nd.html] and Julia code to generate figures from the book: [https://github.com/JuliaReinforcementLearning/ReinforcementLearningAnIntroduction.jl]

3 Likes
13

Thank you for all these opinions! As a general comment, I’d like to say that the course…

  • Is not the first course on programming that the students take (it is indeed at the first year of MSc) but is possibly one of the first with a scripting language. A typical student knows C/C++ (since are addressed in undergraduate courses) and some Python.

  • At the moment, students are not invited to use any of the available tools like those cited by @awasserman, but are encouraged to write their own simple codes. This is mainly because they have to produce very short (2-3 pages) and informative reports and part of the task is to design the analysis. Students are not even required to use the programming language proposed, even if almost all of them do.

@lmiq The course is not about analysis of simulations; it is mainly focused on the physics underlying it. As a homework, students have to validate what they study (i.e. stability of a system of particles, timescales for collapse, …) by preparing the initial conditions (with their code), launching simulations (pre-written code usually not changed by students) and then analyzing the outcome (with their code). Thanks for sharing CellListMap.jl, it can be very useful on many of these computations. The code written for the analysis is not reviewed nor evaluated see the edit below.

@fyzycyst @Jeff_Emanuel You are right. Thinking and analytical skills are the most valuable things one should learn. This is also the reason why I think that introducing the students to a modern language is valuable, especially since they are asked to write their own codes.

As you may have understood, Julia in this course is used as a Python+NumPy substitute. The syntax will be very similar, in my opinion.

We’ll see in 2028, then :smiley:

@j_u The point about books is indeed relevant. IMO Julia is growing very fast, even in terms of feature and syntax and this requires a constant update of support material. I think that this is mainly due to the fact that Julia has been around for less time, so it will solve somehow in the future.

EDIT: from last year, the code for initial conditions generation and for the analysis is to be submitted by the students.

3 Likes