This month in Julia world - 2023-06

Community Newsletter
1

A (potentially?) monthly newsletter, mostly on julia internals, digestible for casual observers. A biased, incomplete, editorialized list of what I found interesting this month, with contributions from the community.

“Internals” Fora (Slack/Zulip/Discourse/GHissues/GHdiscussions):

  • Wonderful learning materials on how to become a Julia compiler hacker (especially the first comment by ToucheSir). discourse
  • Some musings on support for Interfaces in Julia by Keno, Valentin (of AVR fame), Rafael, and Mark. slack/general
  • Escape analysis and its potential future use in minimizing allocations briefly discussed. slack/general
  • Discussion on current options for a trait system in Julia and potential future work (under an ASML job posting on the topic). slack/jobs
  • Keno discusses the many problems with the --check-bounds flag in julia and first steps in dealing with these issues. github
  • A long discussion on how to properly define “public” API in a more standardized way. Valentin Sukera wrote a very detailed proposal that generated many downstream ideas. There is no consensus yet, but many fruitful nearterm steps that volunteers can undertake were brought up. github (and a related discussion on discourse and related ideas in this PR)
  • A discussion on the potential for a stable public API for the effects system in the compiler. slack (and a related discussion on github)
  • More discussion of SumTypes.jl (and Unityper.jl, and Expronicon.jl) as tools for avoiding boxing and dynamic dispatch when having large unions or arrays of disparate types. (see also: abstract data types, union splitting) zulip

Core Julia Repos:

Dustbin of History:

  • Before the recent amazing improvements to TTFX (precompilation caching thanks to pkgimages in Julia 1.9), a lot of work went into potential faster/incremental compilation of sysimages. See Keno’s and Petr’s and Jameson’s work. That work had to be abandoned because of the clashing pkgimage changes, but it might one day be revived. Context: Pkgimages make loading time (i.e. using SomePackage) a bit slower, but make time-to-first-X (a.k.a. TTFX, i.e. the first call to SomePackage.somefunction()) instant. Sysimages make both instant, but are very slow to compile and they set your environment in stone (need recompilation on each Pkg add/update).
  • For a while there has been a desire to have more detailed “provenance tracking” for low level generated code, i.e. knowing what source code it comes from. With the improvements brought by JuliaSyntax, maybe this can be tackled soon.

“Ecosystem” Fora, Maintenance, and Colab Promises (Slack/Zulip/Discourse):

  • A great set of auto differentiation lecture notes including both theory and Julia ecosystem examples. slack
  • A new, very fast and polished Hidden Markov Chain package. discourse
  • An alternative to Unitful.jl which is much easier to keep type-stable (units are not encoded in the type). discourse
  • OpenCV.jl is now maintained by JuliaImages. slack/image-processing
  • JuliaFolds is maintained again (as JuliaFolds2) and in need of more helpers. zulip
  • SimJulia is maintained again (under the JuliaDynamics org), renamed to ConcurrentSim. ResumableFunctions is also maintained once again (under the same org).
  • Jacob Zelko has kicked off monthly JuliaHealth community organization meetings

Soapboxes (blogs/talks):

Sundry:

Please feel free to post below with your own interesting finds, or in-depth explanations, or questions about these developments.

If you would like to help with the draft for next month, please drop your short, well formatted, linked notes in this shared document. Some of it might survive by the time of posting.

150 Likes
2

This newsletter is an amazing initiative ! Thanks for keeping this up, hopefully the momentum lasts :pray:
Small query: how hard would it be to save these few interesting slack threads to a more permanent location?

9 Likes
3

There were a few folks in the last thread that suggested making a github repo or Hack.md account around which to structure these posts. That might be a valuable thing to do if we also want to preserve these slack threads, as it does not seem particularly scalable to copy paste the threads in the google doc draft of the newsletter.

Or maybe a slack/discourse bot that automatically makes the necessary edits…

Actually, I am putting a 100$ bounty, to be paid if someone makes a script that can take a markdown file containing links to slack, and then for each slack link it downloads the corresponding slack thread, uploads it to a github repository as a markdown file, and modifies the original link to point to the new github repo. If the script is written in Julia, then the bounty will be 50$ extra, for a total of 150$

22 Likes
4

Just to say that the discussion about saving knowledge from the slack hole has been had many times over on slack and the consensus was generally that people post on slack on the assumption that it is ephemeral, and content therefore shouldn’t be permanently published somewhere without consent.

8 Likes
5

Just thinking, could we have people consent to having their Slack messages potentially scraped, use the great SlackSDK.jl package created by @avik, pull down relevant Slack messages, and redact those messages/mentions concerning folks who do not consent such that we can still store messages publicly and permanently? We could use Slackbot to recommend a consent message that prompts for someone’s user name such that this scraper tool could have a list of consenting Slack users/non-consenting Slack users and filter accordingly? I loved @logankilpatrick 's prior initiative with emojis and helpdesk once upon a time and I am a huge advocate for pulling this content as appropriate. Zulip is great but there are great conversations happening here on the Slack.

Wondering if that could get around privacy concerns @nilshg …

2 Likes
6

in addition maybe, JuliaHub Receives $13 Million Strategic Investment from AE Industrial Partners HorizonX

10 Likes
7

This is very useful! Is ti possible to spread it to a wider audience?
I suggest Twitter and perhaps the hpcwire website.

3 Likes
8

Good idea!

1 Like
9

Fantastic initiative! The Julia community really needs curating efforts like these.

4 Likes
10

Thanks for preparing this!

I’m surprised all the JuliaFolds repositories had to be forked in a new JuliaFolds2 organization due to an inactive member… I thought one of the main points of GitHub organizations was to avoid this type of problems?

1 Like
11

This had surprised me as well, but it seems that the other members of the organization did not have merge rights to some of the repos.

1 Like
12

That is right. JuliaFolds has only one owner, and we haven’t been able to reach them. We will certainly want to move things back to JuliaFolds whenever the opportunity presents itself.

1 Like
13

JuliaDynamics has two libraries for discrete event simulation, this one and discreteevents.jl. Will ConcurrentSim replace this library or will these libraries serve different purposes?

14

From ConcurrentSim’s readme:

ConcurrentSim.jl and DiscreteEvents.jl both provide for typical event-based simulations. ConcurrentSim.jl is built around coroutines (implemented in ResumableFunctions.jl ), while DiscreteEvents.jl uses Julia’s async primitives via Channels . If you are evaluating which library to you for your goals, ConcurrentSim.jl might be a good choice if you are used to python’s SimPy, but otherwise you are advised to try a small demo project in each and do your own benchmarks.

3 Likes
15

To add to Jacobus’s comment:
ConcurrentSim (previously SimJulia) is a much older library than DiscreteEvents – it is one of the oldest julia libraries actually. Being older does not imply anything about quality, but I am mentioning it because if someone is to be replacing someone else, the onus is probably on DiscreteEvents. For what is worth, ConcurrentSim currently has a maintainer which does not seem to be the case with DiscreteEvents. But both of these libraries are not seeing serious development or support so do not hesitate to reach out if these are tools you need and want to help develop. DiscreteEvents was developed recently during a time when ConcurrentSim did not have a maintainer. Thankfully both libraries are relatively small so someone that needs this type of modeling can benchmark both of them for a small version of their problem and decide for themselves which fits their needs better.

3 Likes
16

General consensus among the Julia Slack admins according to what I understood from @logankilpatrick after asking for access to the Julia Slack through a slack api bot was that it’s better to not allow access to slack threads like this and it’s better to avoid making that data scrapper.

2 Likes
17

I’ve written a Julia script that archives Slack links and have been given the go ahead by Stefan to open source it here: GitHub - Hasnep/slack-archiver: A tool to archive Slack links

I’m not active on the Julia Slack, only the Zulip, so I’ll leave it up to the community if/how they want to deploy it.

5 Likes