Is there anything users can do to help move Apple Silicon support from Tier 3 to Tier 1?

Internals & Design
1

Is there anything users can do to help move Apple Silicon support from Tier 3 to Tier 1? Given that most new Mac machines now have an M1 chip, it seems like there will be more and more Julia programmers wanting to use a Tier 1 Julia release on their shiny new Macs. :slight_smile:

3 Likes
2

Julia is in better shape than most competitors when it comes to *native M1. Rosetta can always be used

3

Report bugs (preferably with RR).

3 Likes
4

rr is Linux only AFAIK.

2 Likes
5

RR is x86 only (funnily enough keno was working on porting it to aarch64) and Linux only.

1 Like
6

Yeah, it sounds like Julia works reasonably well on Rosetta. But I’m greedy—I hate the idea that I’m leaving some performance on the table.

2 Likes
7

An Apple silicon version of MKL.jl would be great. My guess is that is not an easy thing to build and don’t expect it anytime soon.

1 Like
8

IMHO / As I see:
one of the next steps is adding an “AppleSilicon” testing to the Julia BuildBot/CI/CD ( I can’t see AppleSilicon ; only MacOS64+X86_64 )

So need some financial support in:

  • Support Julia on AppleSilicon Buildbot/CI/CD testing ( Hetzner ? )
  • “bug bounty” for some issues
  • Monthly - (re)testing all Julia packages on AppleSilicon.
  • Give (package/core) developers some remote access to debug issues on AppleSilicon ( another Hetzner M1?)
  • …

→ Sponsor @JuliaLang on GitHub Sponsors · GitHub

2 Likes
9

I don’t think that’s really useful before https://github.com/JuliaLang/julia/issues/41440 is resolved. That’s the main stumbling block at the moment. All tests jobs are going to segfault because of that, it isn’t incredibily useful to run jobs that systematically crash for a well known issue. The problem is that that’s not trivial to address.

Additionally, most (but not all?) Julia CI runs on on-premise hardware, I’m not sure renting overpriced cloud machines is the best option money-wise. M1 boxes may already be in the availability of the developers, but until someone takes the time to solve https://github.com/JuliaLang/julia/issues/41440 there is not much need to use them.

That said, also on-premise CI has a non-zero cost, so supporting the project is a good idea anyway.

7 Likes
10

I had no information about this.
It is free? Where can this be accessed?
( I am on Linux … and I don’t want to invest to applesilicon hardware yet )

11

I was talking about the fact Julia core developers already have access to those machines, not that M1 boxes are given away for free to anyone.

12

I suggest taking this approach:

I’d add that to your startup.jl, probably behind a check to make sure you’re on the native version instead of Rosetta (i.e., check Sys.ARCH).

1 Like
13

it is good news.
on the other hand I am thinking in a bigger ecosystem:

the ecosystem is different:
based on the survey:
" Julia users seem to love continuous integration (CI)! A staggering 95% of packages in the ecosystem are set up to use any continuous integration service, with GitHub Action being the most popular service (80%), followed by Travis (42%) and AppVeyor (13%)."

And now Github Actions is not yet supporting Apple M1 VMs (AppleSilicon)

IMHO:
an ecosystem of healthy packages (pre tested on AppleSilicon machines! ) is essential to support Tier1
Because the Julia Release Process contains testing the package ecosystem.

  • “PkgEval is a tool for running the test suites of all Julia packages, which helps us make sure that we haven’t inadvertently broken anything. Each failure is examined when a release is made: we verify that the failure isn’t due to a violation of SemVer and try to make pull requests to fix packages, regardless of the cause of the failure.”

Limited remote access for trusted package developers will be useful.
and I am “thinking in the ecosystem” and not just “in the core developers”.

14

In case you don’t know, Mosè is an author of that survey so he is definitely aware of those stats!

2 Likes
15

We both have different perspectives on the problem. ( or different pain points )

  • He is a core developer with access to the AppleSilicon hardware
  • I am part of the ecosystem without AppleSilicon testing possibilities.

There is free/cheap X86_64 testing for the Julia ecosystem …
but I don’t know of a similar cheap AppleSilicon testing facility.

16

It seems unnecessary to provide free cloud computing time on very expensive hardware which is Tier 3 and on which most non trivial workloads currently seem to segfault as Mose says?

2 Likes
17

Setting aside the segfault issues, this seems like a catch-22:

  • “We don’t need CI for Apple Silicon because Apple Silicon is Tier 3.”
  • “Apple Silicon is Tier 3 because we don’t have CI.”
18

Well, I just wouldn’t agree with that point :slight_smile:

Would it be great to have free Apple Silicon CI available for the entire ecosystem? Sure.
Will we have it at some point? Maybe.
Is this the reason why Julia is Tier 3? Absolutely not.

The way I see it, the Tier levels have hardly anything to do with the (general) ecosystem. They make a statement about Julia itself.

1 Like
19

I think this discussion got a bit off track: When we are talking about CI on M1, and renting it in the cloud, or buying some etc. then I think this can only applies to making that CI hardware available to the CI of the Julia project itself, and thus in the end core devs.

I do not think the Julia project itself can afford to provide M1 CI instances to arbitrary packages. Do you expect them to create an alternative to GitHub Actions, Travis, AppVeyor, CircleCI, etc. ??? No, if you want arbitrary packages to have M1 CI, you’ll have to hope that some of those providers will eventually offer them.

Thus, when it comes to making M1 Tier 2 or even Tier 1, while of course having consistent CI for Julia itself is kinda essential for that, first the already known serious problems ought to be resolved; there is no point setting up a CI for something that will just crash all the time. Once these fix are in place, I am sure there’ll be a way to get M1 builders for the Julia CI. But providing this for the wider package ecosystem is not something the Julia team can take care of.

On the upside, waiting increases the chance that e.g. GitHub Actions will just make something like this available. However, AFAIK GitHub Actions are on Microsoft Azure and I have no idea when, if ever, they will provide M1 support. On the upside, AWS EC2 M1 Mac instances are in preview; once they are generally available, I am hopeful some CI providers using AWS will eventually offer M1 instances.

8 Likes
20

It’s not: to unblock the situation, already known issues have to be fixed. CI is really a minor problem at this point. Also, I had already pointed out a side approach to make more likely packages will work on the M1: test them on aarch64 Linux. It’s clearly not the same thing, but helps a lot for catching architecture-related issues, and I’ve personally caught some this way. It doesn’t help much with platform-specific issues (like the well known segfault).

4 Likes