I haven’t followed the discussion on Slack, so I don’t know if this was discussed: I’m sure that most people are familiar with the quite successful distribution of software on Linux systems. Isn’t that where good lessons can be learned, and perhaps a template found for the Julia distribution of packages that would ensure the same level of trustworthiness that seems to be available on better-known Linux distributions?
A bit of story on the R ecosystem. The main official registry is CRAN which goes through a rigorous application process, gated/somewhat peer-reviewed. If packages break, deprecated, unmaintained, or security flaws are found those are purged from the registry. Similarly you have other high-quality registries such as Bioconductor. Many authors have grown tired of the peer-review gated journal like environment and prefer to have the packages installed directly from Github. You also have other registries maintained by corporations that synch/mirror CRAN with certain differences such as Microsoft (MRAN).
In the end, we need to identify the issues and needs. A package manager / registry is a tool to solve the following problem: “What is the best tool for this task that I can use given my circumstances and needs?” For example, I only want packages that run in my system (compatibility) or that meet my application restrictions (permissive licenses). For many instances one only wants to use production ready maintained documented packages (not in development stages, unmaintained, or deprecated). In many cases, I want to be able to trust the output of the tool and such I might limit those under considerations to those validated with robust testing or have an associated peer-reviewed publication. Malware and security flaws do occur, but is not the only consideration.
My suggestions would be to have a redundant set of systems, curated registries with certain standards (e.g., CRAN), quality metadata in Project.toml for informed decisions by users, a centralized review system (à la Crantastic), tags/reviews seals of approval for filters and labeling (e.g., Non-GMO, Organic).
5 Likes
Couldn’t a malicious person circumvent this by first making an innocuous merge, and then introduce the back-door on his second merge (a week later) when nobody gets a warning?
1 Like
We should also think about the match between the resources (mostly work) required by a curated set of packages, and whether there is a supply of the latter that is matched by the demand for the output.
To put it bluntly, “Julia absolutely needs a set of curated packages” or “serious people/corporations won’t use Julia unless…” are pretty empty statements if not followed by “… and I am willing to contribute work or sponsorship.”
In particular, I am not sympathetic to the needs of corporations or other environments which somehow require audited packages etc, unless they contribute somehow.
1 Like
I was thinking about this problem and idea that “curated repository could decrease attractiveness of JuliaPro” came.
But then I was thinking “what is attractiveness of JuliaPro?”… Product description is too vague. So I start to looking for documentation and found total mess about pyplot (autogenerated?) in JuliaPro Packages API Reference. Which really does not help to improve oppinion about professional level.
So basically this seems to be very accurate:
But I didn’t loose hope and believe that some sponsor could (and will) bake curated registry.
It seems you are thinking that some corporations see competitive advantage to make money using Julia.
I am afraid we are not there yet!
But there are people in corporation environments which want to prove that Julia could do that. Their position is difficult and current package management system doesn’t help them.
There are also people who evaluate Julia. Expecting that they will work for free or sponsoring missing feature is kind of absurd.
2 Likes
This is not true. Pkg(3) has all the facilities required for curated registries and reproducible environments. With the latter, if a programmer insists on carefully auditing a particular set of packages, he can freeze the “state of the world” by committing Manifest.toml and only updating when necessary or time permits.
Pkg is merely a tool. It allows curated registries, that was implicit in the design from the beginning. Someone has to do the work though — there is no escaping this.
In case you missed it, all of the Julia ecosystem is the result of work provided for free (mostly) or by sponsors (a bit).
7 Likes
Stefan had a good point (over in slack), that a “curated” and a “trusted” repository should not be conflated. I suppose “curated” could imply “trusted” but not the other way around. “trusted” should then just mean that we trust that there is not malicious code in there (now and in the future) but would not imply any other attributes, such as quality. I think first we should aim to have “trusted” packages.
4 Likes
Here a suggestion on how trusted-packages could work, piecing together suggestions from above. A trusted package could be defined as:
- it is marked as “could-be-trusted”
- only trusted developers have commits in this package
- it only depends on trusted packages
Point (1) is to make sure that trusted developers do not automatically vouch for all their packages. (Also, more correctly above should be applied to package releases.)
A user can then have a list of trusted-devs, usually this would be provided, say the core-devs. But this list can be updated by the user (globally or per project), say, add all co-workers of their company plus the Jump team.
From this, it could be computed which packages are trusted (or more correctly, which releases of a package are trusted) and which not. A user can then mark a project of his/her as only allowed to use trusted packages and Pkg would then make sure that that is the case. Adding a not trusted package could then give helpful errors: Package Parameters cannot be a dependency, because you do not trust developer @mauro3. To use this package you need to add @mauro3 to your list of trusted developers.
Note that this would not need an additional package repository. It would only need the ability of adding a “could-be-trusted” tag to a package and the ability to calculate trust based on the list of trusted-devs. Also, this would not require any extra time from the package developers as no extra code-reviewing is needed.
A case such as flatmap-stream would be caught by above scheme.
(of course this does not address things such as signed commits, etc.)
Edit: sorry, this post is a bit badly phrased in the sense that I looks like I want to take credit for above ideas. Many have been discussed here and on slack before. I just tried to put them into one place.
4 Likes
What one could do is set up a package-scanner. This would be a piece of software that continuously downloads any new release of any package and automatically scans for suspicious code patterns or behavior.
Potentially, one could use Cassette.jl to create a tool for monitoring Julia code while it is running.
Examples of “suspicios behaviour” might be to contact a web-server whose domain name is hard-coded in the source but is not in a white-list. Or to generate and then evaluate expressions that make system calls. Or…
Anything suspicious would be flagged for a human to verify and take action if needed.
For fun, one might set up an interface for people to submit sneaky code and see if the scanner will catch it. But this must not be a completely automated process that allows malicious actors to test out attacks before using them for real.
The software could be partly open-source (basic infrastructure), but the code that detects each attack vector must be closed-source in order to prevent malicious people from figuring out how to circumvent it without exposing themselves by trying an actual attack.
Just like with code review, this would require a lot of effort, but it would not be as monotonous, so has a higher chance of attracting volunteers. Also the code that detects a given attack vector could keep running even if the person who wrote it loses interest in the scanner project.
1 Like
That looks like a good source of inspiration, yes.
That looks like a database/scanner of existing vulnerabilities. Useful for sure, but only for catching malicious code ex post. Eg apparently event-stream ended up in the database a week after the github issue.
It is up to the person who develops an attack to decide whether it goes directly into the scanner database or is used for malicious purposes first…
I think it would be better if the scanner uses heuristics based on the type of code being added in comparison to what’s already in the package, and then provide one or more “scores” that the user can set thresholds on (probably globally as well as per-package). Triggering any threshold will prevent the package from being automatically updated.
An example of a heuristic would be the number of uses of TCPSocket in relation to the rest of the pre-existing code, or calls to various shell commands with run(), etc.
This approach wouldn’t require closed source code as long as there are plenty of heuristics being used, since gaming all of them (and bypassing most user’s thresholds) would require an extremely high level of cleverness, or would just be impossible.
I also would like to say that closed-sourcing any of this is a bad idea. Aside from being security-through-obscurity, which the security community typically frowns upon, this doesn’t prevent the closed-source scanners from being gamed. You could just submit a bunch of new packages with malware in them to the scanner, each with slight variations on the exploitation mechanism, until one or more of them pass the scanner unnoticed. Then you just plug that into an existing package, and game over.
I don’t think there is a free lunch here; I can’t imagine this not triggering a lot of false positives, which would require (costly) human intervention.
1 Like
Sure, I agree. But if you’re the kind of (corporate/government/paranoid) user that cares about this sort of thing, you’ll be willing to do a review of your triggering packages anyway (might as well). If you don’t really care, just raise the global threshold until it rarely or doesn’t trigger. The point is, give the user the possibility to make these decisions for themselves, even if it adds overhead to those who choose to be strict.
Perhaps I missed something, but my impression of the current situation is that there isn’t anyone systematically auditing (even a subset of) packages. Even the current infrastructure would allow this (with a parallel, “trusted” registry).
Maybe it is happening, just not publicly, but I don’t know if it is possible to meet the demand for this kind of security with the supply of work/money/resources.
If no one is auditing the packages in a trusted registry, then there’s no good reason to trust the trusted registry. I’m only addressing the proposal to have a package scanner, which could be a useful tool for directing the efforts of those who wish to maintain a trusted registry (of which their could be more than one).