RFC: Proposal for estimating unique Julia clients without tracking

General Usage
, , ,
41

I want to chime in with another perspective to what these kinds of threads are usually full of.
I should state up front that I am in no way associated with JuliaHub - keep that in mind, because if the following text was written by someone from that company it world be public relations suicide.

I am also not writing to any one particular person. Rather, I write to counter a set of ideas that have an outsized influence on these kinds of online discussions. See for example, last this was discussed.

I think that pearl clutching over Julia’s package servers storing a few bits of privacy-preserving information is utterly insane and grounded in delusional and antisocial ideas of what freedom means.
When you’re downloading a Julia package, you’re not exercising some ethereal God-given right from the spirit realm, you are consuming a very worldly resource that other people are paying for with their real money. Thinking that people owe you not just free bandwidth, but cannot be allowed to learn anything about you is analogous to beliving your electricity provider must not only provide you free electricity, but that it’s an affront to your freedom if they have any information about how many customers they have, who they are, or where they are placed.

It is an otherwordly belief that can only spread in communities not tethered to physical reality, because this kind of belief is obviously unworkable in real life. When you go shopping with your balaclava and gloves on (so your neighbors can’t identify you), paying for your yoghurt in unmarked cash (your credit card has your name on it!), does it worry you deeply that the supermarket might do statistics on how many cups of yoghurt they sell? Luckily you don’t have to worry they know that they’re selling it to you - we’re not discussing personal identifiable information in this thread - but the fact that they could have numbers on the amount of yoghurt they sell in each store? The audacity!

I don’t know what Julia’s package infrastructure cost to run just in hardware. Maybe € 100k / year? And of course, Stefan Karpinski probably didn’t write the HLL over RSA protocol in his free time, I’m guessing he was on the clock on a JuliaHub salary while coming up with that and making the PR. Thinking that you’re the person being wronged when the company, in return for all those real, actual resources, want to know who they’re giving the resources to is beyond entitled.

And I do mean “beyond entitled”, because this level of entitlement is not a personal defect, but can only be reached through ideology. An ideology I see too often in online circles, especially those with FOSS people: The belief that “freedom” means you can act in society and affect other people, without the possibility of being held accountable to your actions, or even having to defend or identify with your actions because you can be totally anonymous. It’s the same antisocial ideal that brought us cryptocurrency, anonymous holding companies and shadow banking. This belief is foundationally against what “society” means.

Of course you are entitled to your opinion. And I don’t even think Julia would be better off if people with these beliefs disappeared - Julia is better when there are all kinds of people, including people with stupid political and social ideas. But the idea itself is not worthy of respect, and we should not base our discussion on a set of premises so outside the Overton window that physical reality demands. I’m frustrated that every discussion on privacy is hijacked by these foil hat extremists for whom the idea of nuance is lost.

42

I certainly wouldn’t feel comfortable putting it this strongly, and I have a lot more empathy for the privacy-cautious here, but yeah, I more or less agree with you. The standards and concerns here are at some point quite silly.

Still though, I deeply appreciate @StefanKarpinski’s efforts to find a way to get useful information here in as privacy friendly a way as possible, and being so sensitive to people’s fears.

43

Could we produce two set of downloads: one with the privacy-preserving unique Julia client estimation code enabled and another with the code disabled or even removed? Thus one could opt for one or the other at download.

44

I think this would defeat the purpose of getting an unbiased estimate of usage (because it introduces selection bias)

45

If we allow for an opt-out, would we not also introduce selection bias?

46

All user count estimates will have biases, for example, I have different julia installs for CI, running julia on my university cluster, or debugging or testing Pkg.jl itself. Those probably shouldn’t be thought of as different “users”. I’m not sure how useful a user count is, I care more about the contributor and the dependency graph, which we already have nice statistics for.

47

it would, yes. I would not be in favor of any opt-out. I think the privacy controls here are already extremely good compared to the telemetry attached to pretty much literally any other activity one does online (unless living like the unabomber). I would tend to agree that criticizing this proposal on privacy grounds comes across as naive to me.

48

There is no way to disallow people from opting out, it is just a question of how the opt-out should be implemented.

49

We already try to separate out CI — and to be kind you can make sure that one of those environment variables is set in your CI system.

50

That is life. There absolutely MUST be an opt-out, and I think nobody in their right mind thought that not having an opt-out is an option.

The selection bias is not a problem: The “user-agent”, i.e. Pkg, would report “user declined to participate in hll counting”.

I would not use separate builds, and rather store a config variable somewhere (for persistent opt-out), that can be overridden by some environment variable.

Just imagine what the alternative “implementation” of opt-out would be: Some angry people on discourse post a bunch of patches that includes an opt-out mechanism. “oh, you just need to apply this patch to that package (pray that there is no merge conflict) and build from source”. Linux distros (or e.g. AUR) start shipping their forks that include an opt-out mechanism. Obviously each distro with differently named flags. Security updates get delayed for people on the unmaintained private forks. People get forced into using weird versions.

Ultimately, the spirit of free software is that stuff you run on your computer is serving YOU, and not the developer of the software. This is backstopped by the usual mix of
1.“by the power of your own hands”, i.e. your capability of adversarially forking (or binary patching / modding) the piece of software
2. “by the power of your community”, i.e. your peer’s capability of maintaining forks / software that serves YOUR needs (and your power of community-organizing!)
3. “by politics / laws” (e.g. copyright covering GPL stuff, e.g. GDPR, establishing at least some privacy protections)

Adding some more or less privacy-preserving usage statistics, and enabling that by default, accords to that: It is a nudge towards a tiny act of solidarity with the community that ensures that your computer serves YOU.

Excluding an opt-out to this would be utter insanity.

51

Somewhere thereabouts. I’d have to look up the actual numbers, but it’s not nothing.

I did actually do this in my spare time. This is definitely not for work. Probably spent way too much time on it, but it’s a frustrating / fascinating problem. And of course, once I figured out the encrypted sampling thing it became a matter or “damn, this is cool”.

(There is an opt out via environment variable in the PR, of course, no need for separate downloads.) Technically this does introduce bias, but since approximately 0% of people opt out, not really.

And if there are CI systems out there that don’t set these variables but set something else, then let us know what they are so we can update this.

52

Regarding sharded vs. unsharded, there is an interesting possibility: you can have each client’s bucket index be the same across all resource classes but have their geometric samples be different and independent across them. In other words, instead of sending (b_h, k_h) where b_h and k_h both depend on the resource class, they send (b, k_h) where b is the same in every class. Keep in mind that b is only 12 bits and there’s a lot of clients so this is not a key that allows tracking — the occasional rare, large k values are the ones that could be a problem and they’re still sharded. This isn’t even terribly hard to implement. In the current design, the client sends:

y = w (x_0 g^h)^t

where

\log_g(g) = (1, 1, 1, 1) \in \mathbb{Z}_2 \times \mathbb{Z}_B \times \mathbb{Z}_{2^m} \times \mathbb{Z}_{pq}

and g is published by the server along with the other ring details. We can instead have the client send

y = w (x_0 g^{Bh})^t

With that fairly minor change, the h exponent modifies all exponents except for the bucket. @foobar_lv2, I feel like that addresses the bulk of your request bundle fingerprinting concern. Does it? There’s only about 2 bits of info from each geometric sample value, k_h, so I don’t think the same request bundling concern meaningfully applies, especially with the highly skewed distribution (most values are a single bit).

I also think that we should store data aggregated by hour (and later by day, etc.) rather than individual request records, which would also address the issue since you can’t isolate request bursts in aggregated data the same way. But we want to push as much of our mitigations into the protocol which cannot be circumvented, rather than into operational practices, which an operator could fail to implement or implement incorrectly.

53

i know this is a weird question, but will this effect alternative Julia package managers? i dont want to advertise here, but i did make my own alternative Julia package manager so i am curios to if it will effect it or other attempts at that?

54

Not a weird question at all. The pkg protocol is carefully designed to allow alternative implementations and federation. (JuliaHub, in fact, has an alternative implementation.) No, this is an optional part of the protocol; if the client asks for $server/hll_rsa.toml and gets no response then it simply doesn’t send this header. If you want to implement this extension yourself, you can also do that, although probably not worthwhile at most scales outside of the public servers.

55

thank you! its reassuring to hear that its an optional part :slight_smile: my alternative implementation is extremely small right now, so as you said it probably wont be worthwhile at my current scale.
anyway thank you!

56

Yes! I think this is a much better version of sharding!

I honestly would store individual records for ~3 days, for incident response, and then start deleting / bucketing.

After the HLL protocol is fixed, you get to implement it in some way in ajt.

Initially tell the server “nope, I’m opted out of download statistics collection”. Simplest implementation. Maybe also tell the registry that you are ajt instead of Pkg.jl?

Once some dust has settled, implement the protocol, with separate entropy / “identity” (no big deal if users who use both ajt and Pkg look like two different users). You can simplify the protocol in some ways: For example, you could just skip the zero-knowledge-proof that the server’s RSA keys are “good”, i.e. don’t have a privacy-destroying backdoor. This is of course not ideal, but you can practically rely on all the Pkg users to keep the servers honest, i.e. spot backdoored RSA keys. I’m pretty sure that people from Pkg, discourse, and @StefanKarpinski will be happy to help you on that.

Having a completely independent and interoperable and deployed implementation of the RSA-HLL protocol would warm our heart, and make Stefan’s eventual academic paper that much better.

At some point in the future, the implementation wants to be a “real” library. I.e. don’t rely on julia TOML / libGMP, and instead use cryptographic libraries. In the process of reviewing Stefan’s proposal, I have also looked into stuff like libGMP’s bigint parsing. That’s all on the critical path! And while I could not find smoking guns, this is not the kind of code I’d like to rely on for security-critical things. It smells.

This is mitigated by the fact that Pkg is asking to download code to run – a truly evil server can pown the user without trying to find libGMP exploits, simply by delivering backdoored packages. So an initial implementation can get away with that.

But the general protocol / idea is widely applicable: Approximately count individual users of a resource in a way that makes reasonable effort to preserve user privacy, with data embedded in http headers, and good mitigations against both malicious servers and malicious clients. This wants to be an RFC-style protocol with a variety of implementations in different languages (C, C#, javascript, java, with julia bindings for pkg and zig bindings for ajt).

First doing that for Pkg in a very hand-rolled fashion, and gaining experience, makes sense.

57

Before disengaging from this thread, I am posting this final response to place on the public record the exact point this discussion has reached.

Up to this point, this debate was ostensibly about telemetry design, cryptography, and user consent. However, the discussion has now escalated into unmoderated character assassination against community members advocating for standard open-source privacy practices.

Community members who have expressed legitimate concerns about opt-out telemetry and user autonomy are now being explicitly accused of:

Holding views that are "utterly insane and grounded in delusional and antisocial ideas of what freedom means."

Subscribing to the "same antisocial ideal that brought us cryptocurrency, anonymous holding companies and shadow banking."

Displaying a level of "entitlement" that is "not a personal defect, but can only be reached through ideology."

Acting as "foil hat extremists for whom the idea of nuance is lost."

It is deeply concerning that these direct personal insults, aggressive bad-faith characterizations, and plain violations of any standard of civil interpersonal communication have received zero intervention or moderation from project leadership. When hostility of this level is directed at users raising standard FOSS privacy concerns and is allowed to stand unmoderated, it signals that personal degradation is an acceptable tool for silencing any form of dissent and threatens for this to become the norm.

Beyond the personal attacks, this stance makes the implicit financial expectations around this telemetry PR completely explicit. It argues that because JuliaHub subsidizes package infrastructure (“€100k / year”), users are “beyond entitled” if they expect privacy while consuming those resources.

If a corporate sponsor requires user data to justify their server costs, the honest approach is to put that in a Terms of Service agreement, not to hide it behind a cryptographic protocol and opt-out defaults.

I do not share the viewpoint that expecting my local machine to remain private is “antisocial,” nor do I agree that I owe JuliaHub my data in exchange for using an open-source package manager. However, it is abundantly clear that the cultural and ethical consensus of this project’s leadership has shifted.

Judging from the likes on those comments and the complete lack of moderation, this hostility seems to be the accepted viewpoint. If the accepted method for building consensus in the Julia community is to allow the environment to become so toxic that dissenting users can no longer remain, then that tactic has succeeded. I refuse to subject my mental health to an environment that operates this way, nor will I legitimize this standard of discourse by remaining a part of it.

You are free to merge the PR. I will be stepping away from the Julia community entirely. You win.

58

As someone in favor of giving the user a robust opportunity to opt-out, I do not share your sentiment about the nature of this discussion here. Noting this is the second such discussion in a long time about a telemetry mechanism, and that the first approach was eventually withdrawn, I do think there is a bonafide attempt to address both a need to collect basic statistics while maintaining privacy.

You can ascribe people’s expressed opinions as personal attacks, but I do not think that is a useful reading in a discussion like this.

As for moderation, there is a flagging mechanism that this forum uses to draw the attention of the moderators. It’s not clear that necessarily has been used. I should also note that there is a clear separation beteeen project leadership and moderation as well. Stefan in particular does not usually play a direct moderating role here.

I also see there that is a concern about JuliaHub as a corporate entity. I do have to say that there is a significant attempt to separate the interests of the open source Julia community from the corporate interests of JuliaHub, which I find very unusual when such a corporate participant does exist. I only have to look at Mojo and Modular / Qualcomm to see how stark the difference is there.

Anyways, take care and thank you for your expressing your opinion.

59

I don’t see bad faith, in the sense that @jakobnissen would be misrepresenting the views of a certain FOSS sub-culture. He is not ascribing to this sub-culture positions they do not hold. What he is doing is characterizing those positions in somewhat strong language as “insane” and “antisocial” – in his opinion (which I’d be inclined to share to some extent, even if not to the fullest extent).

There is nothing wrong with the expectation of privacy… which is what this proposal is trying to provide. That is not the same as insisting on absolute privacy (whatever that can mean in any coherent sense). As with most issues, any position that is absolute is almost guaranteed to be deeply misguided, and likely (hopefully) held by a fringe but sometimes very vocal minority. I find it useful to call out such misguided positions, so as not to let them override what is a strong consensus (based on “likes” in this thread) on a more nuanced position. Maybe the language to do that here was a bit stronger than absolutely necessary, but, as far as

goes: Even the strongly worded characterization of these dissenting user’s positions seems like a far cry from “toxic” to me. This is not a situation where a group of dissenters is hounded or harassed, but merely is receiving some strong pushback on particular positions, directly relevant to the topic in this thread. I would recall (and personally endorse)

which is a direct invitation to remain, even if it means receiving pushback on certain ideas in threads such as this. I would hope that there is plenty of common ground in other areas and threads on this site that such a disagreement should not reasonably drive anyone away.

60

Not really looking to engage in this thread, but thought some authoritative statements were needed to correct incorrect information:

  1. The cost of Julia infrastructure exceeds 100k €/year by a factor of ~2-4 depending on what and how you count.
  2. JuliaHub does not significantly subsidize infrastructure cost, maybe $10k-20k a year amortized (but does of course pay salaries of people who work on Julia at a scale quite a bit larger than that and also sponsors JuliaCon, which may flow back into infra).
  3. Most of the subsidization of infrastructure cost is in form of in-kind sponsorship by Amazon (~ $120k/year), Fastly and CloudFlare, plus hardware at MIT and JuliaHub.
  4. JuliaHub does not really care about this data. It’s primarily useful for the project and associated academics to request funding support from industry and governments in order to demonstrate impact. There was a time when JuliaHub also did this kind of work, but at this point that’s not really part of the business anymore. It also helps to know what packages are popular so we know how much to prioritize a package when things break.

I know it’s the Internet, but it’s really strange to have the people who love one of the things you’ve poured your life’s work into think of the other as some kind of evil boogeyman. In particular the character impugnment is both unnecessary and to my mind in significant bad faith. As i said, I don’t have too much interest in further engaging here - too much actual work to do, but I felt that it needed to be said.