# Announcing: package download stats!

**URL:** <https://discourse.julialang.org/t/announcing-package-download-stats/69073>\
**Category:** Announcements\
**Tags:** package, statistics\
**Created:** [October 1, 2021, 8:42pm UTC](https://discourse.julialang.org/t/announcing-package-download-stats/69073 "2021-10-01T20:42:39Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [October 1, 2021, 8:42pm UTC](https://discourse.julialang.org/t/announcing-package-download-stats/69073/1 "2021-10-01T20:42:39Z")

</div>

After some years of getting the package server architecture in place and working (mostly) reliably, @staticfloat and I finally had some time to work on collecting logs into a data warehouse (we’re using [Snowflake](https://www.snowflake.com)) and designing a set of queries over the logs that we can run and publish regularly. The current public aggregated stats are available with the prefix

```julia-auto
https://julialang-logs.s3.amazonaws.com/public_outputs/current/

```

followed by a rollup name and the suffix `.csv.gz` indicating that all the files are gzip-compressed CSV files. The rollups that we currently publish are variations on the following basic views, listed here with the rollup name and the log fields that each one is indexed by:

- `client_types`: `client_type`
- `julia_systems`: `julia_system`, `client_type`
- `julia_versions`: `julia_version_prefix`, `client_type`
- `resource_types`: `resource_type`, `status`, `client_type`
- `package_requests`: `package_uuid`, `status`, `client_type`

In addition to these, there are variants of these rollups that also have `region` and/or `date` as keys, which is indicated by appending `_by_region` and/or `_by_date` to the rollup name. The full list of rollups is:

- [`client_types`](https://julialang-logs.s3.amazonaws.com/public_outputs/current/client_types.csv.gz)
- [`client_types_by_region`](https://julialang-logs.s3.amazonaws.com/public_outputs/current/client_types_by_region.csv.gz)
- [`client_types_by_date`](https://julialang-logs.s3.amazonaws.com/public_outputs/current/client_types_by_date.csv.gz)
- [`client_types_by_region_by_date`](https://julialang-logs.s3.amazonaws.com/public_outputs/current/client_types_by_region_by_date.csv.gz)
- [`julia_systems`](https://julialang-logs.s3.amazonaws.com/public_outputs/current/julia_systems.csv.gz)
- [`julia_systems_by_region`](https://julialang-logs.s3.amazonaws.com/public_outputs/current/julia_systems_by_region.csv.gz)
- [`julia_systems_by_date`](https://julialang-logs.s3.amazonaws.com/public_outputs/current/julia_systems_by_date.csv.gz)
- [`julia_systems_by_region_by_date`](https://julialang-logs.s3.amazonaws.com/public_outputs/current/julia_systems_by_region_by_date.csv.gz)
- [`julia_versions`](https://julialang-logs.s3.amazonaws.com/public_outputs/current/julia_versions.csv.gz)
- [`julia_versions_by_region`](https://julialang-logs.s3.amazonaws.com/public_outputs/current/julia_versions_by_region.csv.gz)
- [`julia_versions_by_date`](https://julialang-logs.s3.amazonaws.com/public_outputs/current/julia_versions_by_date.csv.gz)
- [`julia_versions_by_region_by_date`](https://julialang-logs.s3.amazonaws.com/public_outputs/current/julia_versions_by_region_by_date.csv.gz)
- [`resource_types`](https://julialang-logs.s3.amazonaws.com/public_outputs/current/resource_types.csv.gz)
- [`resource_types_by_region`](https://julialang-logs.s3.amazonaws.com/public_outputs/current/resource_types_by_region.csv.gz)
- [`resource_types_by_date`](https://julialang-logs.s3.amazonaws.com/public_outputs/current/resource_types_by_date.csv.gz)
- [`resource_types_by_region_by_date`](https://julialang-logs.s3.amazonaws.com/public_outputs/current/resource_types_by_region_by_date.csv.gz)
- [`package_requests`](https://julialang-logs.s3.amazonaws.com/public_outputs/current/package_requests.csv.gz)
- [`package_requests_by_region`](https://julialang-logs.s3.amazonaws.com/public_outputs/current/package_requests_by_region.csv.gz)
- [`package_requests_by_date`](https://julialang-logs.s3.amazonaws.com/public_outputs/current/package_requests_by_date.csv.gz)
- [`package_requests_by_region_by_date`](https://julialang-logs.s3.amazonaws.com/public_outputs/current/package_requests_by_region_by_date.csv.gz)

There is also a `log_recency` CSV file published here:

- [`log_recency`](https://julialang-logs.s3.amazonaws.com/public_outputs/current/log_recency.csv.gz)

This CSV file has two columns: `time_utc` and `pkgserver` with a row for each package server where `time_utc` is the latest log entry available for that server. This is helpful to check if some package servers are delayed in uploading their logs. There is also a row in this file where the `pkgserver` is empty (null/missing) and the `time_utc` field is the time at which the query was run, which should be just before all the CSV files were uploaded.

These CSV files are generated each day at 4am UTC over the past year’s worth of logs. Logs are uploaded from servers every 2 hours, so aggregating at 4am and excluding the current day means that we should typically have complete logs from the last day that’s included in the stats, but we’ll never have stats including the _current_ day. For the first four hours of each UTC-day, the last day of logs will be from the day before yesterday.

### Rollup keys

The fields by which we aggregate various rollups are the following:

- `client_type`: one of `user`, `ci` or empty (null/missing) indicating whether the request appears to be from a normal user, a CI run or some other kind of client like a normal web browser, web crawler or some server that’s mirroring the package servers.

- `julia_system`: a hypen-separated “tuple” indicating the client system’s characteristics that Pkg uses to determine what system-specific pre-compiled binaries to install on that system. For example, the most common system tuple for regular users (`client_type == "user"`) over the past month is `x86_64-linux-gnu-libgfortran4-cxx11-libstdcxx26-julia_version+1.6.1`, which indicates:

- `julia_version_prefix`: the Julia version of the client with any build number stripped away. For releases this just the version number, but for people who build Julia from source, version numbers may look like `1.8.0-DEV.485` (my current version number), which would be truncated to `1.8.0-DEV` in rollups.

- `resource_type`: this classifies the types of things clients can request from package servers into:

- `status`: the HTTP status code of the request, where 2xx responses indicate success and other status codes indicate various kinds of failure.

- `package_uuid`: for package requests this is the UUID that identifies the package being requested. When this is one of the key columns, logs are limited to those with `request_type == "package"`.

- `region`: when clients make requests to `pkg.julialang.org` they are redirected to a specific package server that should be close to them geographically, which helps make sure that package downloads are fast no matter where you are in the world. This field identifies region of the package server that served their request. The regions where we currently have package servers are:

- `date`: the date on which a request was made in the UTC timezone.

### Rollup values

Each rollup table is keyed by some subset of the above fields, but what we’re interested in is various aggregated data about the set of request logs that have that set of key field values. These are the aggregates we currently compute for each slice of logs:

- `request_addrs`: the approximate number of unique requesting IP addresses. Details below on why this is approximate and not exact.

- `request_count`: the number of requests.

- `successes`: the number of requests which resulted in a 2xx HTTP response code. Only included if `status` is not one of the key fields of the rollup. To get the success rate, divide by `request_count`.

- `cache_misses`: the number of requests which resulted in the package server attempting to fetch a resource from an upstream storage server. To get the cache miss rate, divide by `request_count`.

- `body_bytes_sent`: total number of bytes served in the bodies of HTTP responses for all requests (i.e. not including HTTP headers or TLS/IP data). To get average request body size, divide by `request_count`.

- `request_time`: total time spent serving these requests. To get average request time, divide by `request_count`.

- `date_count`: the number of distinct UTC dates when requests occurred. Only included if `date` is not one of the key fields of the rollup.

- `date_min`: the earliest date of any request in this group. Only included if `date` is not one of the key fields of the rollup.

- `date_max`: the latest date of any request in this group. Only included if `date` is not one of the key fields of the rollup.

#### Anonymity and unique IP address count approximation

The reported numbers of unique IP addresses for package server requests are approximate because we don’t store IP addresses in our data warehouse. Instead, we store a [HyperLogLog](https://en.wikipedia.org/wiki/HyperLogLog) hash value of each IP address that allows us to accurately approximate the number of unique IP address to within ±1.6% error on average. This technique allows us to accurately estimate the number of unique IP addresses in any set of request logs while preserving quite strong anonymity properties with respect to individual IP addresses:

- No IP address can be uniquely identified by its hash value.

- There are only 102,400 distinct hash values, each less than 17 bits.

- Since there are more than 300k distinct IP addresses that have made requests in the past month, there are already about three IP addresses among the ones we’ve actually seen (which we don’t record) per distinct hash value on average. This ratio will only increase over time since the set of hash values remains the same but we will more IP addresses.

- Most IP addresses have hash values with far more collisions than this: a typical IP address will share a hash value with dozens of IP addresses that we’ve seen and over half a million IPv4 addresses from the full 32-bit space, and over 4×10^34 addresses from the IPv6 address space.

This HyperLogLog hash technique allows us estimate the number of unique IP addresses in any subset of logs while preserving the anonymity of end-users even in our internal data systems. As an extra measure of privacy protection, we do not publish even HyperLogLog hash values, but only the aggregate counts derived from them.

### Examples

Here’s the current contents of the `client_type_by_region` rollup as a gist:

> <https://gist.github.com/StefanKarpinski/217fdf89f621704683dc2df4dc5b1ab8>

Click through for a nice tabular view, courtesy of GitHub. To download and uncompress the latest version of this rollup table yourself, you can run this UNIX command:

```sh
curl -s https://julialang-logs.s3.amazonaws.com/public_outputs/current/client_types_by_region.csv.gz | gzcat

```

But any way you care to download it and uncompress it will work, and after that it’s just a (nicely formatted) CSV file. The key columns of this rollup are `client_type` and `region`. The value columns are all of the possible values columns since neither `status` nor `date` are in set of key columns. Some random observations:

- The US eastern region is the busiest by both number of users (well, IP addresses) and requests for both CI traffic and real user traffic.
- The US western region is the next busiest for CI requests whereas Europe is the next busiest in terms of real user requests.
- India had CI requests for only two days in the last month including a total of only 72 requests from (approximately) 13 IP addresses.

One data set that will no doubt be of particular interest to people is the `package_requests` rollup, which is keyed by `package_uuid`, `status` and `client_type` and includes probably most interestingly, the number of unique request addresses requests with `client_type == "user"`, which is a decent proxy for “how many people use this package?” In the past month, the three most popular packages by this metric were:

- [`Compat`](https://github.com/JuliaLang/Compat.jl) with 37,548 users\*,
- [`Parsers`](https://github.com/JuliaData/Parsers.jl) with 36,438 users\*, and
- [`DataAPI`](https://github.com/JuliaData/DataAPI.jl) with 33,981 users\*.

\*users = “Number of unique IP addresses which requested said package without any indicators of being a CI process.”

### Conclusion

There’s certainly many more interesting things to be gleaned from all this data and we hope that some people will take a look at these data sets and do some interesting analysis, stand up some cool visualization apps, etc. There are also almost certainly more ways to slice and dice our logs, so if anyone has any suggestions for new data sets they’d like to see, please let us know!

---

<div class="post-metadata">

**Author:** ![vlandau](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vlandau/32/8986_2.png) [@vlandau](https://discourse.julialang.org/u/vlandau)\
**Post date:** [October 1, 2021, 9:00pm UTC](https://discourse.julialang.org/t/announcing-package-download-stats/69073/2 "2021-10-01T21:00:40Z")

</div>

This is great! Thank you to all those involved for their work on this.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [October 1, 2021, 9:09pm UTC](https://discourse.julialang.org/t/announcing-package-download-stats/69073/3 "2021-10-01T21:09:11Z")

</div>

This was just myself and @staticfloat which somewhat explains why it took so long 😬 😁. Very glad to be done with it though and we’re now in a position to add more ways to slice the data pretty easily. In the future, we’ll also add historical snapshot rollups for each year in separate files in a `/historical/$year/` directory with the same general schema.

---

<div class="post-metadata">

**Author:** ![logankilpatrick](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/logankilpatrick/32/38123_2.png) [@logankilpatrick](https://discourse.julialang.org/u/logankilpatrick)\
**Post date:** [October 1, 2021, 9:18pm UTC](https://discourse.julialang.org/t/announcing-package-download-stats/69073/4 "2021-10-01T21:18:01Z")

</div>

This is awesome!! Incredible work @StefanKarpinski and @staticfloat.

Quick question: how does one take these stats and create a GitHub badge for a packages README?

---

<div class="post-metadata">

**Author:** ![oschulz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oschulz/32/2998_2.png) [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Post date:** [October 1, 2021, 9:26pm UTC](https://discourse.julialang.org/t/announcing-package-download-stats/69073/5 "2021-10-01T21:26:08Z")

</div>

Will these stats (resp. a short per-package summary) be visible on JuliaHub, eventually?

---

<div class="post-metadata">

**Author:** ![jeremiahpslewis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeremiahpslewis/32/22766_2.png) [@jeremiahpslewis](https://discourse.julialang.org/u/jeremiahpslewis)\
**Post date:** [October 1, 2021, 9:27pm UTC](https://discourse.julialang.org/t/announcing-package-download-stats/69073/6 "2021-10-01T21:27:31Z")

</div>

Love this! Would it be possible to turn the sql scripts / data warehouse definitions into a public repository so that the community can see and collaborate on the definition of these (and future) metrics?

While there is lots to be done with these aggregates, analysis fizzles out over time as high level data gets ‘wrung dry’, there’s probably even more that can be done via different raw → roll up transformations and I’d expect the hive mind might have opinions. 😉

While giving (anyone) access to the raw data is clearly a data privacy risk, the scripts themselves should be harmless? This might then be the seed for generating synthetic data via Soss/Turing generative models which could be incorporated into CI/CD and let more people prototype analysis against non-actual data that can then map onto the real data.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [October 1, 2021, 10:09pm UTC](https://discourse.julialang.org/t/announcing-package-download-stats/69073/7 "2021-10-01T22:09:54Z")

</div>

Yeah, would be great if someone made that happen! Now that the devops bits and the parts that require access to data that we don’t want to publish in unaggregated form are done hopefully others will carry on with stuff like that.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [October 1, 2021, 10:10pm UTC](https://discourse.julialang.org/t/announcing-package-download-stats/69073/8 "2021-10-01T22:10:21Z")

</div>

That would be great, I’m sure that will happen soon.

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [October 1, 2021, 10:17pm UTC](https://discourse.julialang.org/t/announcing-package-download-stats/69073/9 "2021-10-01T22:17:35Z")

</div>

Yes, that’s something I want to do. Currently the code is all in snowflake worksheets. I’d prefer to use something that’s in a git repo, but haven’t figured out a good workflow with snowflake and git yet. But yes, the SQL queries should be public and people can propose new ones that we can add. Having synthetic generated data sets to play with is a very good idea as well!

---

<div class="post-metadata">

**Author:** ![avik](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/avik/32/17_2.png) [@avik](https://discourse.julialang.org/u/avik)\
**Post date:** [October 1, 2021, 10:21pm UTC](https://discourse.julialang.org/t/announcing-package-download-stats/69073/10 "2021-10-01T22:21:20Z")

</div>

This data should be added to the badges that Juliahub currently exposes, soon (ish)

---

<div class="post-metadata">

**Author:** ![oschulz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oschulz/32/2998_2.png) [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Post date:** [October 1, 2021, 10:38pm UTC](https://discourse.julialang.org/t/announcing-package-download-stats/69073/11 "2021-10-01T22:38:24Z")

</div>

I guess we’d want to filter out downloads by GitHub Actions and other CI systems, if possible (based on IP)?

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [October 2, 2021, 1:18am UTC](https://discourse.julialang.org/t/announcing-package-download-stats/69073/12 "2021-10-02T01:18:18Z")

</div>

> [@oschulz](#):
>
> filter out downloads by GitHub Actions and other CI systems

> [@StefanKarpinski](#):
>
> without any indicators of being a CI process.”

I believe this has been done to some extent?

---

<div class="post-metadata">

**Author:** ![StefanKarpinski](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/stefankarpinski/32/24_2.png) [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)\
**Post date:** [October 2, 2021, 1:29am UTC](https://discourse.julialang.org/t/announcing-package-download-stats/69073/13 "2021-10-02T01:29:37Z")

</div>

Yes, the Julia client sends indicators of whether environment variables that are set by CI services are present or not and we check for those to decide if something is a CI request.

---

<div class="post-metadata">

**Author:** ![jeremiahpslewis](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jeremiahpslewis/32/22766_2.png) [@jeremiahpslewis](https://discourse.julialang.org/u/jeremiahpslewis)\
**Post date:** [October 2, 2021, 6:25am UTC](https://discourse.julialang.org/t/announcing-package-download-stats/69073/14 "2021-10-02T06:25:47Z")

</div>

Not sure if this is relevant as I haven’t used snowflake previously, but for Postgres and Redshift data proceeding, I’ve found dbt a brilliant tool for structuring and executing query based data pipelines.

> **[What is dbt? | dbt Developer Hub](https://docs.getdbt.com/docs/introduction)**
>
> dbt compiles and runs your analytics code against your data platform, enabling you and your team to collaborate on a single source of truth for metrics, insights, and business definitions. This single source of truth, combined with the ability to...

> **[Snowflake configurations | dbt Developer Hub](https://docs.getdbt.com/reference/resource-configs/snowflake-configs)**
>
> Snowflake Configurations - Read this in-depth guide to learn about configurations in dbt.

---

<div class="post-metadata">

**Author:** ![baggepinnen](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/baggepinnen/32/693_2.png) [@baggepinnen](https://discourse.julialang.org/u/baggepinnen)\
**Post date:** [October 2, 2021, 7:12am UTC](https://discourse.julialang.org/t/announcing-package-download-stats/69073/15 "2021-10-02T07:12:59Z")

</div>

Here’s the data for last month with UUIDs converted into package names if anyone is curious and don’t feel like wrangling the data themselves  
[https://gist.github.com/baggepinnen/e05e8dc6989980f558691d9ce6771801](https://gist.github.com/baggepinnen/e05e8dc6989980f558691d9ce6771801)

The first columns are sorted based on number of users and the last columns based on organization name.  
The script use to produce the CSV file is here  
[https://gist.github.com/baggepinnen/b98625dbfd2a7194a744f90926ef6f66](https://gist.github.com/baggepinnen/b98625dbfd2a7194a744f90926ef6f66)

---

<div class="post-metadata">

**Author:** ![giordano](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/giordano/32/2166_2.png) [@giordano](https://discourse.julialang.org/u/giordano)\
**Post date:** [October 2, 2021, 7:38am UTC](https://discourse.julialang.org/t/announcing-package-download-stats/69073/16 "2021-10-02T07:38:15Z")

</div>

> [@StefanKarpinski](#):
>
> \*users = “Number of unique IP addresses which requested said package without any indicators of being a CI process.”

To be clear, these numbers refer to unique IPs installing the package within the time window, right? So stable packages with unfrequent releases won’t be (re)downloaded very often by regular users, and users on dynamic IPs may be counted multiple times, correct?

---

<div class="post-metadata">

**Author:** ![ppalmes](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ppalmes/32/6838_2.png) [@ppalmes](https://discourse.julialang.org/u/ppalmes)\
**Post date:** [October 2, 2021, 8:00am UTC](https://discourse.julialang.org/t/announcing-package-download-stats/69073/17 "2021-10-02T08:00:36Z")

</div>

this is cool. here’s my suggestions:

- create a github repo with FAQ, Docs, etc and where everyone can contribute their scripts or codes to extract, analyze, cluster, predict, visualize the data
- have a list of questions we want to know given the datasets so that some of these scripts who have solutions can be associated with these questions
- include a bof or workshop in the next juliacon related to these activities
- have a slack/zulips/discourse channel for anything about these datasets for brainstorming and follow-up discussion about insights on the data

---

<div class="post-metadata">

**Author:** ![pfitzseb](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pfitzseb/32/45566_2.png) [@pfitzseb](https://discourse.julialang.org/u/pfitzseb)\
**Post date:** [October 2, 2021, 9:21am UTC](https://discourse.julialang.org/t/announcing-package-download-stats/69073/18 "2021-10-02T09:21:44Z")

</div>

> [@giordano](#):
>
> To be clear, these numbers refer to unique IPs installing the package within the time window, right?

Yes, IIUC. So in theory one could just tag a new patch release to estimate the number of “unique active users”.

---

<div class="post-metadata">

**Author:** ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)\
**Post date:** [October 2, 2021, 9:58am UTC](https://discourse.julialang.org/t/announcing-package-download-stats/69073/19 "2021-10-02T09:58:01Z")

</div>

This is great. Is the content of the original post going to be put somewhere more permanent and updateable as new views of the data come along?

---

<div class="post-metadata">

**Author:** ![oschulz](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oschulz/32/2998_2.png) [@oschulz](https://discourse.julialang.org/u/oschulz)\
**Post date:** [October 2, 2021, 10:20am UTC](https://discourse.julialang.org/t/announcing-package-download-stats/69073/20 "2021-10-02T10:20:11Z")

</div>

> [@jling](#):
>
> > [@StefanKarpinski](#):
> >
> > without any indicators of being a CI process.”
> 
> I believe this has been done to some extent?

Ah, sorry, I did overlook that. Nice!

[Next page](https://discourse.julialang.org/t/announcing-package-download-stats/69073.md?page=2)