# Package size crazziness

**URL:** <https://discourse.julialang.org/t/package-size-crazziness/14644>\
**Category:** General Usage\
**Created:** [September 6, 2018, 9:43pm UTC](https://discourse.julialang.org/t/package-size-crazziness/14644 "2018-09-06T21:43:42Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [September 6, 2018, 9:43pm UTC](https://discourse.julialang.org/t/package-size-crazziness/14644/1 "2018-09-06T21:43:42Z")

</div>

_Prelude I_. I do everything in my laptop where disk size is far from infinite (and cost for SSD disk are not low).

_Prelude II_. I work on Windows

I made an exercise to confirm one suspicion about the size of packages installed with dependencies dealt by the BinaryBuilder. For that I installed the [GDAL](https://github.com/JuliaGeo/GDAL.jl), which is a program that I happen to know a bit off (I mean, the C lib).

With not much surprise my suspicious were confirmed. After installing the package its disk usage was 500 Mb. A fifth of it (~100 Mb) are due to installing files (\*.tar.gz) that are not removed after being uncompressed. But the largest chunk of the total size are to blame to mingw build. I do build GDAL with Visual Studio and the total size is about 25 Mb. So from ~25 to ~400 Mb (the total 500 less the non deleted ~100 Mb tar files) goes an awful difference.

Don’t know the solution to this, but to pretend that mingw builds are a good replacement to VS on Windows … guys take care.

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [September 6, 2018, 10:03pm UTC](https://discourse.julialang.org/t/package-size-crazziness/14644/2 "2018-09-06T22:03:04Z")

</div>

I have to admit, the tone of your post really bothers me. In particular,

> Don’t know the solution to this, but to pretend that mingw builds are a good replacement to VS on Windows … guys take care.

seems _extremely_ dismissive (and rude) to the people who are working hard to provide a working ecosystem. No one is “pretending” anything. Binary packaging is difficult and completely thankless work, and providing working cross-platform binaries is even more so.

None of that is to suggest that you can’t complain about it. Just please remember that this is a best effort from people who care about getting it right.

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [September 6, 2018, 10:08pm UTC](https://discourse.julialang.org/t/package-size-crazziness/14644/3 "2018-09-06T22:08:25Z")

</div>

I apologize if that was the impression that I passed. Really, no mean to be rude or neglect the work of others. But I feel that this aspect of the things on Windows tends to be easily overlooked.

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [September 6, 2018, 10:12pm UTC](https://discourse.julialang.org/t/package-size-crazziness/14644/4 "2018-09-06T22:12:16Z")

</div>

No worries 🙂

You might have more luck over in the `#bindeps2` channel on Slack, which seems to be pretty lively. Removing the extra downloaded tarfiles seems like an easy win, although switching from a `mingw` build to VS sounds a lot scarier. Is storage space the only reason to prefer the VS build? I realize laptops are pretty constrained, but, to be fair, 500 Mb worth of SSD costs about $0.09 USD ([e.g.](https://www.amazon.com/Samsung-Inch-Internal-MZ-76E1T0B-AM/dp/B078DPCY3T/ref=sr_1_1?s=pc&ie=UTF8&qid=1536270791&sr=1-1&keywords=1tb+ssd&dpID=41qR7C253KL&preST=_SX300_QL70_&dpSrc=srch)) these days so it’s not as bad as it once was.

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [September 6, 2018, 10:24pm UTC](https://discourse.julialang.org/t/package-size-crazziness/14644/5 "2018-09-06T22:24:15Z")

</div>

> [@rdeits](#):
>
> Is storage space the only reason to prefer the VS build?

For my own builds a far more important reason is debugging. VS builds are a breeze to debug whilst mingw … I don’t even try.

But don’t minimize the size issue. Of course 500 Mb is a nothing but multiplied by many packages. And the next time I upgrade my disk size, it will come with a new laptop attached (my current one is 2014 MacBook Pro, which was not cheap at all).

BTW, did you know that if one get distracted and let one package that depends on Conda (and have not set the right env var in case one already have python installed) will get a nice 1.7 Gb Conda installation with tons of things that will never be used by that package?

---

<div class="post-metadata">

**Author:** ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)\
**Post date:** [September 6, 2018, 10:26pm UTC](https://discourse.julialang.org/t/package-size-crazziness/14644/6 "2018-09-06T22:26:51Z")

</div>

Oof, yeah, that’s a lot.

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [September 7, 2018, 5:54am UTC](https://discourse.julialang.org/t/package-size-crazziness/14644/7 "2018-09-07T05:54:08Z")

</div>

> [@joa-quim](#):
>
> I work on Windows […] Don’t know the solution to this

That’s easy, install Linux 😉

---

<div class="post-metadata">

**Author:** ![tk3369](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tk3369/32/2824_2.png) [@tk3369](https://discourse.julialang.org/u/tk3369)\
**Post date:** [September 7, 2018, 6:27am UTC](https://discourse.julialang.org/t/package-size-crazziness/14644/8 "2018-09-07T06:27:12Z")

</div>

It could be easier on Windows 10, use docker to run julia.

---

<div class="post-metadata">

**Author:** ![tamasgal](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamasgal/32/27946_2.png) [@tamasgal](https://discourse.julialang.org/u/tamasgal)\
**Post date:** [September 7, 2018, 6:55am UTC](https://discourse.julialang.org/t/package-size-crazziness/14644/9 "2018-09-07T06:55:15Z")

</div>

Given the fact that VS is requiring:

> - Hard disk space: up to 130 GB of available space, depending on features installed; typical installations require 20-50 GB of free space.

the 400MB are not too bad of a compromise to start with 😉

I mean, if you require VS as a hard dependency, we will end up with users complaining about the 20-50GB. There are people who don’t use/depend\_on VS 🙂

---

<div class="post-metadata">

**Author:** ![visr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/visr/32/17204_2.png) [@visr](https://discourse.julialang.org/u/visr)\
**Post date:** [September 7, 2018, 7:00am UTC](https://discourse.julialang.org/t/package-size-crazziness/14644/10 "2018-09-07T07:00:55Z")

</div>

Yeah I would love it if we could get the GDAL deps folder down from 500MB on Windows. Maybe simple additions like stripping binaries might help? Any help/tips on how to do that are appreciated on [GDALBuilder](https://github.com/JuliaGeo/GDALBuilder), (or BinaryBuilder docs).

One thing that will shave off about 20% is no longer having to include the dependencies. I believe that’s on the BinaryBuilder roadmap to handle that.

If a new BinaryBuilder release is out I will try my hand at GDAL 2.3 as well.

---

<div class="post-metadata">

**Author:** ![visr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/visr/32/17204_2.png) [@visr](https://discourse.julialang.org/u/visr)\
**Post date:** [September 7, 2018, 7:18am UTC](https://discourse.julialang.org/t/package-size-crazziness/14644/11 "2018-09-07T07:18:36Z")

</div>

I see the sizes are actually more or less comparable across the different operating systems: [Releases · JuliaGeo/GDALBuilder · GitHub](https://github.com/JuliaGeo/GDALBuilder/releases).

So no need to turn this thread into a Windows vs others thread. But general tips on how to reduce the size of builds will be welcome. This will benefit all. Especially those that don’t have the fortune of high bandwith internet and/or large storage devices.

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [September 7, 2018, 10:34am UTC](https://discourse.julialang.org/t/package-size-crazziness/14644/12 "2018-09-07T10:34:38Z")

</div>

Ok, I did not check the \*nix sizes, but the Windows can be clearly smaller. Reducing the dependencies is not a good decision. Without its dependencies (namely netCDF and HDF) GDAL has a much reduced utility. [This site](http://www.gisinternals.com/release.php) has many Win builds with a lot of GDAL dependencies and the zip file is still about 40 Mb.

But my issue with this type of solutions is that they install the packages (at least on Win) in a place that only Julia knows about, so they wont be useful for use outside of Julia. And, it doesn’t use the fact that the binary dependencies may already be installed in the system.

Sorry to pick up GDAL for this discussion, but as I said in other post this is one program that I know better.

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [September 7, 2018, 10:40am UTC](https://discourse.julialang.org/t/package-size-crazziness/14644/13 "2018-09-07T10:40:13Z")

</div>

> [@tamasgal](#):
>
> There are people who don’t use/depend\_on VS

Ho said anything related to requiring a VS installation (who, though big, 3-4 Gb, is still far from 50-50Gb))?

---

<div class="post-metadata">

**Author:** ![visr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/visr/32/17204_2.png) [@visr](https://discourse.julialang.org/u/visr)\
**Post date:** [September 7, 2018, 11:13am UTC](https://discourse.julialang.org/t/package-size-crazziness/14644/14 "2018-09-07T11:13:20Z")

</div>

> [@joa-quim](#):
>
> Reducing the dependencies is not a good decision.

Sorry I don’t mean getting rid of them and their functionality. But currently if you add `LibGEOS.jl` and `GDAL.jl`, it will install it twice. Soon it should be possible to share the installation within julia.

> [@joa-quim](#):
>
> my issue with this type of solutions is that they install the packages (at least on Win) in a place that only Julia knows about, so they wont be useful for use outside of Julia

That’s true. But as a `GDAL.jl` author things will be much simpler for me knowing exactly what binaries users are getting, and knowing that they will essentially be compiled the same across platforms. Not the most efficient space wise, but if setup well, things will “just work”, and that’s worth a lot.

> [@joa-quim](#):
>
> Sorry to pick up GDAL for this discussion

No it’s a good example, and you are right that the VS binaries are much smaller. I think it is likely that this is nothing fundamental though, and we can reduce this.

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [September 7, 2018, 11:26am UTC](https://discourse.julialang.org/t/package-size-crazziness/14644/15 "2018-09-07T11:26:47Z")

</div>

> [@visr](#):
>
> That’s true. But as a `GDAL.jl` author things will be much simpler for me knowing exactly what binaries users are getting, and knowing that they will essentially be compiled the same across platforms. Not the most efficient space wise, but if setup well, things will “just work”, and that’s worth a lot.

I understand that perfectly. In fact I’m faced with the same issue for GMT, but can’t the build system try to detect if a GDAL is already installed (for instance if `gdalinfo` runs ok)? The names of the GDAL shared lib change with versions but at least on \*nix that is a symlink whose name does not change. And in case it is, than no need to install the GDAL.jl own dependencies.

---

<div class="post-metadata">

**Author:** ![visr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/visr/32/17204_2.png) [@visr](https://discourse.julialang.org/u/visr)\
**Post date:** [September 7, 2018, 11:36am UTC](https://discourse.julialang.org/t/package-size-crazziness/14644/16 "2018-09-07T11:36:50Z")

</div>

If we can add that as a non-default option I would be for. But besides testing if `gdalinfo` works, I also mean that I can now make assumptions on which version is installed, which formats are available, etc. If this would be the default behavior we probably get issues from users that had it working, then uninstalled GDAL, not knowing GDAL.jl relied on it, and have GDAL.jl now no longer working either.

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [September 7, 2018, 11:51am UTC](https://discourse.julialang.org/t/package-size-crazziness/14644/17 "2018-09-07T11:51:35Z")

</div>

`gdalinfo --version` and `gdalinfo --formats` would tell you that. But you are right about a user uninstalling GDAL, but for me people have to have a minimum knowledge of what they are doing.

---

<div class="post-metadata">

**Author:** ![tshort](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tshort/32/43_2.png) [@tshort](https://discourse.julialang.org/u/tshort)\
**Post date:** [September 7, 2018, 12:04pm UTC](https://discourse.julialang.org/t/package-size-crazziness/14644/18 "2018-09-07T12:04:11Z")

</div>

In the Win64 GDAL library built by GDALBuilder, an easy way to trim size is to remove `lib/libgdal.a` (159 MB out of 314 MB).

---

<div class="post-metadata">

**Author:** ![visr](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/visr/32/17204_2.png) [@visr](https://discourse.julialang.org/u/visr)\
**Post date:** [September 7, 2018, 12:17pm UTC](https://discourse.julialang.org/t/package-size-crazziness/14644/19 "2018-09-07T12:17:38Z")

</div>

> [@joa-quim](#):
>
> `gdalinfo --version` and `gdalinfo --formats` would tell you that.

True but that then puts an extra burden on the GDAL.jl developer to do these checks. Unless somebody else is willing to put in the work to support different GDAL installations in GDAL.jl, I would prefer to make it possible to “bring your own GDAL”, but then you are on your own on what works and what doesn’t.

> [@joa-quim](#):
>
> people have to have a minimum knowledge of what they are doing

I spent too much time already fixing broken GDAL installations on other peoples computers, to get something to work. Most of them don’t know/don’t care what GDAL is.

---

<div class="post-metadata">

**Author:** ![joa-quim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/joa-quim/32/227_2.png) [@joa-quim](https://discourse.julialang.org/u/joa-quim)\
**Post date:** [September 7, 2018, 12:36pm UTC](https://discourse.julialang.org/t/package-size-crazziness/14644/20 "2018-09-07T12:36:29Z")

</div>

> [@visr](#):
>
> True but that then puts an extra burden on the GDAL.jl developer to do these checks.

Very true.

The “bring you own GDAL” would have another potential (big) gain, which is the number of drivers. Currently you only have GEOS and PROJ, but not netCDF, HDF5, etc… However, since you say that you rely on knowing what formats are available I don’t know if this would really be a benefit.

[Next page](https://discourse.julialang.org/t/package-size-crazziness/14644.md?page=2)