# “large” scale software in Julia

**URL:** <https://discourse.julialang.org/t/large-scale-software-in-julia/29895>\
**Category:** New to Julia\
**Tags:** development, code-organization\
**Created:** [October 14, 2019, 11:20am UTC](https://discourse.julialang.org/t/large-scale-software-in-julia/29895 "2019-10-14T11:20:55Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![DK1PL](https://avatars.discourse-cdn.com/v4/letter/d/5f9b8f/32.png) [@DK1PL](https://discourse.julialang.org/u/DK1PL)\
**Post date:** [October 14, 2019, 11:20am UTC](https://discourse.julialang.org/t/large-scale-software-in-julia/29895/1 "2019-10-14T11:20:55Z")

</div>

Hi,

I’m new to Julia and I’m learning Julia working on small private project for the HF antenna design but if I’m thinking about the “large” scale software then I don’t see currently any way to do this in Julia. For example one of our libraries, my daily work, is about 1.16 x 10^6 of C and 250 x 10^3 of C++ lines of code. Even if I assume that Julia source code is much shorter that C/C++ code then potential new source code will be large. How is your experience with large scale software packages in Julia?

Regards  
Christoph

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [October 14, 2019, 11:27am UTC](https://discourse.julialang.org/t/large-scale-software-in-julia/29895/2 "2019-10-14T11:27:06Z")

</div>

> [@DK1PL](#):
>
> I don’t see currently any way to do this in Julia

It’s not clear why you think this. There is no limit on source code size in Julia either.

> [@DK1PL](#):
>
> one of our libraries, my daily work, is about 1.16 x 10^6 of C and 250 x 10^3 of C++ lines of code

To ease the transition, maybe you reuse most of this first. See

[https://docs.julialang.org/en/v1/manual/calling-c-and-fortran-code/](https://docs.julialang.org/en/v1/manual/calling-c-and-fortran-code/)

> [@DK1PL](#):
>
> experience with large scale software packages in Julia?

A public example of a large-scale project is

> **[GitHub - jeff-regier/Celeste.jl: Scalable inference for a generative model of...](https://github.com/jeff-regier/Celeste.jl)**
>
> Scalable inference for a generative model of astronomical images - GitHub - jeff-regier/Celeste.jl: Scalable inference for a generative model of astronomical images

You may also be interested in discussions like

> [@Large programs: structuring modules & include such that to increase performance and readability](https://discourse.julialang.org/t/large-programs-structuring-modules-include-such-that-to-increase-performance-and-readability/29102):
>
> I am writing a large hydrological model for which I require to divide the program in many modules incorporated in individual files. I am questioning what will be an efficient and readable approach to share the modules across the numerous files? I found that for e.g. Include(“Filename.jl”) in every module is clean but not the fastest especially when I need to perform loops (which will call include many times.) I found that an efficient way is to include(FileName) in the MAIN() program and th…

---

<div class="post-metadata">

**Author:** ![DK1PL](https://avatars.discourse-cdn.com/v4/letter/d/5f9b8f/32.png) [@DK1PL](https://discourse.julialang.org/u/DK1PL)\
**Post date:** [October 14, 2019, 12:01pm UTC](https://discourse.julialang.org/t/large-scale-software-in-julia/29895/3 "2019-10-14T12:01:22Z")

</div>

Hi,

it is not a simple question of source code size limits but more the question of source code organization, build and test at the daily work. Our software package (library) is spited into 20 subpackages and the makefiles manage the build on the differed abstraction levels. Because our development is test driven I have a set of 6000 test case which should be check before a single push into the repository. Integration and regression tests are on the top of the 6000 “basic” test cases.

I have a look into jeff-regier/Celeste.jl but this “only” 16k line of code package with few test cases – so far I can recognized this.

Regards  
Christoph

---

<div class="post-metadata">

**Author:** ![cortner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cortner/32/204_2.png) [@cortner](https://discourse.julialang.org/u/cortner)\
**Post date:** [October 14, 2019, 12:12pm UTC](https://discourse.julialang.org/t/large-scale-software-in-julia/29895/4 "2019-10-14T12:12:01Z")

</div>

I’ve recently switched to managing my own registry. This makes it surprisingly easy to manage multiple interdependent specialised packages and controlling compatibility between them. Most Julia packages use test driven development, there is nothing different here. If you want to be cautious - first pass all tests locally, then push, then the tests run again on Travis in multiple environments. And only when you are satisfied with both layers of testing you bump the version number. If you start a new “project” based on your package eco-system you specify which compatible versions of your packages are allowed. I’m not a software developer by any measure, but I’ve found the tools available in the Julia eco-system extremely helpful to me in these regards. It may be worth reading

- [https://github.com/JuliaLang/Pkg.jl](https://github.com/JuliaLang/Pkg.jl)
- [https://github.com/GunnarFarneback/Registrator.jl](https://github.com/GunnarFarneback/Registrator.jl)
- [https://lhendricks.org/julia\_notes.pdf](https://lhendricks.org/julia_notes.pdf)  
(links from discussion in [General vs Custom Registries](https://discourse.julialang.org/t/general-vs-custom-registries/28687))

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [October 14, 2019, 12:31pm UTC](https://discourse.julialang.org/t/large-scale-software-in-julia/29895/5 "2019-10-14T12:31:55Z")

</div>

TDD is pretty much the norm for Julia packages too, usually set up on Travis for open source projects (but for a large, non-OSS project, you may want to get a paid plan, or set up your own CI environment to get faster turnaround).

Splitting code into multiple packages is also very common in Julia. A nice example is the DiffEq ecosystem:

[https://github.com/JuliaDiffEq/](https://github.com/JuliaDiffEq/)

It is still not clear to me why you imagine Julia could pose problems for your setup. Everything you are asking for is available and commonly used.

If you are seriously considering Julia, I would recommend programming in it for a few months to get a feel for the language and the tooling. Then you should be in a position to make a good decision.

---

<div class="post-metadata">

**Author:** ![DK1PL](https://avatars.discourse-cdn.com/v4/letter/d/5f9b8f/32.png) [@DK1PL](https://discourse.julialang.org/u/DK1PL)\
**Post date:** [October 15, 2019, 6:21am UTC](https://discourse.julialang.org/t/large-scale-software-in-julia/29895/6 "2019-10-15T06:21:37Z")

</div>

> [@Tamas\_Papp](#):
>
> If you are seriously considering Julia, I would recommend programming in it for a few months to get a feel for the language and the tooling.

It this what I’m currently doing. I started a small “test” project for the hf antenna design and this will take awhile to finish it. I’m focusing myself in this project on using Julia and functional programing. At the moment my test program (package) is at about 1000 lines of code spited into 3 modules and 10 files. As you see is is small at the moment.

Because my daily job is a large scale C/C++ software I’ve been thinking if I could write this kind of software in Julia. From my point of view isn’t practicable to compile from scratch the full source code by every testrun and one of the solutions could be the incremental compilation. At the moment I’m looking for the way to precompile a part of my source code and this is the main reason of my post in this group.

Regards  
Christoph

---

<div class="post-metadata">

**Author:** ![mohamed82008](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mohamed82008/32/18171_2.png) [@mohamed82008](https://discourse.julialang.org/u/mohamed82008)\
**Post date:** [October 15, 2019, 6:39am UTC](https://discourse.julialang.org/t/large-scale-software-in-julia/29895/7 "2019-10-15T06:39:11Z")

</div>

> [@DK1PL](#):
>
> At the moment I’m looking for the way to precompile a part of my source code and this is the main reason of my post in this group.

Break it into modules. Let each module be a fully tested unit with clear APIs so that one module doesn’t need to change when changing or developing other modules. It might take a few iterations to figure out a good splitting, so start with sub-modules then when compile or test times get too large, split them. Also check out Revise.jl which will only recompile the parts changed when developing so you can run your tests in a single Julia session without the need to recompile the whole package again and again. Perhaps other people have better advice.