# Julia as a universal platform for statistical software development

**URL:** <https://discourse.julialang.org/t/julia-as-a-universal-platform-for-statistical-software-development/113017>\
**Category:** Community\
**Tags:** announcement\
**Created:** [April 16, 2024, 2:11pm UTC](https://discourse.julialang.org/t/julia-as-a-universal-platform-for-statistical-software-development/113017 "2024-04-16T14:11:28Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![droodman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/droodman/32/26652_2.png) [@droodman](https://discourse.julialang.org/u/droodman)\
**Post date:** [April 16, 2024, 2:11pm UTC](https://discourse.julialang.org/t/julia-as-a-universal-platform-for-statistical-software-development/113017/1 "2024-04-16T14:11:28Z")

</div>

I just posted [a paper](https://arxiv.org/abs/2404.09309) presenting a vision of using Julia as a universal back end for statistical software development. Do things once in Julia–optimize, maintain, add features–then call from R or Python or Matlab or Stata.

The motivating example is my pair of packages for Stata, which is a commercial statistics program that is popular in the social sciences and elsewhere. `julia` provides a C-based bridge from Stata to Julia, like JuliaCall for R. It has routines for fast data copying and provides an ersatz Julia REPL inside of Stata. Running on top of it is `reghdfejl` which mimics a popular Stata program, `reghdfe`, for fitting linear models with many fixed-effect dummies. The new version presents nearly the same user and programmatic interface but runs ~10X faster on hard problems, by calling FixedEffectModels.jl.

I also present the example of WildBootTests.jl, which is used as a back end for both a Stata and an R package.

I think it will be tough for Julia catch up with R and the like as a home for end users doing statistical analysis. But I think its a great environment for the core work of implementing numerical methods. Why code things separately for R, Python, etc., when we can just do it once, in Julia?

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [April 16, 2024, 4:30pm UTC](https://discourse.julialang.org/t/julia-as-a-universal-platform-for-statistical-software-development/113017/2 "2024-04-16T16:30:11Z")

</div>

That’s brilliant, thanks for sharing.

I’ve come to the same conclusion independently, as the (pretty much sole right now unfortunately) author of SynthControl.jl I’ve gone through various recent synthetic control method implementations in R, Stata, Matlab, C++ (or Rcpp) and - biased as I am - am convinced that Julia is the natural home for the development of these methods which are often pushed forward by domain experts who prefer coding in interactive languages, but have a computational complexity that means writing a loop in R just isn’t an option.

Unfortunately I think statistics is one of the biggest laggards in Julia adoption since I started using Julia (~2013), with most applied econometrics people seemingly only considering R when looking outside of Stata.

As usual there’s not much use whining about it though, we can just continue to build and do good work, so thanks for doing what you’re doing!

---

<div class="post-metadata">

**Author:** ![Paul\_Soderlind](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/paul_soderlind/32/1753_2.png) [@Paul\_Soderlind](https://discourse.julialang.org/u/Paul_Soderlind)\
**Post date:** [April 16, 2024, 6:51pm UTC](https://discourse.julialang.org/t/julia-as-a-universal-platform-for-statistical-software-development/113017/3 "2024-04-16T18:51:54Z")

</div>

Looks interesting. (I’ll have a careful look tomorrow.)

While I havn’t published any econometrics packages, [my financial econometrics tutorial](https://github.com/PaulSoderlind/FinancialEconometrics) (using a module of functions) isn’t too far from one. Maybe of some interest.

---

<div class="post-metadata">

**Author:** ![biona001](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/biona001/32/16497_2.png) [@biona001](https://discourse.julialang.org/u/biona001)\
**Post date:** [April 16, 2024, 11:44pm UTC](https://discourse.julialang.org/t/julia-as-a-universal-platform-for-statistical-software-development/113017/4 "2024-04-16T23:44:03Z")

</div>

> [@droodman](#):
>
> Do things once in Julia–optimize, maintain, add features–then call from R or Python or Matlab or Stata.

I tried this approach - writing [R](https://github.com/biona001/knockoffsr) an [python](https://github.com/biona001/knockoffspy) wrappers for one of my julia package (following example of `DifferentialEquations.jl`). But I feel this is currently not viable. For example, `JuliaCall` (which calls Julia from R) appears to be abandoned, and even basic [installation issues](https://github.com/Non-Contradiction/JuliaCall/issues/198) will likely never get resolved. As a result, half the time my collaborators cannot successfully install these wrappers without my help. Also, I feel our community did not documented well how to write your Julia code to make wrapping in R/Python/whatever easy, and I actually had to re-structure a lot of my originally working Julia code to make the wrappers work.

I also thought it’ll be cool to suggest R/Julia rather than R/C++, but in practice, I feel this is more of a dream than reality.

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [April 17, 2024, 1:51am UTC](https://discourse.julialang.org/t/julia-as-a-universal-platform-for-statistical-software-development/113017/5 "2024-04-17T01:51:59Z")

</div>

> [@biona001](#):
>
> For example, `JuliaCall` (which calls Julia from R) appears to be abandoned, and even basic [installation issues](https://github.com/Non-Contradiction/JuliaCall/issues/198) will likely never get resolved.

This seems to be more about an annoying Apple macOS “feature” than anything else.

[https://discussions.apple.com/thread/253714860?sortBy=best](https://discussions.apple.com/thread/253714860?sortBy=best)

---

<div class="post-metadata">

**Author:** ![dilumaluthge](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dilumaluthge/32/29283_2.png) [@dilumaluthge](https://discourse.julialang.org/u/dilumaluthge)\
**Post date:** [April 17, 2024, 2:48am UTC](https://discourse.julialang.org/t/julia-as-a-universal-platform-for-statistical-software-development/113017/6 "2024-04-17T02:48:40Z")

</div>

It looks like JuliaCall is downloading the `.dmg` version of Julia on macOS. They could try downloading the `.tar.gz` instead, which might resolve that one specific issue.

Edit: [Install Julia from tarball on macOS by DilumAluthge · Pull Request #227 · Non-Contradiction/JuliaCall · GitHub](https://github.com/Non-Contradiction/JuliaCall/pull/227)

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [April 17, 2024, 3:45am UTC](https://discourse.julialang.org/t/julia-as-a-universal-platform-for-statistical-software-development/113017/7 "2024-04-17T03:45:46Z")

</div>

@Non-Contradiction does seem to quite busy as of late. Last commit there was nine months ago. Maybe @kdpsingh 's group might have an interest in continuing maintenance?

Returning to the original topic, we did have to work through some issues with embedding Julia so there are still some rough points. That said, it was not incredibly difficult either.

> [@How to embed Julia in interactive mode](https://discourse.julialang.org/t/how-to-embed-julia-in-interactive-mode/112264/1):
>
> Another in my continuing series of questions about how to embed Julia… @mkitti and @GunnarFarneback have been very helpful in the past. I am embedding Julia in another program and am trying to approximately simulate the REPL. To this end I would like to launch Julia in interactive mode. My understanding is that this would affect, for example, the behavior of [soft scoping](https://docs.julialang.org/en/v1/manual/variables-and-scoping/#local-scope). However, when I launch it after passing the -i option, isinteractive() still returns false. Here’s my code, in Windows: #…

---

<div class="post-metadata">

**Author:** ![Alexander\_Knudson](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alexander_knudson/32/215656_2.png) [@Alexander\_Knudson](https://discourse.julialang.org/u/Alexander_Knudson)\
**Post date:** [April 17, 2024, 4:42am UTC](https://discourse.julialang.org/t/julia-as-a-universal-platform-for-statistical-software-development/113017/8 "2024-04-17T04:42:24Z")

</div>

The package I helped develop, Bigsimr.jl, is another example of a “backend” for R and Python, and it was developed with R in mind such that passing in real-valued scalars or vectors just works like in R. The thing to keep in mind is that in R everything is a vector and all\* numbers are doubles (even integers unless explicitly cast as an integer).

---

<div class="post-metadata">

**Author:** ![lrnv](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lrnv/32/19373_2.png) [@lrnv](https://discourse.julialang.org/u/lrnv)\
**Post date:** [April 17, 2024, 7:37am UTC](https://discourse.julialang.org/t/julia-as-a-universal-platform-for-statistical-software-development/113017/9 "2024-04-17T07:37:21Z")

</div>

> [@biona001](#):
>
> I tried this approach - writing [R](https://github.com/biona001/knockoffsr) an [python](https://github.com/biona001/knockoffspy) wrappers for one of my julia package (following example of `DifferentialEquations.jl`). But I feel this is currently not viable. For example, `JuliaCall` (which calls Julia from R) appears to be abandoned, and even basic [installation issues](https://github.com/Non-Contradiction/JuliaCall/issues/198) will likely never get resolved. As a result, half the time my collaborators cannot successfully install these wrappers without my help. Also, I feel our community did not documented well how to write your Julia code to make wrapping in R/Python/whatever easy, and I actually had to re-structure a lot of my originally working Julia code to make the wrappers work.
> 
> I also thought it’ll be cool to suggest R/Julia rather than R/C++, but in practice, I feel this is more of a dream than reality.

Proof-reading myself I was basically saying exactly the same thing as you, so let me scratch what i wrote and simply +1 your post.

---

<div class="post-metadata">

**Author:** ![dmbates](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dmbates/32/44_2.png) [@dmbates](https://discourse.julialang.org/u/dmbates)\
**Post date:** [April 17, 2024, 4:09pm UTC](https://discourse.julialang.org/t/julia-as-a-universal-platform-for-statistical-software-development/113017/10 "2024-04-17T16:09:07Z")

</div>

It is much, much easier to communicate to an inferior R process from a Julia process than the other way around. The RCall.jl package is 100% Julia code which calls the C API for R and models the R data structures in Julia. Given that all the internal data structures in R are based on a C struct called SEXPREC (which is a union struct) and most of the C API for R passes and receives such pointers to such structs (called SEXP), it is reasonably straightforward to model these structs as Julia types and to interface with R’s C API.

To try to work in the other direction, starting an inferior Julia process from an R process and communicating with it from R, is much more difficult. You need to write the glue code in C/C++ and even with that I think the JuliaCall package for R still requires RCall.jl to be installed on the Julia side for some of the interfacing of internal types.

The `@R_str` macro and the “R mode” toggle for the Julia REPL provide the ability to do something like writing R code in the Julia REPL.

I know it is more difficult to convince users to start the Julia REPL and connect to R than to start the R REPL but I still think it is worthwhile encouraging them to do so. Having said that, I will note that I haven’t been as successful convincing users of the lme4 package to R to migrate to the MixedModels.jl package for Julia as I had hoped.

---

<div class="post-metadata">

**Author:** ![droodman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/droodman/32/26652_2.png) [@droodman](https://discourse.julialang.org/u/droodman)\
**Post date:** [April 17, 2024, 8:39pm UTC](https://discourse.julialang.org/t/julia-as-a-universal-platform-for-statistical-software-development/113017/11 "2024-04-17T20:39:44Z")

</div>

> [@biona001](#):
>
> I tried this approach - writing [R](https://github.com/biona001/knockoffsr) an [python](https://github.com/biona001/knockoffspy) wrappers for one of my julia package (following example of `DifferentialEquations.jl`). But I feel this is currently not viable. For example, `JuliaCall` (which calls Julia from R) appears to be abandoned, and even basic [installation issues](https://github.com/Non-Contradiction/JuliaCall/issues/198) will likely never get resolved.

Does JuliaConnectoR work any better? I believe that’s what fwildclusterboot in R uses to call WildBootTests.jl.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [April 17, 2024, 10:55pm UTC](https://discourse.julialang.org/t/julia-as-a-universal-platform-for-statistical-software-development/113017/12 "2024-04-17T22:55:15Z")

</div>

> [@droodman](#):
>
> presenting a vision of using Julia as a universal back end for statistical software development. Do things once in Julia–optimize, maintain, add features–then call from R or Python or Matlab or Stata.

It’s a great vision, that I share (for selfish reasons, helps add or improve Julia packages). I would say there’s nothing about the vision limiting it to stats… so you could amend the title, or keep as you wish. Thanks for adding Stata to the list. Even if only viable for it or other language, e.g. Python, then worthwhile. I know you can call easily to Python, I did (then with PyCall; also to MATLAB and Octave), and from Python, and believe it’s very solid with PythonCall, hopefully as good for Stata (does it download Julia for you? and take care of Julia package dependencies?). As explained can call from R, and to (with temporary maintenance issues, should be easily fixed) R.

---

<div class="post-metadata">

**Author:** ![droodman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/droodman/32/26652_2.png) [@droodman](https://discourse.julialang.org/u/droodman)\
**Post date:** [April 18, 2024, 12:55am UTC](https://discourse.julialang.org/t/julia-as-a-universal-platform-for-statistical-software-development/113017/13 "2024-04-18T00:55:41Z")

</div>

I completely agree it’s not limited to stats. I forget to say that in my post here. (I threw “statistical” in there as a signal of relevance to the professional community that is closest to being my home.)

No the Stata package doesn’t download Julia. That’s a great idea if I can reliably automate it. The barrier I see is the Windows Store installation for juliaup. It can be done with `winget install julia -s msstore` as a shell command. But when I run that I get prompted to “agree to all the source agreements terms.” Which is enough to impede automatic installation. Right now the documentation just tells people to go to the Windows Store and install Julia, which maybe is good enough.

---

<div class="post-metadata">

**Author:** ![TheCedarPrince](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/thecedarprince/32/17323_2.png) [@TheCedarPrince](https://discourse.julialang.org/u/TheCedarPrince)\
**Post date:** [April 18, 2024, 1:51am UTC](https://discourse.julialang.org/t/julia-as-a-universal-platform-for-statistical-software-development/113017/14 "2024-04-18T01:51:58Z")

</div>

> [@droodman](#):
>
> Does JuliaConnectoR work any better? I believe that’s what fwildclusterboot in R uses to call [WildBootTests.jl](https://juliahub.com/ui/Packages/General/WildBootTests).

Just wanted to say that I actually love JuliaConnectoR and provision my package, OMOPCDMCohortCreator.jl as well through the wrapper.

In fact, here is my documentation for how JuliaConnectoR works with my packgae: [Using OMOPCDMCohortCreator with R 🏴‍☠️ · OMOPCDMCohortCreator.jl](https://juliahealth.org/OMOPCDMCohortCreator.jl/stable/tutorials/r_tutorial/)

---

<div class="post-metadata">

**Author:** ![lrnv](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/lrnv/32/19373_2.png) [@lrnv](https://discourse.julialang.org/u/lrnv)\
**Post date:** [April 19, 2024, 7:23am UTC](https://discourse.julialang.org/t/julia-as-a-universal-platform-for-statistical-software-development/113017/15 "2024-04-19T07:23:07Z")

</div>

Maybe you could take a look at how JuliaconectoR and/or JuliaCall (to call julia from R) are doing the installation part ?