# Why is it reliable to use open source packages for research?

**URL:** <https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823>\
**Category:** Offtopic\
**Tags:** package\
**Created:** [May 4, 2024, 3:49am UTC](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823 "2024-05-04T03:49:28Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![GregVernon](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gregvernon/32/5611_2.png) [@GregVernon](https://discourse.julialang.org/u/GregVernon)\
**Post date:** [May 9, 2024, 5:33pm UTC](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823/23 "2024-05-09T17:33:35Z")

</div>

> [@tbeason](#):
>
> How well is new functionality in Matlab / Stata / SAS tested? We don’t know.

I will say, however, that Matlab has built up a lot of _trust_. They have an incredibly-responsive customer service, post known issues publicly, provide regular bug-fixes, etc. There are similar closed-source software that have similar amounts of trust: LS-DYNA, Abaqus, COMSOL, etc.

---

<div class="post-metadata">

**Author:** ![WuSiren](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/wusiren/32/42529_2.png) [@WuSiren](https://discourse.julialang.org/u/WuSiren)\
**Post date:** [May 9, 2024, 5:42pm UTC](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823/24 "2024-05-09T17:42:24Z")

</div>

I myself feel more safe using commercial software like MATLAB than any other open-source software although the latter is indeed more transparent and appealing. Don’t know why.

---

<div class="post-metadata">

**Author:** ![cjdoris](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cjdoris/32/213133_2.png) [@cjdoris](https://discourse.julialang.org/u/cjdoris)\
**Post date:** [May 9, 2024, 5:53pm UTC](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823/25 "2024-05-09T17:53:08Z")

</div>

> [@WuSiren](#):
>
> I myself feel more safe using commercial software like MATLAB […] Don’t know why.

To quote Douglas Adams: because it’s reassuringly expensive.

---

<div class="post-metadata">

**Author:** ![ufechner7](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ufechner7/32/51363_2.png) [@ufechner7](https://discourse.julialang.org/u/ufechner7)\
**Post date:** [May 9, 2024, 6:24pm UTC](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823/26 "2024-05-09T18:24:16Z")

</div>

I encountered very bad, undocumented bugs in Matlab/ Simulink… Example: if you try to linearize a Simulink system that has two or more algebraic loops you get completely wrong results without any warning. With `ModelingToolkit.jl` it just works. I would say your feeling is misleading…

---

<div class="post-metadata">

**Author:** ![jar1](https://avatars.discourse-cdn.com/v4/letter/j/c0e974/32.png) [@jar1](https://discourse.julialang.org/u/jar1)\
**Post date:** [May 9, 2024, 6:40pm UTC](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823/28 "2024-05-09T18:40:45Z")

</div>

I (like some, unlike others) feel that Julia usage has higher risk of producing incorrect results than I’d like. Given that, I think it’s good to calibrate fear/uncertainty/doubt levels so that (1) new users know they should be extra careful if they can afford that risk and go elsewhere if they can’t and (2) developers see there’s still concern about this and know that there is user demand for further investment in quality assurance.

But since FUD is also annoying, I expect there’s a balance to be found between too much and too little griping.

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [May 9, 2024, 6:48pm UTC](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823/30 "2024-05-09T18:48:52Z")

</div>

> [@jar1](#):
>
> Julia usage has higher risk of producing incorrect results

This reminds me of [Pentium FDIV bug - Wikipedia](https://en.wikipedia.org/wiki/Pentium_FDIV_bug) .

I don’t think that Julia has a higher risk for incorrect results. Not more than R or python. How is this higher risk argued? At the end it’s native code produced by LLVM. I can’t imagine Julia anything different than other things compiled by LLVM. Can someone enlighten me, where this higher risk comes from?

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [May 9, 2024, 6:58pm UTC](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823/32 "2024-05-09T18:58:56Z")

</div>

Laying out concrete issues and appropriately setting expectations is not FUD — it’s the exact opposite.

> [@oheil](#):
>
> Can someone enlighten me, where this higher risk comes from?

The Julia language is highly permissive. It allows you to combine packages that have never been combined before. That might work and be amazing. It might error. Or it might silently do something wrong. I don’t see a higher risk with individual packages as compared to other languages, but it’s the ecosystem’s high composability that poses a slightly more unique risk (and benefit!) — something that you just need to be aware of and test/validate if there aren’t existing use-cases in the test suites.

---

<div class="post-metadata">

**Author:** ![oheil](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oheil/32/220745_2.png) [@oheil](https://discourse.julialang.org/u/oheil)\
**Post date:** [May 9, 2024, 7:02pm UTC](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823/34 "2024-05-09T19:02:45Z")

</div>

> [@mbauman](#):
>
> something that you just need to be aware of and test/validate

and that’s what I strongly argue here for any software you use as a researcher 😉

---

<div class="post-metadata">

**Author:** ![liuyxpp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/liuyxpp/32/9870_2.png) [@liuyxpp](https://discourse.julialang.org/u/liuyxpp)\
**Post date:** [May 10, 2024, 1:15am UTC](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823/39 "2024-05-10T01:15:58Z")

</div>

Where does your conclusion come from? This kind of baseless conclusion about Julia is so harmful. Please don’t say it again unless you have concrete evidence when comparing to at least one other language in a massive way.

---

<div class="post-metadata">

**Author:** ![yurivish](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/yurivish/32/307_2.png) [@yurivish](https://discourse.julialang.org/u/yurivish)\
**Post date:** [May 10, 2024, 3:16am UTC](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823/40 "2024-05-10T03:16:44Z")

</div>

> [@mbauman](#):
>
> > [@eweiss](#):
> >
> > A quick look at some of Julia’s statistical modules shows that they have simple test cases that are used to validate the results. Since I did not do an extensive survey of what various test directories contain, they will remain unnamed.
> > 
> > Will they work properly in edge cases? Who knows.
> 
> Right, if there are concrete places where packages should have more tests, please just open an issue — or better yet, help contribute those tests directly. Contributing tests is the best way to (a) confirm that the functionality you care about works as you want and (b) ensure that it continues to work through continued development.
> 
> There’s no need for FUD here — the code itself along with the filed issues and ongoing development work is right out in the open.

Is it [FUD](https://en.wikipedia.org/wiki/Fear,_uncertainty,_and_doubt) if it’s true?

A few months ago I happened to run into a bug that’s the perfect example of exactly the scenario @eweiss outlined in his post.

I was using Julia for extremely basic data analysis – finding the maximum value in a column of a CSV file – and discovered that the answer was simply wrong. I only spotted the bug because I manually inspected the results and the number seemed to me to be too low.

I filed an [issue](https://github.com/JuliaData/CSV.jl/issues/1128) that outlined the bug, identified its cause, and contained a reproducible test case. In it, I wrote

> The implementation is [here](https://github.com/JuliaData/SentinelArrays.jl/blob/fa840f994ae821d921a9973fbd5e244d35102b1c/src/chainedvector.jl#L838) and is [tested](https://github.com/JuliaData/SentinelArrays.jl/blob/fa840f994ae821d921a9973fbd5e244d35102b1c/test/chainedvector.jl#L253), though the test only tests a simple special case for which the values and indices are exactly equal.

---

<div class="post-metadata">

**Author:** ![jling](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jling/32/212909_2.png) [@jling](https://discourse.julialang.org/u/jling)\
**Post date:** [May 10, 2024, 3:45am UTC](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823/41 "2024-05-10T03:45:44Z")

</div>

> <https://github.com/JuliaData/SentinelArrays.jl/issues/97#issuecomment-2103803383>
>
> This package incorrectly implements \[\`argmax\`\](https://docs.julialang.org/en/v1/…base/collections/#Base.argmax) for \`ChainedVector\`s. It returns the maximum value, rather than its index. 
> 
> \`\`\`julia
> julia\> using SentinelArrays
> 
> julia\> arrays = \[
> \[18, 70, 92, 15, 65\],
> \[25, 14, 95, 54, 57\]
> \];
> 
> julia\> cv = ChainedVector(arrays);
> 
> julia\> argmax(cv)
> 95
> 
> julia\> argmax(collect(cv))
> 8
> \`\`\`
> 
> 
> 
> The implementation is \[here\](https://github.com/JuliaData/SentinelArrays.jl/blob/fa840f994ae821d921a9973fbd5e244d35102b1c/src/chainedvector.jl#L838) and is \[tested\](https://github.com/JuliaData/SentinelArrays.jl/blob/fa840f994ae821d921a9973fbd5e244d35102b1c/test/chainedvector.jl#L253), though the test only tests a simple special case for which the values and indices are exactly equal.
> 
> This issue was initially reported \[upstream\](https://github.com/JuliaData/CSV.jl/issues/1128).

looks like it’s worse than that – not only the `argmax/argmin` are taking the wrong element from `findmax/findmin`. it looks like `findmin` itself is just completely wrong. (`findmax` seems to be fine)

> [@yurivish](#):
>
> test only tests a simple special case for which the values and indices are exactly equal.

this happens to the best of us though, but yeah, this particular unit test is filled with tests but most are basically useless checks primarily because of this `value = index` property, also partly because the tests are just super redundant…

now I’m trying to recall if I ever used `argmin/findmin/argmax/findmax` for publications since it’s definitely [used](https://github.com/JuliaHEP/UnROOT.jl/blob/cc3ea9bcd582feba8708de4d6f262c31ea3669bc/Project.toml#L25) in our data I/O

---

<div class="post-metadata">

**Author:** ![nilshg](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/nilshg/32/2283_2.png) [@nilshg](https://discourse.julialang.org/u/nilshg)\
**Post date:** [May 10, 2024, 7:49am UTC](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823/42 "2024-05-10T07:49:26Z")

</div>

Not to derail this thread further (can a broad question in Offtopic be derailed) but @Sukera, somewhat following on from our discussion on Zulip a few weeks ago, is this something that Supposition.jl would have helped with?

---

<div class="post-metadata">

**Author:** ![Sukera](https://avatars.discourse-cdn.com/v4/letter/s/ce7236/32.png) [@Sukera](https://discourse.julialang.org/u/Sukera)\
**Post date:** [May 10, 2024, 8:01am UTC](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823/43 "2024-05-10T08:01:42Z")

</div>

> [@nilshg](#):
>
> is this something that [Supposition.jl](https://juliahub.com/ui/Packages/General/Supposition) would have helped with?

Yes:

```julia
julia> using Supposition, SentinelArrays

# generate `Vector{UInt8}` with at most 100 elements
julia> data_vector = Data.Vectors(Data.Integers{UInt8}(); max_size=100);

# generate a vector of vectors, with 1 to 10 vectors
julia> vecs = Data.Vectors(data_vector; min_size=1, max_size=10);

# create a ChainedVector
julia> chainedvecs = map(vecs) do vs
           # can't reduce over empty arrays, so ignore them
           all(isempty, vs) && reject!()
           ChainedVector(vs)
       end;

# Does the found index actually refer to the maximum?
julia> isargmax_maximum(cv) = cv[argmax(cv)] == maximum(cv)

# Run the fuzzing/property test
julia> @check db=false isargmax_maximum(chainedvecs);
┌ Error: Property errored!
│ Description = "isargmax_maximum"
│ arg_1 =
│ 1-element ChainedVector{UInt8, Vector{UInt8}}:
│ 0x00
│ exception =
│ BoundsError: attempt to access 1-element ChainedVector{UInt8, Vector{UInt8}} at index [0x00]
│ Stacktrace:
│ [1] throw_boundserror(A::ChainedVector{UInt8, Vector{UInt8}}, I::Tuple{UInt8})
│ @ Base ./essentials.jl:14
│ [2] checkbounds
│ @ ./abstractarray.jl:699 [inlined]
│ [3] getindex
│ @ ~/.julia/packages/SentinelArrays/1kRo4/src/chainedvector.jl:94 [inlined]
│ [4] isargmax_maximum(cv::ChainedVector{UInt8, Vector{UInt8}})
│ @ Main ./REPL[39]:1
└ @ Supposition ~/Documents/projects/Supposition.jl/src/testset.jl:287
Test Summary: | Error Total Time
isargmax_maximum | 1 1 0.7s

```

The important part is checking the correct/expected properties (in this case that the return value of `argmax` is the index of the maximum element). It’s possible to do that with just naively generated random data; Supposition.jl “just” does the job of reducing a found problem to a minimal example (here, `ChainedVector([[0]])`).

This is very off topic now, but I think part of the problem with `argmax` in particular is that it does return the element, rather than the index, in the `argmax(f, col)` case. See [Unintuitive findmin and findmax · Issue #39203 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/39203) and [`argmax` behaves differently from other higher-order functions · Issue #48502 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/issues/48502) for discussion and [Add `indmin`/`indmax` by Seelengrab · Pull Request #41339 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/pull/41339) for my (old, non-mergeable) attempt at clarifying the situation.

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [May 10, 2024, 12:11pm UTC](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823/44 "2024-05-10T12:11:16Z")

</div>

Great to have you back! Obviously that’s not FUD — it’s a concrete issue that is clearly a bug. Thanks for filing the issues!

---

<div class="post-metadata">

**Author:** ![Tamas\_Papp](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tamas_papp/32/25949_2.png) [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)\
**Post date:** [May 10, 2024, 2:37pm UTC](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823/45 "2024-05-10T14:37:15Z")

</div>

> [@mbauman](#):
>
> I don’t see a higher risk with individual packages as compared to other languages, but it’s the ecosystem’s high composability that poses a slightly more unique risk

OTOH, Julia is outstanding when it comes to relatively small packages of generic code, which can be very thoroughly tested and (ideally) expose a simple interface that the user can grasp in its entirety. You make a very valid point, but it is unclear whether the net effect would be more or fewer bugs.

In any case, I think the _license_ for the code (ie whether a library is FOSS) is just one dimension for evaluating its reliability. _Code maturity_ is another, and closely related, the _number of users_. Personally, I also put a lot of weight on the _principal authors and maintainers_, as there are quite a few people in the Julia community whose code I would trust a lot because I have seen their coding style.

One cannot make blanket statements about either closed or open source software. I agree with @oheil they only way to build trust in software is to evaluate it, so if I end up using a package for more than exploratory work I usually look at the source code, test coverage, testing style, and in some cases end up adding tests in PRs. This way the effort I have put into evaluating code ends up benefiting others.

---

<div class="post-metadata">

**Author:** ![mihalybaci](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mihalybaci/32/13528_2.png) [@mihalybaci](https://discourse.julialang.org/u/mihalybaci)\
**Post date:** [May 10, 2024, 5:32pm UTC](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823/46 "2024-05-10T17:32:03Z")

</div>

This is why I like open source software, and following issues on Discourse. Here we have an example of a bug in `argmax`, I can go look at the github issue, then follow that to the PR to fix it, and see that it merged ~5 hours ago. I can actually watch the process work. Sometimes it doesn’t work as fast as wanted, but closed source doesn’t necessary work fast either.

Things improve over time, and it is reassuring to watch it happen in the open. I mean, how many bugs were in Numeric before it became NumPy? (that was a rhetorical question)

---

<div class="post-metadata">

**Author:** ![mkitti](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mkitti/32/12459_2.png) [@mkitti](https://discourse.julialang.org/u/mkitti)\
**Post date:** [May 11, 2024, 10:59pm UTC](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823/47 "2024-05-11T22:59:06Z")

</div>

I worked on generating and adding additional tests for SentinelArrays.jl here:

> <https://github.com/JuliaData/SentinelArrays.jl/pull/99>
>
> This pull request adds more tests including randomly generated \`ChainVector\`s of… \`Int\`, \`Float64\`, and \`Rational\`, incorporates #98 .
> 
> Functions fixed
> \* \`argmin\`
> \* \`argmax\`
> \* \`findmin\`
> \* \`findmax\`
> \* \`findXwithfirst\`
> 
> Fixed #101

In the process I found another bug with `findmax`:

> <https://github.com/JuliaData/SentinelArrays.jl/issues/101>
>
> @quinnj \`findmax\` is still broken when there are repeated values.
> 
> \`\`\`
> julia\>… c = ChainedVector(\[\[7, 21, 31\], \[45\], \[53, 53\]\])
> 6-element ChainedVector{Int64, Vector{Int64}}:
> 7
> 21
> 31
> 45
> 53
> 53
> 
> julia\> findmax(c)
> (53, 6)
> 
> julia\> findmax(c |\> collect)
> (53, 5)
> \`\`\`
> 
> Fixed by #99

While I am glad the original issue was fixed, I’m wondering why the attention to the package was so narrow to ignore the other pull request addesssing perhaps the deeper issue - insufficient testing. I’m sure it was an oversight due to limited time.

Certainly when presented with a concrete bug the focus must not only be on fixing that bug but addressing why that bug was able to exist in the first place without being caught by CI.

---

<div class="post-metadata">

**Author:** ![tecosaur](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tecosaur/32/23206_2.png) [@tecosaur](https://discourse.julialang.org/u/tecosaur)\
**Post date:** [May 12, 2024, 2:00am UTC](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823/48 "2024-05-12T02:00:45Z")

</div>

It would also be good if tools like Supposition.jl could be more widely adopted and so help avoid “oops, we tested a special case where the bug doesn’t occur” cases.

---

<div class="post-metadata">

**Author:** ![xlxs4](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/xlxs4/32/224566_2.png) [@xlxs4](https://discourse.julialang.org/u/xlxs4)\
**Post date:** [May 12, 2024, 1:30pm UTC](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823/49 "2024-05-12T13:30:55Z")

</div>

FWIW, I recall an old blog post trying AFL with Julia: [Bugs in Julia with AFL and C-Reduce · maleadt](https://blog.maleadt.net/2018/11/16/julia_bugs/)

As also highlighted in the post, a possible avenue could be a Julia package wrapping [AFL++](https://github.com/AFLplusplus/AFLplusplus)

---

<div class="post-metadata">

**Author:** ![o314](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/o314/32/252_2.png) [@o314](https://discourse.julialang.org/u/o314)\
**Post date:** [May 12, 2024, 5:03pm UTC](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823/50 "2024-05-12T17:03:41Z")

</div>

In civil engineering, it is a common practice to use two equivalents softwares but from different vendors to check their results one against the other when doing some quantitative mechanical design.

Not a problem of open or closed source but a consideration about consistent and regular control

[Previous page](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823.md?page=1)

[Next page](https://discourse.julialang.org/t/why-is-it-reliable-to-use-open-source-packages-for-research/113823.md?page=3)