Why is it reliable to use open source packages for research?

Offtopic
23

I will say, however, that Matlab has built up a lot of trust. They have an incredibly-responsive customer service, post known issues publicly, provide regular bug-fixes, etc. There are similar closed-source software that have similar amounts of trust: LS-DYNA, Abaqus, COMSOL, etc.

2 Likes
24

I myself feel more safe using commercial software like MATLAB than any other open-source software although the latter is indeed more transparent and appealing. Donā€™t know why.

25

To quote Douglas Adams: because itā€™s reassuringly expensive.

23 Likes
26

I encountered very bad, undocumented bugs in Matlab/ Simulinkā€¦ Example: if you try to linearize a Simulink system that has two or more algebraic loops you get completely wrong results without any warning. With ModelingToolkit.jl it just works. I would say your feeling is misleadingā€¦

9 Likes
28

I (like some, unlike others) feel that Julia usage has higher risk of producing incorrect results than Iā€™d like. Given that, I think itā€™s good to calibrate fear/uncertainty/doubt levels so that (1) new users know they should be extra careful if they can afford that risk and go elsewhere if they canā€™t and (2) developers see thereā€™s still concern about this and know that there is user demand for further investment in quality assurance.

But since FUD is also annoying, I expect thereā€™s a balance to be found between too much and too little griping.

4 Likes
30

This reminds me of Pentium FDIV bug - Wikipedia .

I donā€™t think that Julia has a higher risk for incorrect results. Not more than R or python. How is this higher risk argued? At the end itā€™s native code produced by LLVM. I canā€™t imagine Julia anything different than other things compiled by LLVM. Can someone enlighten me, where this higher risk comes from?

32

Laying out concrete issues and appropriately setting expectations is not FUD ā€” itā€™s the exact opposite.

The Julia language is highly permissive. It allows you to combine packages that have never been combined before. That might work and be amazing. It might error. Or it might silently do something wrong. I donā€™t see a higher risk with individual packages as compared to other languages, but itā€™s the ecosystemā€™s high composability that poses a slightly more unique risk (and benefit!) ā€” something that you just need to be aware of and test/validate if there arenā€™t existing use-cases in the test suites.

10 Likes
34

and thatā€™s what I strongly argue here for any software you use as a researcher :wink:

1 Like
39

Where does your conclusion come from? This kind of baseless conclusion about Julia is so harmful. Please donā€™t say it again unless you have concrete evidence when comparing to at least one other language in a massive way.

3 Likes
40

Is it FUD if itā€™s true?

A few months ago I happened to run into a bug thatā€™s the perfect example of exactly the scenario @eweiss outlined in his post.

I was using Julia for extremely basic data analysis ā€“ finding the maximum value in a column of a CSV file ā€“ and discovered that the answer was simply wrong. I only spotted the bug because I manually inspected the results and the number seemed to me to be too low.

I filed an issue that outlined the bug, identified its cause, and contained a reproducible test case. In it, I wrote

The implementation is here and is tested, though the test only tests a simple special case for which the values and indices are exactly equal.

15 Likes
41

looks like itā€™s worse than that ā€“ not only the argmax/argmin are taking the wrong element from findmax/findmin. it looks like findmin itself is just completely wrong. (findmax seems to be fine)

this happens to the best of us though, but yeah, this particular unit test is filled with tests but most are basically useless checks primarily because of this value = index property, also partly because the tests are just super redundantā€¦

now Iā€™m trying to recall if I ever used argmin/findmin/argmax/findmax for publications since itā€™s definitely used in our data I/O

1 Like
42

Not to derail this thread further (can a broad question in Offtopic be derailed) but @Sukera, somewhat following on from our discussion on Zulip a few weeks ago, is this something that Supposition.jl would have helped with?

43

Yes:

julia> using Supposition, SentinelArrays

# generate `Vector{UInt8}` with at most 100 elements
julia> data_vector = Data.Vectors(Data.Integers{UInt8}(); max_size=100);

# generate a vector of vectors, with 1 to 10 vectors
julia> vecs = Data.Vectors(data_vector; min_size=1, max_size=10);

# create a ChainedVector
julia> chainedvecs = map(vecs) do vs
           # can't reduce over empty arrays, so ignore them
           all(isempty, vs) && reject!()
           ChainedVector(vs)
       end;

# Does the found index actually refer to the maximum?
julia> isargmax_maximum(cv) = cv[argmax(cv)] == maximum(cv)

# Run the fuzzing/property test
julia> @check db=false isargmax_maximum(chainedvecs);
ā”Œ Error: Property errored!
ā”‚   Description = "isargmax_maximum"
ā”‚   arg_1 =
ā”‚    1-element ChainedVector{UInt8, Vector{UInt8}}:
ā”‚     0x00
ā”‚   exception =
ā”‚    BoundsError: attempt to access 1-element ChainedVector{UInt8, Vector{UInt8}} at index [0x00]
ā”‚    Stacktrace:
ā”‚     [1] throw_boundserror(A::ChainedVector{UInt8, Vector{UInt8}}, I::Tuple{UInt8})
ā”‚       @ Base ./essentials.jl:14
ā”‚     [2] checkbounds
ā”‚       @ ./abstractarray.jl:699 [inlined]
ā”‚     [3] getindex
ā”‚       @ ~/.julia/packages/SentinelArrays/1kRo4/src/chainedvector.jl:94 [inlined]
ā”‚     [4] isargmax_maximum(cv::ChainedVector{UInt8, Vector{UInt8}})
ā”‚       @ Main ./REPL[39]:1
ā”” @ Supposition ~/Documents/projects/Supposition.jl/src/testset.jl:287
Test Summary:    | Error  Total  Time
isargmax_maximum |     1      1  0.7s

The important part is checking the correct/expected properties (in this case that the return value of argmax is the index of the maximum element). Itā€™s possible to do that with just naively generated random data; Supposition.jl ā€œjustā€ does the job of reducing a found problem to a minimal example (here, ChainedVector([[0]])).

This is very off topic now, but I think part of the problem with argmax in particular is that it does return the element, rather than the index, in the argmax(f, col) case. See Unintuitive findmin and findmax Ā· Issue #39203 Ā· JuliaLang/julia Ā· GitHub and `argmax` behaves differently from other higher-order functions Ā· Issue #48502 Ā· JuliaLang/julia Ā· GitHub for discussion and Add `indmin`/`indmax` by Seelengrab Ā· Pull Request #41339 Ā· JuliaLang/julia Ā· GitHub for my (old, non-mergeable) attempt at clarifying the situation.

7 Likes
44

Great to have you back! Obviously thatā€™s not FUD ā€” itā€™s a concrete issue that is clearly a bug. Thanks for filing the issues!

14 Likes
45

OTOH, Julia is outstanding when it comes to relatively small packages of generic code, which can be very thoroughly tested and (ideally) expose a simple interface that the user can grasp in its entirety. You make a very valid point, but it is unclear whether the net effect would be more or fewer bugs.

In any case, I think the license for the code (ie whether a library is FOSS) is just one dimension for evaluating its reliability. Code maturity is another, and closely related, the number of users. Personally, I also put a lot of weight on the principal authors and maintainers, as there are quite a few people in the Julia community whose code I would trust a lot because I have seen their coding style.

One cannot make blanket statements about either closed or open source software. I agree with @oheil they only way to build trust in software is to evaluate it, so if I end up using a package for more than exploratory work I usually look at the source code, test coverage, testing style, and in some cases end up adding tests in PRs. This way the effort I have put into evaluating code ends up benefiting others.

17 Likes
46

This is why I like open source software, and following issues on Discourse. Here we have an example of a bug in argmax, I can go look at the github issue, then follow that to the PR to fix it, and see that it merged ~5 hours ago. I can actually watch the process work. Sometimes it doesnā€™t work as fast as wanted, but closed source doesnā€™t necessary work fast either.

Things improve over time, and it is reassuring to watch it happen in the open. I mean, how many bugs were in Numeric before it became NumPy? (that was a rhetorical question)

6 Likes
47

I worked on generating and adding additional tests for SentinelArrays.jl here:

In the process I found another bug with findmax:

While I am glad the original issue was fixed, Iā€™m wondering why the attention to the package was so narrow to ignore the other pull request addesssing perhaps the deeper issue - insufficient testing. Iā€™m sure it was an oversight due to limited time.

Certainly when presented with a concrete bug the focus must not only be on fixing that bug but addressing why that bug was able to exist in the first place without being caught by CI.

6 Likes
48

It would also be good if tools like Supposition.jl could be more widely adopted and so help avoid ā€œoops, we tested a special case where the bug doesnā€™t occurā€ cases.

11 Likes
49

FWIW, I recall an old blog post trying AFL with Julia: Bugs in Julia with AFL and C-Reduce Ā· maleadt

As also highlighted in the post, a possible avenue could be a Julia package wrapping AFL++

1 Like
50

In civil engineering, it is a common practice to use two equivalents softwares but from different vendors to check their results one against the other when doing some quantitative mechanical design.

Not a problem of open or closed source but a consideration about consistent and regular control

6 Likes