Julia, JuMP, and Gurobi what MUST be cited?

Community
1

Sorry if this is not the right place, but community seems to be the category that better fit this question.

I am sending a conference paper that has a very strict page limit, they are 10 pages including the references. I have cited about five papers on my subject, and three more because of Julia, JuMP, and Gurobi. The citing format is kind space-wasting and if I removed the three citations I could avoid removing one or two paragraphs. My question is, what must be cited? I have no problem citing these technologies when they are not counted in the page limit (or there is a reasonable page limit just for bibliography). I mention the technologies I used in the paper, and I give the link to my julia code in github. There is some authoritative source that can answer my question whether is sufficient to just mention them in the text (I just use them to test my mathematical model) or I am ethically obligated to add them to the bibliography?

2

There is no authoritative source that can answer this.

I have seen both. In general for academics hard citations are more valuable than any blob of text (in case there is a proper reference that you can cite).

What would you prefer if you were the author of one of these technologies?

3

There is no authoritative source that can answer this.

Are you an authoritative source for that statement, XD. Seriously, are you from the language development? I find if some of the, I do not know, three main authors of the language said that I should cite it would be unethical to not do.

What would you prefer if you were the author of one of these technologies?

This is a terrible way to put it because: (1) what I think do not matter; (2) I develop all my code with Unlicense, so it is public domain, and would prefer my name to be forgotten and that nobody bothered to cite me because of code.

4

The authors of these packages are not authoritative sources. They (as I) don’t have the authority to tell you what to do or not to do. They can just make suggestions.

This is a terrible way to put it because: (1) what I think do not matter; (2) I develop all my code with Unlicense, so it is public domain, and would prefer my name to be forgotten and that nobody bothered to cite me because of code.

I am not saying that you should think like someone citing your software, but like someone who made a piece of software and a journal paper that one can cite when using the software. Just some basic golden rule here.

P.S.: According to Introduction · JuMP and Research they “request” you to cite them properly.

1 Like
5

Here’s one opinion: it depens what field you are in, and it depends how central these packages were to the work.

If you’re doing (say) ecology and just happened to use Julia because you got sick of Matlab, to do some fairly routine things, then there’s no obligation. (Although obviously still a friendly gesture, if you have space.)

If your entire paper was made possible by the unique abilities of (say) JuMP, that’s why you could do work your colleagues thought was impossible, then you have to cite it.

If you’re in CS, and actually building on these things or commenting on them, then it’s like everything else you’re citing, or not citing for lack of space.

1 Like
6

If your entire paper was made possible by the unique abilities of (say) JuMP, that’s why you could do work your colleagues thought was impossible, […]
If you’re in CS, and actually building on these things or commenting on them, […]

I completely agree on those points. If this was any of these cases, I
would have no doubt about which I should cite them, as they would be
the same as as any other paper I was using.

The disstinction here, for example, is that JuMP, at least, was the
completely opposite of “made possible by the unique abilities”. I
almost was not able to do the paper, and in the end accepted a
suboptimal outcome, just because I decided to use JuMP. If I could go
back in time I would have not used it in that specific moment for all
the time lost because of version conflicts, abandoned features,
variable deletion implemented in a very slow fashion, and so on. This
made me lost more time than a loss of productivity for using
CPLEX.jl/Gurobi.jl directly, for example.

7

Oh then you write a snarky footnote instead of a citation… and because you’re not a monster you delete it before submission.

1 Like
8

I’m sorry that this was your experience with JuMP. So that we can be aware of the relevant issues, could you be more specific about what you mean by version conflicts and abandoned features?

I followed the discussions about variable deletion in the gitter channel, and from my perspective there was a productive exchange between you and the other core developers about the issue and how to go about addressing it. For the benefit of readers of this thread, variable deletion in the Gurobi and CPLEX APIs requires renumbering variable indices, so if you do it eagerly O(n) times, you pay an O(n^2) cost. This can be a performance bottleneck if you structure your code to perform lots of deletions. JuMP should have an O(n) way to delete O(n) variables, but it hasn’t been implemented yet. For context, variable deletion was not possible at all before JuMP 0.19, so the upward trajectory on this issue should give you some hope for it working better in the future.

As @thofma mentioned, we do “kindly request” papers to cite uses of JuMP. The question of moral obligation is obviously complex. In this particular case I would usually ask an academic advisor, a coauthor, or the conference program committee. On one hand, omitting the citation is quite unlikely to be grounds for rejection of the paper. On the other hand, citations to JuMP are one of few metrics that we have available to demonstrate its use, so if nobody cited JuMP, it would negatively impact JuMP’s sustainability.

Anyway, we appreciate your contributions to JuMP and hope it will be useful next time around.

9 Likes
9

Here’s my (non-authorative) opinion:

If you’re using free, open-source software that was written by academics, then check which paper the authors want you to cite, and cite that. Citations are counted by automated systems, and a higher count means better chances of continued funding for that author.

If you are using commercial software, then citing is optional. You already paid for it, so you don’t owe the author anything. (Still, if you do cite, the authors will appreciate it.)

3 Likes
10

@miles.lubin Only to make some things clear:

When I said “If I could go back in time I would have not used it in that specific moment […]”, I meant exactly that. Given I had the foresight to see the problems I would encounter, I would not have used JuMP at that particular moment. I am not saying JuMP is a bad idea, that I will not use it in the future, that I do not want it to succeed, or even that I will not contribute to it again (I finally submitted the paper and next week is vacation to me, so I was thinking if I would try to contribute with something small enough this week).

The root of the problem is that I started using JuMP in the worst possible moment. [RANT MODE STARTED] I wrote JuMP code for the version 0.18, then I adapted it for 0.19, then I found 0.19 abandoned some features, then I go back, then I discover that a feature I would need in the future were not implemented in 0.18 (nor will be now, obviously), I have to make choices of what I will implement for the paper, I decide to keep with 0.19 because of some feature it has, and discover that, for example, variable deletion is implemented in a way that makes it worse than useless to me, not only I cannot use it but it also lured me to use 0.19 version and made me lose quite some time trying to understand what was happening, then I discover the problem is Gurobi specific, great, no problem, I will use CPLEX, for the final version of the paper it is even better to me to use CPLEX, then CPLEX.jl gives strange errors that I try to fix, “uh, strange, why it reverted to a super old version of the package, did I do something wrong?”, no, JuMP 0.19 does not work with current CPLEX.jl (but it is CPLEX.jl fault that the old version was installed, making me confused), it is very probable that CPLEX.jl compatibility will be fixed only after I submit the paper, hmmm, reverting to 0.18 makes me lose the possibility to warm-start (what I would have expected to be kinda of a basic feature, but maybe there are hidden complexities to it), well, fuck my NP-Hard problem, the real problem I should be studying is what is the maximal set of JuMP features I can get working to write my paper.[RANT MODE FINISHED]

I am not blaming JuMP more than I am blaming myself. I did know JuMP have not reached version 1.0 yet, and used it anyway. I should have given more consideration to the zero in front of the version number. My advisor was very impressed with JuMP (because of what she heard about it in conferences, but it is a long time since she touched any code) and wanted me to try it. Another professor I admire was using JuMP to teach some classes. I tried it, it worked at first, and I then maybe overcommited to it. Part of my bitterness comes from the fact I then became an evangelist, got two friends at the lab to use it and my co-advisor to at least try it. But our lab needs to use advanced features of solvers, and we discovered that, for us, in the current state, JuMP is more like a toy. It is fun to use, but it is not able to keep with we need from it. So did not I end up falling into a hole, but I brought at least one of my friends from the lab with me (he had ported a lot of his code to Julia before starting to hit walls).

I am very grateful to you, in particular, for your suggestions in the gitter channel. You took your time to describe the root of my problem when I asked (the way the state was handled in Gurobi.jl+JuMP internals), and gave me suggestions of workarounds when I gave up on my patch to solve the root of my problems “the right way”.

1 Like
11

My 2¢: if you cite JuMP, it’s far less necessary to cite Julia itself; so long as you cite the Julia packages that you use directly, that’s sufficient (although a citation if you have space is always nice). I can’t speak for Gurobi, but given that it’s a rather costly commercial product, I suspect that a mention rather than a citation would be sufficient, as they are not paid in citations.

2 Likes
12

I am sorry to hear about your frustrating experience with some free software. Yet I wonder if its authors deserve this kind of tone: it seems they tried to help you and they are working on solving the problem.

13

Which features are you referring to that were abandoned from JuMP 0.18? The only feature I’m aware of is solver-independent callbacks, which will be making a return in JuMP 0.21. The new CPLEX/MOI wrapper compatible with JuMP 0.20 that @odow rewrote was released on November 9th.

I have to push back on the suggestion that JuMP is a toy for your use cases. If you’re already comfortable enough to use CPLEX.jl or Gurobi.jl directly, you can essentially use anything you like from low-level APIs in JuMP’s direct mode. This may not be the easiest way to accomplish what you want compared with the polished Python or C++ APIs for each specific commercial solver, but the low-level APIs are certainly much more accessible from JuMP than from any other solver-independent modeling language that I’m aware of.

6 Likes
14

Yet I wonder if its authors deserve this kind of tone:

My rant was not directed at the library authors, as it should be exceedingly clear by the first paragraph and the phrase preceding the rant. Also, I enclosed the rant in an uppercased warning. So I do not really get what you are trying to convey here, Tamas. I already made clear that I am grateful, if not to all of them at least to miles.lubin, for the help.

15

Under what circumstances work should be cited? Well, citing works used have numerous advantages such as being able to place your work in the literature and context which would help readers (and reviewers) better understand your work. Another advantage is to have credit given to those that had useful work contribute towards your work. For example, allowing them to receive credit which may help them in the careers, etc. Another vital benefit is for reproducibility of your analysis. Without a clear delineation of your methods, it would be hard to have reproducible analysis / result. One last consideration is that journals might have specific guidelines on what and how work must be cited.

I consider citation more as a great tool that has tons of benefits rather than a task I need to do to avoid charges of plagiarism or academic dishonesty. Given certain limitations such as space, I would at least mention the software, have an appendix with the repository and the Project.toml/Manifest.toml, and cite the most relevant tools.

It might be more pertinent to cite Flux.jl than DataStructures.jl even if both are dependencies in your work. Maybe you could at least have the URL to a repository and further references there.

16

Which features are you referring to that were abandoned from JuMP 0.18?

hmmm, The callbacks were a problem for one of our research lanes (in which we were working a benders decomposition) but I have to admit the argument is kinda weak here, because in the end we discovered we needed the capacity to reject new incumbents without adding a restriction, what is not supported by all solvers.

However, I had clear memory of having to apologize to my advisor for the constant switching and it was for the research object of this last paper. I have looked at the changelog (https://github.com/JuliaOpt/JuMP.jl/blob/master/NEWS.md) and I am not sure if it was a performance regression, or the fact the solving the relaxed model was not implemented. I remember having performance problems in one or other point (besides the Gurobi var deletion problem) but I cannot assure you it was that. I need to solve the model relaxation even more than one time (i.e., build, solve relaxation, change some things, run relaxation again), and some of the code for this paper is a generic method to take a model and relax their variables (https://github.com/henriquebecker91/guillotine-models/blob/master/GuillotineModels/src/ModelUtils.jl#L124), so I guess in a first moment I did go back to 0.18.5 just to avoid having to write a method to do this (or not even crossed my mind that writing it was a reasonable option) and in the end writing such method was the smallest of the problems.

The new CPLEX/MOI wrapper compatible with JuMP 0.20 that @odow rewrote was released on November 9th.

In Oct 17, I asked how CPLEX/MOI wrapper rewrite was going and the answers were that “it is a non-trivial amount of work” (https://gitter.im/JuliaOpt/JuMP-dev?at=5da8d99057c2517c6a280c90) and “going to look at updating CPLEX.jl next week” (https://gitter.im/JuliaOpt/JuMP-dev?at=5da8b72ff88b526fb957458b). At this point the deadline (Nov 29) was already “too close for my tastes” and I had to decide what I would run, which version of JuMP and solver I would use, to also know what features of my code would be available in the experiments and consequently which paper I would be writing. I was not aware that the rewrite ended before the deadline, but it would be kind of short notice to do the necessary adaptations, re-run experiments, and everything.

If you’re already comfortable enough to use CPLEX.jl or Gurobi.jl directly, you can essentially use anything you like from low-level APIs in JuMP’s direct mode.

This boldened assumption is the problem. There is the problem they are based in the interface of the C library instead, for example, the C++ one that is a little higher level and better to work with, but this is not my biggest problem here. My biggest problem is that they are largely undocumented (both basically say: we do not suggest using this package alone, use it with JuMP). I know the C interface of both solvers is available and that the packages try to follow it, but I end up losing time with nitpicking, like which exact types Julia is expecting (as the C types are not exactly the same), if the name of the function was changed in some way (to follow a style guide), etc…

I will try to exercise the technique you propose (using direct mode together with solver-specific methods when necessary), thanks for the suggestion. However, I think that as a developer of these projects you have a much clearer vision of the interfaces of these low-level wrappers and is overestimating my productivity as I scramble through the REPL trying to check where is the Julia method for some C routine documented in the original solver docs.

17

Cite JuMP only and mention “references therein”, since that probably has all necessary references to Julia. Gurobi, well, everybody knows Gurobi, you can put a footnote with a link to their website if you really feel like it. But I would rather put a link to Julia’s homepage then :grinning:

2 Likes
18

Thanks for expanding. The state of the solver wrappers is a known issue, which is why we recommend non-experts to use JuMP rather than low-level wrappers. If you expect to have tidy, documented solver-specific wrappers for one of the commercial solvers, I would recommend using an officially supported interface. We’re hoping that more commercial solver developers will create and support Julia interfaces soon, like Artelys and Mosek have done. As you might understand, the JuMP developers don’t often feel strongly motivated to do unpaid work that benefits only users of a particular commercial software package.

For cross-reference, this discussion also continued at Solver-dependent callbacks in JuMP -- How to do it right? - #5 by miles.lubin.

3 Likes