# Discussion about integer overflow

**URL:** <https://discourse.julialang.org/t/discussion-about-integer-overflow/69627>\
**Category:** Internals & Design\
**Tags:** numbers, integer-overflow\
**Created:** [October 11, 2021, 12:44pm UTC](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627 "2021-10-11T12:44:33Z")\
**Posts on this page:** 20\
**Page:** 6

<div class="post-metadata">

**Author:** ![kkretschmer](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kkretschmer/32/7474_2.png) [@kkretschmer](https://discourse.julialang.org/u/kkretschmer)\
**Post date:** [October 13, 2021, 10:09am UTC](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627/102 "2021-10-13T10:09:29Z")

</div>

> [@lungben](#):
>
> - Memory: assuming that the integer adresses single bytes, this would correspond to 9 \* 10^18 bytes or 9 \* 10^6 TB (even when wasting the adress room of negative integers). This should be out-of-reach for quite a while (except maybe for the largest supercomputer).

This is a problem I ran into where array sizes came from a binary exchange format using narrower integers (UInt32 in this case). Passing these directly to `Mmap.mmap` led to the allocation of a too small memory mapping in some cases which then resulted in a segfault or some other error.

The result was: [Avoid potential integer overflow in Mmap.mmap by kkretschmer · Pull Request #41186 · JuliaLang/julia · GitHub](https://github.com/JuliaLang/julia/pull/41186)

---

<div class="post-metadata">

**Author:** ![vjd](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/vjd/32/2644_2.png) [@vjd](https://discourse.julialang.org/u/vjd)\
**Post date:** [October 13, 2021, 10:41am UTC](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627/103 "2021-10-13T10:41:26Z")

</div>

30 vs 3 mg is still a 10 fold difference and doing a simple PKPD simulation with both doses will give your a proportional change in the metrics, e.g _AUC_. Such an error would be obvious to the analyst before it is passed down even for QC to a colleague. While I understand the spirit of the discussion is to ensure the tools are failsafe, in my experience, being in Pharma too, source of errors are usually due to analyst or the assumptions that they make. As @mbauman and others pointed out, being regulatory compliant and following the process is something that is taken very seriously.

Speaking of R as regulatory compliant, that took its own sweet time, close to 12 years before the FDA/EMA started accepting it. But if one puts that aside, the fact of the matter is that regulatory bodies have no mandate on what tool to use, and they cannot as they have to stay unbiased. The regulatory concerns that are being mentioned in this discussion are mostly related to business process and business users of the tools.

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [October 13, 2021, 11:15am UTC](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627/104 "2021-10-13T11:15:09Z")

</div>

> [@viraltux](#):
>
> I find more worrying this:
> 
> ```julia
> > julia> 1 / 10^20000
> > Inf
> 
> ```

Because of overflows, and getting rid of them with this idea would be a great _option_:

> [@viraltux](#):
>
> how about implementing a flag for a safe Julia (e.g. `julia --safe)` ?

> [@viraltux](#):
>
> **would just throwing an error also slow down the entire language?**

Doesn’t need to be.\*

> [@GunnarFarneback](#):
>
> has been [discussed](https://discourse.julialang.org/t/a-plea-for-int-overflow-checking-as-the-default/3338) repeatedly, has been implemented in a [package](https://github.com/JeffreySarnoff/SaferIntegers.jl), and finally [Julia is not at that stage of development anymore](https://discourse.julialang.org/t/psa-julia-is-not-at-that-stage-of-development-anymore/44872).

That was about changing the default, but the default type for integers doesn’t need to change, only the non-default option, so we do not have to wait for Julia 2.0 (only if we change the default later, what I’m not proposing now).

The question is what should replace the default integers, and I’m a bit conflicted, BigInt or SaferIntegers.jl. The argument for the former is that it’s already built into Julia, and it would give a great incentive to optimize it. I know it’s possible (even already done in some package?). Julia has a policy of not adding stuff Julia itself doesn’t need. That would have been an argument to not have BigInt and rationals in the first place… I would personally be ok with adding SaferIntegers.jl as a stdlib (and even removing BigInt if it’s chosen, moved to a package).

People might say you can already just use e.g. SaferIntegers.jl yourself and change the integer literals in the REPL as an option through a package available. But you have to opt into that from the REPL (or rely on package authors to change to an alternative type). I think we should also have the option for scripts, so that end-users can change the default without changing any code.

`*` What I have in mind, is that even if the default for literals would change, we could still use machine integers for stuff that matters e.g. array indexing (or even floats seems to work for JavaScript… I know V8 changes to integers behind the scenes). I’m not arguing for making this option fast right away, just available as a non-default option, that people would actually know about if looking at: `julia --help`

I have my own ideas how to make overflow checks fast, mostly by avoiding them in a lot of cases. Even a compromise of NOT checking for for them for additions only (the most important operation) would go a long way. It hardly ever is the cause of overflowing for 64-bit. That would be multiplying directly or e.g. exponentiation, that can be optimized.

* * *

It’s intriguing what can happen with overflows, already warned against (not exploited, yet?) in C (could hypothetically happen in pure Julia code, and even if it/Julia fixed, because if/since issue at lower level in libc/malloc/calloc not fixed, and could also even happen in (otherwise) safe Java) for (most of?) RTOS operating systems and more: [Multiple RTOS (Update E) | CISA](https://us-cert.cisa.gov/ics/advisories/icsa-21-119-04)

> EXECUTIVE SUMMARY
> 
> - ATTENTION: Exploitable remotely/low attack complexity
> - Vendors: Multiple
> - Equipment: Multiple
> - Vulnerabilities: Integer Overflow or Wraparound

E.g. [BadAlloc Vulnerability Affecting BlackBerry QNX RTOS | CISA](https://us-cert.cisa.gov/ncas/alerts/aa21-229a)

> On August 17, 2021, BlackBerry publicly disclosed that its QNX Real Time Operating System (RTOS) is affected by a [BadAlloc](https://us-cert.cisa.gov/ics/advisories/icsa-21-119-04) vulnerability—CVE-2021-22156. BadAlloc is a collection of vulnerabilities affecting multiple RTOSs and supporting libraries.[[1](https://support.blackberry.com/kb/articleDetail?articleNumber=000082334)] A remote attacker could exploit CVE-2021-22156 to cause a denial-of-service condition or execute arbitrary code on affected devices.[[2](https://support.blackberry.com/kb/articleDetail?articleNumber=000082334)]  
> […]  
> CVE-2021-22156 is an integer overflow vulnerability affecting the `calloc()` function in the C runtime library of multiple BlackBerry QNX products.
> 
> - U.S. Nuclear Regulatory Commission Security Advisory: [Blackberry QNX Vulnerability](https://www.nrc.gov/docs/ML2121/ML21217A177.pdf)

It may be overblown as nuclear reactors shouldn’t have an internet connection?

---

<div class="post-metadata">

**Author:** ![Oscar\_Smith](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/oscar_smith/32/25343_2.png) [@Oscar\_Smith](https://discourse.julialang.org/u/Oscar_Smith)\
**Post date:** [October 13, 2021, 1:06pm UTC](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627/105 "2021-10-13T13:06:56Z")

</div>

the one possible action item here is to add a warning for integer overflow for integer exponentiation. I’m pretty sure we could do it with a relatively small performance hit (10-20%) and is not an operation that is especially performance critical.

Edit: In fact @keno made a PR for this in 2017 [https://github.com/JuliaLang/julia/pull/21600/files](https://github.com/JuliaLang/julia/pull/21600/files)

---

<div class="post-metadata">

**Author:** ![viraltux](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/viraltux/32/15236_2.png) [@viraltux](https://discourse.julialang.org/u/viraltux)\
**Post date:** [October 13, 2021, 2:32pm UTC](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627/107 "2021-10-13T14:32:48Z")

</div>

> [@abulak](#):
>
> I love how you subtly imply that correlation may have something to do with causation 😛  
> We all know that frivolous usage of integer/fp arithmetic may lead to mathematically wrong answers (in any language), so I really don’t understand what’s the point. I was still on board when @tim.holy summarized the story. I’m lost by now in somehow eristic discussion that followed…

Hey! Long time no see…

Discussion that I was trying to move away from and refocus multiple times into having the best of both worlds scenarios conversation, without much success unfortunately… 🙂

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [October 13, 2021, 2:40pm UTC](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627/108 "2021-10-13T14:40:16Z")

</div>

It’s just challenging because [Julia is not at that stage of development anymore](https://discourse.julialang.org/t/psa-julia-is-not-at-that-stage-of-development-anymore/44872). Lack of “success” here has less to do with the conversation itself and more about its feasibility. I can guarantee you that `julia --check-overflow` would be broken before it even gets to the REPL. It’d take significant effort to make this work, and does not have an appreciable impact on any of the compliance standards I’ve been working with.

---

<div class="post-metadata">

**Author:** ![viraltux](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/viraltux/32/15236_2.png) [@viraltux](https://discourse.julialang.org/u/viraltux)\
**Post date:** [October 13, 2021, 2:40pm UTC](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627/109 "2021-10-13T14:40:24Z")

</div>

> [@vjd](#):
>
> Speaking of R as regulatory compliant, that took its own sweet time, close to 12 years before the FDA/EMA started accepting it.

Exactly my point, with all the checks that are technically possible and still took 12 years.

---

<div class="post-metadata">

**Author:** ![viraltux](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/viraltux/32/15236_2.png) [@viraltux](https://discourse.julialang.org/u/viraltux)\
**Post date:** [October 13, 2021, 2:41pm UTC](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627/110 "2021-10-13T14:41:53Z")

</div>

> [@mbauman](#):
>
> It’s just challenging because [Julia is not at that stage of development anymore](https://discourse.julialang.org/t/psa-julia-is-not-at-that-stage-of-development-anymore/44872). Lack of “success” here has less to do with the conversation itself and more about its feasibility. I can guarantee you that `julia --check-overflow` would be broken before it even gets to the REPL.

It’s that hard to implement? Well, it is the way it is, at the end the community of users will decide in which direction to go.

---

<div class="post-metadata">

**Author:** ![viraltux](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/viraltux/32/15236_2.png) [@viraltux](https://discourse.julialang.org/u/viraltux)\
**Post date:** [October 13, 2021, 2:50pm UTC](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627/111 "2021-10-13T14:50:04Z")

</div>

> [@czylabsonasa](#):
>
> One of the drawbacks of the default double approach is:
> 
> ```julia
> > x = 2^60 + 100; y = 2^60; sqrt(x-y)
> [1] 0
> > 
> 
> ```

On the other hand:

```julia
R> x = 2^60 + 100; y = 2^60; sqrt(x-y)
[1] 0

R> x = 2^60 + 100.; y = 2^60; sqrt(x-y)
[1] 0

```

and in Julia

```julia
julia> x = 2^60 + 100; y = 2^60; sqrt(x-y)
10.0

julia> x = 2^60 + 100.; y = 2^60; sqrt(x-y)
0.0

```

An that one might be a problem hard to spot. I don’t think it should be controversial that there are many more ways in which Julia can fail than R simply because Julia does not check for overflows and R does.

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [October 13, 2021, 2:53pm UTC](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627/112 "2021-10-13T14:53:31Z")

</div>

> [@viraltux](#):
>
> Julia does not check for overflows and R does.

R does NOT check for overflows afaik it just by default makes it hard to use integers

---

<div class="post-metadata">

**Author:** ![viraltux](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/viraltux/32/15236_2.png) [@viraltux](https://discourse.julialang.org/u/viraltux)\
**Post date:** [October 13, 2021, 2:53pm UTC](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627/113 "2021-10-13T14:53:58Z")

</div>

> [@StefanKarpinski](#):
>
> I am taking it quite seriously and this is a real error analysis based on modular arithmetic…

I would really like to focus on the best of both world scenarios if that is even possible, the back and forward about Julia fails but R fails too… I don’t think this is taking us anywhere.

---

<div class="post-metadata">

**Author:** ![Per](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/per/32/10387_2.png) [@Per](https://discourse.julialang.org/u/Per)\
**Post date:** [October 13, 2021, 2:54pm UTC](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627/114 "2021-10-13T14:54:02Z")

</div>

Something that could help new users, and would have zero impact on run-time performance is to add a warning a parse-level for the literal `10` to literal integer powers that overflow. The warning would read something like:

“The expression `10^21` results in integer overflow. To avoid overflow, use `1e21` for a floating-point value, or `big(10)^21` for a `BigInt`. Write `(10)^21` to compute an `Int64` without this warning.”

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [October 13, 2021, 3:11pm UTC](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627/115 "2021-10-13T15:11:50Z")

</div>

Exactly, I actually think this entire discussion is really fixable with a Linter

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [October 13, 2021, 3:14pm UTC](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627/116 "2021-10-13T15:14:16Z")

</div>

My point is that griping about the lack of success in a two-day conversation with _lots_ of engagement and that has remained remarkably productive with even [syntax suggestions](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627/69) and [an escalation to triage](https://github.com/JuliaLang/julia/pull/21600#issuecomment-942297847) from core developers is premature. These things take time, energy, and effort in addition to the consensus-building… and everyone’s prioritizations are different.

---

<div class="post-metadata">

**Author:** ![viraltux](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/viraltux/32/15236_2.png) [@viraltux](https://discourse.julialang.org/u/viraltux)\
**Post date:** [October 13, 2021, 3:15pm UTC](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627/117 "2021-10-13T15:15:17Z")

</div>

> [@johnmyleswhite](#):
>
> So I think the score is either even or slightly in Julia’s favor. But you seem to have a different understanding, which I’d like you to flesh out in some greater detail.

I don’t think it is just me; consider that every other language oriented to numerical analysis decided to do these checks. Is it possible that all of them were wrong? That their languages could be faster avoiding those checks and still have a “slight” improvement? It seems unlikely.

---

<div class="post-metadata">

**Author:** ![dlakelan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dlakelan/32/8491_2.png) [@dlakelan](https://discourse.julialang.org/u/dlakelan)\
**Post date:** [October 13, 2021, 3:19pm UTC](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627/118 "2021-10-13T15:19:52Z")

</div>

> [@viraltux](#):
>
> decided to do these checks

THERE ARE NO CHECKS! they just decided that you can by default only get floating point numbers! They took away options. You are arguing for example that people who do cryptography should have to jump through hoops to get integers and manipulate them. That’s unacceptable.

in R:

```julia
> typeof(2L)
[1] "integer"
> typeof(2L^65)
[1] "double"
> 

```

> [@viraltux](#):
>
> That their languages could be faster avoiding those checks and still have “slightly” improvement?

Julia is like 100x faster than R, that’s not slight improvement. You simply CAN NOT do problems in R that you can do in Julia

Consider that if Julia automatically promoted anything^2 to float you would have NO autodifferentiation or type stability or anything.

---

<div class="post-metadata">

**Author:** ![mbauman](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mbauman/32/31082_2.png) [@mbauman](https://discourse.julialang.org/u/mbauman)\
**Post date:** [October 13, 2021, 3:33pm UTC](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627/119 "2021-10-13T15:33:48Z")

</div>

> [@viraltux](#):
>
> consider that every other language oriented to numerical analysis decided to do these checks.

That’s simply not true. Numpy/Pandas is a great example here — although Python itself will auto-promote its integers to bignums as needed, the moment you move into the data analysis stacks you get the same overflow behaviors as Julia. So not only do they overflow like Julia, they do so inconsistently within the same language.

R and Matlab make it difficult to get integer behaviors in the first place and instead lean more heavily on floating point numbers — which are tricky in their own right and don’t behave like scientists typically expect, either.

---

<div class="post-metadata">

**Author:** ![viraltux](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/viraltux/32/15236_2.png) [@viraltux](https://discourse.julialang.org/u/viraltux)\
**Post date:** [October 13, 2021, 3:49pm UTC](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627/120 "2021-10-13T15:49:10Z")

</div>

> [@mbauman](#):
>
> That’s simply not true. Numpy/Pandas is a great example here — although Python

Well, I would not include Python into the numerical analysis group any more that Java, or any other generic language. Point taken though.

---

<div class="post-metadata">

**Author:** ![viraltux](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/viraltux/32/15236_2.png) [@viraltux](https://discourse.julialang.org/u/viraltux)\
**Post date:** [October 13, 2021, 3:51pm UTC](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627/121 "2021-10-13T15:51:24Z")

</div>

> [@mbauman](#):
>
> My point is that griping about the lack of success in a two-day conversation with _lots_ of engagement and that has remained remarkably productive with even [syntax suggestions](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627/69) and [an escalation to triage](https://github.com/JuliaLang/julia/pull/21600#issuecomment-942297847) from core developers is premature. These things take time, energy, and effort in addition to the consensus-building… and everyone’s prioritizations are different.

Actually I am very glad to hear this Matt! Conversations among people having an honest intent to improve things can be very productive. ​I am very glad to see positive outcomes, thank you for sharing!

The conversation has been pretty lengthy and I think that even we might not agree on everything we all have shared our points of view as clearly as we could, so besides you I will also extend my thanks to all of you guys that engaged in the conversation with an honest intent of improving the language we all love, and specially so to the core developers for sharing their knowledge and experience with us.

To those seemingly troubled people using Discourse to vent their frustrations I wish they can find peace.

---

<div class="post-metadata">

**Author:** ![Jake](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jake/32/46007_2.png) [@Jake](https://discourse.julialang.org/u/Jake)\
**Post date:** [October 13, 2021, 4:27pm UTC](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627/122 "2021-10-13T16:27:48Z")

</div>

My first calculator was the 2nd generation HP21. The second was the HP 34C, which had the magical integrate and solve keys. At some point I found an article written by the professor who implemented the numerical algorithms used by these keys. One of the things that sticks out is that if a numerical algorithm is well understood, it is possible to make a valid problem where it will give the incorrect result.  
I really appreciated this discussion, where some of the design philosophy behind the Julia implementations of algorithms has been discussed. And what the tradeoffs are of the various design decisions, not only in speed but in failure mechanisms .  
As an engineer, the onus is on me to understand the tools that I use and ensure that the results are reasonable. This has been emphasized in this discussion, that testing is extremely important. If a roof truss system fails, it is the responsibility of the engineer who signed off on the drawings, not the responsibility of the person who wrote the FEA software that performed the roof truss calculations.  
Personally I am extremely happy that the Julia core developers have been able to discuss their design decisions and show why they make sense and make Julia into a more versatile language than many of the other languages discussed. I am also glad that they did not make their design decisions based on being the same as all the other languages out there. I appreciate the rational approach. Keep up the good work!

[Previous page](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627.md?page=5)

[Next page](https://discourse.julialang.org/t/discussion-about-integer-overflow/69627.md?page=7)