[ANN] SafeREPL: use BigInt by default at the REPL

With Julia 1.5 around the corner, I’m happy to present SafeREPL, a little experimental package which by default interprets Int and Int128 REPL literals as BigInt, and Float64 literals as BigFloat:

julia> using SafeREPL

julia> factorial(40)
815915283247897734345611269596115894272000000000

julia> sqrt(2.0)
1.414213562373095048801688724209698078569671875376948073176679737990732478462102

julia> c = 299_792_458; c^3 # the cube of the speed of light can even be computed safely!
26944002417373989539335912

The meaning of literals can be changed with the swapliterals! function:

julia> swapliterals!(Int => Float64)

julia> 2
2.0

So really, “safe” applies mostly only to the defaults (which is expected to be the typical use-case). But otherwise the package is “safe-agnostic”. Also, the Float64 default could be changed in the future (Float64 literals might be better off left alone).

See the README for details.

There is also a twin package, SwapLiterals, which doesn’t touch the REPL (and therefore won’t require Julia 1.5 once registered) but provides a @swapliterals macro to swap literals in source code.

Use the following to install (Julia 1.5 is required):

using Pkg
pkg"add https://github.com/rfourquet/SafeREPL.jl:SwapLiterals"
pkg"add https://github.com/rfourquet/SafeREPL.jl"

It’s a good time for suggesting alternative package names, in particular for SwapLiterals, before these get registered.

I would like to thank the few people involved in this Zulip chat for their ideas. For example @ColinCaine suggested to use pairs instead of keywords, and @non-Jedi suggested the name literally!(Int=>Float64) instead of swapliterals!(Int=>Float64), which I find quite nice.

Feedback here or on github is very welcome.
Stay safe :slight_smile:

16 Likes

It not that I don’t like this, just this existed before (was it just little know?) under: https://github.com/MasonProtter/ReplMaker.jl as you know. And maybe it’s better this way, not as a switchable REPL mode (since you can with a function), or should both be combined?

1 Like

Well, indeed the possibility existed before as a REPL mode, even before ReplMaker. And honestly, I initially didn’t think at all I would publish this package, but then it became non-trivial enough to be useful to other people. Few points to try to address your questions:

  1. The purpose of ReplMaker is to make it easy to create new REPL modes, so that package is not the right place to develop a robust “swap literals” functionality. The “Big-Mode” from ReplMaker’s README is for demo purpose, it’s not sufficient as a full replacement of the default REPL (e.g. try typing "1" in this “Big-Mode”). So I don’t think the two packages should be combined.
  2. The core of the SafeREPL package is implementing swapping literals in the AST, not only BigInt, and not only at the REPL. There is occasionally demand for such a thing, e.g. in the zulip chat I linked to above, the OP wanted to interpret all Int literals as Int8. Some people complained that the type of unsigned literals depends on the number of digits. They can now easily have all unsigned literals be UInt64. etc.
  3. One of the target audience of SafeREPL is beginners being confused/annoyed with how default literals can lead to wrong results. using SafeREPL is a mimimal-overhead solution, and is easier than using a custom REPL-mode: how to set-up the REPL-mode so that it’s the default when launching Julia? And also when exiting help-mode? How to enter directly help-mode when typing ?? these probably have answers, but I prefered to explore this “new approach enabled by Julia 1.5” (call it NAEBJ). It somehow seems to give more first-class status to modes that you want on all the time.
  4. This might be more composable; e.g. I have another toy package taking advantage of NAEBJ; using REPL-modes, how can I easily use both functionalities (from the two packages) at the same time? It’s certainly possible to come up with a solution, but using NAEBJ seems easier, package authors don’t need coordination.

Anyway, NAEBJ is new and I found it interesting to exploit it; real usage will provide more elements to compare approaches.

6 Likes

This is some nice work @rfourquet. It’s good to see people think seriously about ways to easily provide arbitrary precision arithmetic (or other reinterpretations of literals) in a way that causes minimal friction.


Not to be defensive, but I’m a little confused about this statement:

I’m not sure what exactly you’re referring to. Could you clarify what the problem is with typing 1 in this mode? Here’s what I see:

julia> using ReplMaker

julia> function Big_parse(str)
           Meta.parse(replace(str, r"[\+\-]?\d+(?:\.\d+)?(?:[ef][\+\-]?\d+)?" => x -> "big\"$x\""))
       end
Big_parse (generic function with 1 method)

julia> initrepl(Big_parse, 
                prompt_text="BigJulia> ",
                prompt_color = :red, 
                start_key='>', 
                mode_name="Big-Mode")
REPL mode Big-Mode initialized. Press > to enter and backspace to exit.

julia> >

BigJulia> 1
1

BigJulia> typeof(1)
BigInt

If I am to give you honest feedback here, I think SwapLiterals.jl is a cool and useful pacakge and if it’s used for something like SafeREPL.jl, that should be in an isolated REPL mode (not necessarily one provided by ReplMaker.jl, but I think a dependancy on ReplMaker would make your life easier :man_shrugging:). The way you’re hot-modifying the existing, standard julia repl mode to make it evaluate code in a non-standard way makes me very uncomfortable.

To my mind, repl modes are basically a nice way to automatically apply a macro (sometimes a string macro) to code automatically. They change the meaning of code. It should always be obvious that such a thing is happening such as with a custom prompt and it should be easy to get back to standard julia when needed. This is why julia requires that macro invocations begin with the @ sigil, otherwise it would never be clear when you’re looking at code that will be evaluated normally.

I understand that you show how to add stuff to your startup.jl to make it more clear what’s going on and be able to toggle this behaviour on or off, but I think it’s a serious flaw to make those things opt-in rather than opt-out and I don’t like that they can only be done before the repl has been started up.

That’s more to do with SwapLiterals.jl though, right? Presumably something using actual repl modes could have the same relation to SwapLiterals.jl as SafeREPL.jl does, correct?

I think that having a non-standard REPL that evaluates code differently is very likely to cause other, unforeseen problems for new users that they may not be equipped to deal with if they’re unable to use repl modes. I do not look forward to trying to help someone in #helpdesk debug their code only to learn 20 minutes into the conversation that they had something like SafeREPL enabled but forgot.

Composability is an interesting problem. Thinking of repl-modes as string macros, you basically can never have the sort of composability where two separate string macros get to operate on the string, but if your transformation only needs to operate on the AST level, rather than the string level, it would actually be pretty easy to do that in replmaker. You basically just do

initrepl(transformation2 ∘ transformation1; kwargs...)

I could even make a method where initrepl takes in a Vector of transformations and applies them sequentially and allow people to push! or modify that vector whenever with the only caveat that the first transformation needs to take in a string, and the subsequent trasnformations need to be able to handle whatever output the previous transformation gives. Could be a String or an Expr I suppose.

2 Likes

Thanks Mason for your feedback!

So I should start by saying that I find ReplMaker absolutely great and used it a lot and will continue to do so. I actually had the intention (but forgot) to mention in my previous post that SwapLiterals could be used as a backend of a ReplMaker mode.

I hope you didn’t take my comment as a critique of ReplMaker. I was refuting the idea that “this existed before under ReplMaker”: 1) SafeREPL uses a different mechanism than REPL modes, with a different trade-off, and 2) the “BigInt-mode” from ReplMaker's README isn’t on feature parity with what is avalaible in SwapLiterals (which can replace more literals, and is currently more robust for BigInt). The problem with "1" that I gave as an exemple was including the string quotes:

BigJulia> "1"
ERROR: Base.Meta.ParseError("cannot juxtapose string literal")
[...]

When I said

it strictly refers to the fact that SwapLiterals and ReplMaker have orthogonal purposes and must no be merged and developed in the same project. But they could/should definitely be combined by users.

I will answer that indeed SwapLiterals should be used with ReplMaker by people wanting a separate mode, but I’m wanting to explore in SafeREPL this possibility of “hot-modifying the existing, standard julia repl mode”. I totally might not foresee some problems, but applying code transformation in the normal Julia mode vs. from within a separate REPL mode doesn’t make me more uncomfortable at all.
Moreover, the ability to have REPL modes is fantastic, but I personally prefer them for non-julia code; when writing julia code, I’d rather use the julia-mode.

Well, I don’t really understand this one. Surely the package won’t force people to enable this visual indicator that code transformation is at play, but they are responsible for themselves, and the README already guides them through the steps. Also, they already have to opt-in by installing the package and using it, it’s up to them to choose the specific options.

Absolutely, I was unprecise as I didn’t want to stress there the distinction between these two new packages.

I sincerely apologize in advance if this package wastes people’s time… If this happens, a warning could be put in the README. But I will be more precise: using ReplMaker is dead-simple, so I was not implying that anyone around can not deal with REPL modes. But for the use-case we discuss here, the result is objectively not as smooth as with “hot-modifying” the standard REPL. Again, simply getting into help-mode from another mode for example requires by default to go back through the normal mode first. REPL-modes revolve around the julia> mode. It’s certainly possible to work-around that, but not for a beginner; and I couldn’t tell them how off the top of my head, even though I studied multiple times the REPL code.

So the intention with SafeREPL is to explore Julia where literals loose their absolute meaning. Yes, they have defaults, but this can be changed, in a trivial way. I typically wouldn’t use a REPL mode where integer literals are BigInt, simply because I wouldn’t bother, writing big(2) here and there is lower overhead than activating a REPL mode all the time just in case. But using SafeREPL in startup.jl offers ““safety”” by default with an overhead that I can tolerate so far (the overhead is being bothered by methods which don’t accept BigInt).

Yes, that will definitely be useful and worth implementing if there is demand for it. ReplMaker could be the “coordination” I was alluding to.

And as a last note, I’m also curious to see how this package is received by people who really really don’t like that Julia is “unsafe/fast” by default and think that this should be changed for a greater success of the language.

3 Likes

I am not sure how that will play out, but it will be certainly handy to be able to just refer those people to this package instead of yet another discussion of that design choice.

6 Likes

I now added a small snippet in the README to show how to use SwapLiterals as the backend of a ReplMaker mode.

Since you emphasize “Safe”, maybe rather change to https://github.com/JuliaMath/DecFP.jl/releases/tag/v1.0.0 for floats. It’s not “Big” too (in the sense of BigInt), but at least it’s floating point so I guess doesn’t matter.

DecFP is supported, as shown in the README, but I’d rather not have it has default, as it’s not clear at this point that this is a better option than BigFloat (I’m not clear either on whether Float64 should be replaced by default).
So I prefer not to add this dependency (which also is not pure Julia). Also, @time using DecFP is half a second for me, 30 times slower than @time using SafeREPL: I prefer to have using SafeREPL add a very lightweight cost by default when put in startup.jl.

Then you should use/consider (or since incomplete Rationals instead?):

julia> @time using Decimals
  0.035799 seconds (27.75 k allocations: 1.761 MiB)

At runtime it’s slower than DecFP, and people could always change to it or back to regular floats.

EDIT: I see a TODO for exponentiation, but since DecFP 1.0 and DecFP_jll, I think it should be added to Julia’s standard library (people could also add it, themselves, to the system image for faster startup): https://github.com/JuliaMath/DecFP.jl/commit/8eca99ec2350b8e4180d01becaf66b08944f6f0b

$ julia -O0

julia> @time using DecFP_jll
  0.173529 seconds (253.11 k allocations: 13.753 MiB)

Maybe useless to know this fast, as I think you also need the wrapper that includes it (unclear to me it can also do with -O0):

julia> @time using DecFP
  0.472506 seconds (574.17 k allocations: 32.659 MiB)

Good point for Decimals. However, I’m still not convinced that using decimal numbers by default rather than binary numbers is better; I guess only user-feedback would tell.
As a side note, the current printing for Decimals seems like a total no-go for the use-case of being the default in SafeREPL:

julia> Decimal(1.2)
Decimal(0, 12, -1)

Can I use it and ohmyrepl at the same time?

I didn’t tried both packages together till now, but I don’t see why not, and 10 seconds of testing didn’t reveal any problem.

When I think about it, since your throwing speed out of the window anyway (with BigInt replacing Int), Rationals would be the best substitute for floats (just as Perl 6, now renamed Raku, uses by default, unlike Perl 5; new version also added multiple dispatch). And on display changing to Decimal or something, or some other way to get in decimal form, without like this:

julia> 1.1+1  # for your package, otherwise showing 2.1
2.099999999999999999999999999999999999999999999999999999999999999999999999999986

It just needs to be fixed in the package? I thought, maybe they want to be explicit, it’s a different type (I see pros and cons, but even DecFP doesn’t), unlike:

julia> BigInt(1)
1

FYI:

julia> Dec64(1.1)  # This actually works, without your package, which has this side-effect:
ERROR: StackOverflowError:
Stacktrace:
 [1] Dec64(::BigFloat) at /home/pharaldsson_sym/.julia/packages/DecFP/MVfs9/src/DecFP.jl:117
 [2] convert(::Type{Dec64}, ::BigFloat) at ./number.jl:7
 ... (the last 2 lines are repeated 39990 more times)
 [79983] Dec64(::BigFloat) at /home/pharaldsson_sym/.julia/packages/DecFP/MVfs9/src/DecFP.jl:117

Nice package! Does this allow overriding the show method of a custom type in our deps without warnings?

For example, in one of the dependencies of my AcuteML packages (EzXML), the show method for Node type is uninformative and opinionated. I always wanted to override its show method without warnings.

Default show for one of the types:

EzXML.Node(<ELEMENT_NODE[root]@0x00007fd9f2a1b5f0>)

Does SafeREPL allow removing this pointer from the show?

Again, I don’t know that this would be best, it really depends on the use case. I think it is useful to have defaults, but it must be emphasized that the user has the choice over what is replaced by what. It’s clear that the defaults can’t satisfy everyone, so it seems premature to engage too much in discussing what are the best defaults before knowing more what users use instead of the defaults.
Personally, I typically wouldn’t want to use rationals by default, as I usually understand better the number written in decimal form; I know you suggested:

but this is out of the scope of the package, which is not in the business of modifying the output nor overriding show methods, nor defining new number types.

Yes, that’s why I said “currently”, but I would guess the authors have a good reason to have their current printing.

It’s very unlikely that SafeREPL will depend on a package for its defaults when decent types are available in Base for that purpose: it both reduce loading time, and cognitive overhead for users as they are more likely to know Base number types than these provided in a package. I could imagine using a type provided in a package if the majority of users would report using that type as their default.

I didn’t know this one, but this is indicated in the “Caveat” section of the README: many many functions assume that the user will enter a Float64 or an Int because this is the type of un-annotated literals; so it’s unavoidable to encounter such errors while using SafeREPL. I hope the situation will improve, but in the meantime you can $-quote your numbers in such situations, e.g. Dec64($1.1), or convert then back to Int/Float64, e.g. Dec64(Float64(1.1)).

3 Likes

Thanks!

No, this package does nothing to the output of the REPL, and doesn’t override package’s show methods.

In this particular case, if show(::IO, ::MIME"text/plain", x::TheType) is not defined, you might have a chance to define it outside of the package without warning. Also, without changing the default, the maintainers of the package might accept more easily a patch which change the printing depending on the value of an IOContext attribute, which you can then set in the REPL’s iocontext (since Julia 1.4).

2 Likes

For reference, there’s already a PR for this (IIUC), by @amellnik: https://github.com/JuliaMath/Decimals.jl/pull/24.

1 Like