Exclude pattern from the REPL history

General Usage
1

Is there any way to prevent some inputs that match a specific pattern from being stored in the REPL history?

2

I don’t know of any easy way to do this, why do you want to do this? You can either switch off history entirely or increase the number of saved history items. I also made a package that will allow you to search for specific patterns in the history, but they still get saved, the link to it is GitHub - GHTaarn/History.jl: History functionality in the Julia REPL similar to what bash provides

2 Likes
3

Not sure how useful this is, but:

  • there are a number of packages for working with the REPL history (though not exactly what you want), they might help or their code might be useful. See REPLHistory, History and even OhMyREPL.

  • Julia’s REPL history is just a text file which you can directly access. It’s usually at ~/.julia/logs/repl_history.jl but can be found with REPL.find_hist_file().

  • the REPL’s least remembered modes are forward and reverse history search (^S and ^R respectively).

Hope something there is useful.

1 Like
4

Thank you @GHTaarn @screw_dog

An Emacs program I wrote sends a specific pattern of Julia code to the REPL. Because they are not reusable at all, I want to exclude them from the history to avoid clutter.

I think a functionality to exclude pattern from history may be useful in some cases in general. For example in bash, there’s HISTIGNORE environment variable for this purpose.

3 Likes
5

If you use history from past Julia sessions, then you could try to increase the number of history items that Julia saves and create a startup script that filters the history file through grep -v before Julia is started. I am not familiar enough with REPL internals to know how to do exactly what you are looking for.

6

Not op, but I frequently have situations where eg I’m adding an API key or something to the environment, and I’d rather it not be stored in plain text in my REPL history.

I usually try to remember to set the environment variable before starting the repl or delete manually with sed, but it would be nice to have a way to exclude it from the beginning

2 Likes
7

one trick for that is to use e.g.

ENV["PASSWORD"] = Base.shred!(s -> read(s, String), Base.getpass("Password"))

Then you can paste the key into the prompt instead of the REPL. (Idk if there’s actually a security advantage of Base.getpass over Base.prompt in this case, since it’s going to an env variable anyway, but I guess it doesn’t hurt).

2 Likes
8

I found that I can exclude a specific pattern from history by adding the following to my startup.jl. This redefines REPL.add_history. The most part is just a copy of original REPL.add_history function definition from REPL.jl and I just added a line to check a pattern to skip. I know a monkey patching like this is something to be avoided in general. This works but I appreciate a better solution.

atreplinit() do repl
    @eval begin
        import REPL: add_history, REPLHistoryProvider, PromptState, LineEdit, mode_idx

        function add_history(hist::REPLHistoryProvider, s::PromptState)
            str = rstrip(String(take!(copy(s.input_buffer))))

            occursin(r"#PATTERN_TO_EXCLUDE", str) && return

            isempty(strip(str)) && return
            mode = mode_idx(hist, LineEdit.mode(s))
            !isempty(hist.history) &&
                isequal(mode, hist.modes[end]) && str == hist.history[end] && return
            push!(hist.modes, mode)
            push!(hist.history, str)
            hist.history_file === nothing && return
            entry = """
                # time: $(Libc.strftime("%Y-%m-%d %H:%M:%S %Z", time()))
                # mode: $mode
                $(replace(str, r"^"ms => "\t"))
                """
            try
                seekend(hist.history_file)
            catch err
                (err isa SystemError) || rethrow()
                hist_open_file(hist)
            end
            print(hist.history_file, entry)
            flush(hist.history_file)
            nothing
        end
    end
end
1 Like
9

It would be a nice PR to add a hook to add_history that lets people exclude patterns more easily.

4 Likes
10

@StefanKarpinski What would be a good design of “hook” for this? I am thinking something like below.

atreplinit() do repl
    repl.options.hist_ignore_pattern = r"PATTERN"
end
2 Likes
11

Might be nice if it could also take a vector of patterns and then apply any(pat-> contains(input, pat), patterns)

Not sure what the performance implications would be…

3 Likes
12

It’s interactive, I’m pretty sure performance doesn’t matter at all.

3 Likes
13

To anchor intuitions here, this is how long it takes PCRE, on an M2 Macbook, to find a line at the end of the Project Gutenberg edition of the King James Bible, which I happen to have around to do like-for-like benchmarks with a particular research paper.

julia> @time match(r"donations to the Project Gutenberg", kjv)
  0.000282 seconds (3 allocations: 176 bytes)
RegexMatch("donations to the Project Gutenberg")

You would need at minimum of three figures worth of regex to notice the time it would take to run them all on a line of REPl inpud.

2 Likes
14

… Regex search has very instance-dependent performance. The intuition here is mostly relevant to this simple (and long) Regex.

Specifically, the search here could be sublinear (not even reading the entire Bible) - amen!

1 Like
15

Okay… but like I said, I’m using KJV for some benchmarks and that isn’t the only one.

julia> @time collect(eachmatch(r"(Abraham|Sarah|Moses|Matthew|Tubalcain)", kjv));
  0.006095 seconds (4.63 k allocations: 404.047 KiB)

julia> length(collect(eachmatch(r"(Abraham|Sarah|Moses|Matthew|Tubalcain)", kjv)))
1157

It used to be pretty straightforward to force PCRE into catastrophic backtracking (not that this example would), but most of those cases have been eliminated, you really gotta work for it now.

I really don’t think there is a reasonable number of exclusion patterns which someone might apply to a REPL line which would cause them to notice the time it takes to do so.

1 Like
16

OK. Thank you all. I will file a PR. The option would accept any of a string, a regexp, or a list of those.

2 Likes
17

Now I am about to start working on this. This is my first time to contribute to a stdlib and I’d like to know the right way for it.

I thought ]dev REPL would work but it didn’t. I got an error like The following package names could not be resolved: REPL.

I also tried Revise.track(REPL) but an error again invalid redefinition of constant REPL.Options.

Are there a document for the steps of stdlib development?

18

I don’t have much experience with this, but I started by forking GitHub - JuliaLang/julia: The Julia Programming Language and cloning my fork. I then created a new branch in that clone and edited the relevant .jl files. Ideally I guess that one should make the project, but I haven’t succeeded with that yet. Instead I downloaded the nightly build, replaced the relevant .jl files with my edited versions and started the nightly build to test them.

This is of course not very practical if you have big changes, but I have not had that yet.