REPL, copy/paste and auto indent in julia 0.7

@rfourquet

I am sorry I found something more directly related to auto-indent.

If the pre-indented code is indented using space everything is fine.
But if the spaces are replaced by real tabulators \t the pastinf is still messed up.

function test()
	println("1")
	println("2")
	println("3")
	println("4")
	println("5")
	println("6")
	println("7")
	println("8")
	println("9")
	println("10")
	println("11")
	println("12")
	println("13")
	println("14")
	println("15")
	println("16")
	println("17")
	println("18")
	println("19")
	println("20")
	println("21")
	println()
	println()
	println()
	println()
	println()
	println()
	println()
	println()
	println()
	println()
	println()
	println()
end

where indentation is \t this is the result:

julia> function test()
           println("1")
       println("2")
           println("3")
       println("4")
           println("5")
       println("6")
           println("7")
       println("8")
           println("9")
       println("10")
           println("11")
       println("12")
           println("13")
       println("14")
           println("15")
       println("16")
           println("17")
       println("18")
           println("19")
       println("20")
           println("21")
       println()
           println()
       println()
           println()
       println()
           println()
       println()
           println()
       println()
           println()
       println()
           println()
       end
test (generic function with 1 method)

:disappointed_relieved:

I can’t reproduce this problem for now, but does it depend on the patch from REPL: disable auto-indent when code is likely being pasted (fix #25186) by rfourquet · Pull Request #29129 · JuliaLang/julia · GitHub ? or does it happen in any case?

It is only in the patched version.
0.6.2 indents nicely.
0.7 and 1.0 indents according to the auto-indent problem.
Only the patched version which indents erratic like above shown.

My guess is, that in function

function edit_insert(s::PromptState, c)
...
# delete auto-inserted spaces

there is some missing space/tab distinction.

So I can’t reproduce because tmux doesn’t handle tabs with copy-mode. I really don’t understand why in your example the indentation is wrong only every second line. Does it change if you increase/decrease the value of auto_indent_time_threshold ?

I think we can’t do something about it in the edit_insert function, as at that time any inserted tab will have already been transformed into a series of spaces. When tab is typed at the REPL, a specific function is called, which decides whether to complete code or to indent (by inserting tabs). So I guess to solve this problem we would have to update the edit_tab function with the same logic as in the PR, which I’m quite relunctant doing: I would expect tabs in source code to be rare, so I’d rather say: if you terminal doesn’t support bracketed paste and you need to paste code containing tabs, just disable (temporarily) the auto-indent feature. We could add a keyboard shortcut to switch the feature on/off.

I found another implementation based on your timing idea but without changing the buffer after auto-indent already occured. Your advice on how to debug REPL while in the REPL worked perfect! Thanks!
So my proposal is based on your code in rf/autoindentpaste and prevents auto-indent beforehand, which solves the issues with tabulators:

REPL.jl:

mutable struct Options
    hascolor::Bool
    extra_keymap::Union{Dict,Vector{<:Dict}}
    # controls the presumed tab width of code pasted into the REPL.
    # Must satisfy `0 < tabwidth <= 16`.
    tabwidth::Int
    # Maximum number of entries in the kill ring queue.
    # Beyond this number, oldest entries are discarded first.
    kill_ring_max::Int
    region_animation_duration::Float64
    beep_duration::Float64
    beep_blink::Float64
    beep_maxduration::Float64
    beep_colors::Vector{String}
    beep_use_current::Bool
    backspace_align::Bool
    backspace_adjust::Bool
    confirm_exit::Bool # ^D must be repeated to confirm exit
    auto_indent::Bool # indent a newline like line above
    auto_indent_tmp_off::Bool # switch auto_indent temporarily off if copy&paste
    auto_indent_bracketed_paste::Bool # set to true if terminal knows paste mode
    # cancel auto-indent when next character is entered within this time frame :
    auto_indent_time_threshold::Float64
end

Options(;
        hascolor = true,
        extra_keymap = AnyDict[],
        tabwidth = 8,
        kill_ring_max = 100,
        region_animation_duration = 0.2,
        beep_duration = 0.2, beep_blink = 0.2, beep_maxduration = 1.0,
        beep_colors = ["\e[90m"], # gray (text_colors not yet available)
        beep_use_current = true,
        backspace_align = true, backspace_adjust = backspace_align,
        confirm_exit = false,
        auto_indent = true,
        auto_indent_tmp_off = false,
        auto_indent_bracketed_paste = false,
        auto_indent_time_threshold = 0.005) =
            Options(hascolor, extra_keymap, tabwidth,
                    kill_ring_max, region_animation_duration,
                    beep_duration, beep_blink, beep_maxduration,
                    beep_colors, beep_use_current,
                    backspace_align, backspace_adjust, confirm_exit,
                    auto_indent, auto_indent_tmp_off, auto_indent_bracketed_paste, auto_indent_time_threshold)

LineEdit.jl:

function edit_insert(s::PromptState, c)
    push_undo(s)
    buf = s.input_buffer

    if ! options(s).auto_indent_bracketed_paste
        pos=position(buf)
        if pos > 0
            if buf.data[pos] != _space && string(c) != " "
                options(s).auto_indent_tmp_off = false
            end
            if buf.data[pos] == _space
                #tabulators are already expanded to space
                #this expansion may take longer than auto_indent_time_threshold which breaks the timing
                s.last_newline = time()
            else
                #if characters after new line are coming in very fast
                #its probably copy&paste => switch auto-indent off for the next coming new line
                if ! options(s).auto_indent_tmp_off && time() - s.last_newline < options(s).auto_indent_time_threshold
                    options(s).auto_indent_tmp_off = true
                end
            end
        end
    end

    str = string(c)
    edit_insert(buf, str)
    offset = s.ias.curs_row == 1 || s.indent < 0 ?
        sizeof(prompt_string(s.p.prompt)) : s.indent
    if !('\n' in str) && eof(buf) &&
        ((position(buf) - beginofline(buf) + # size of current line
          offset + sizeof(str) - 1) < width(terminal(s)))
        # Avoid full update when appending characters to the end
        # and an update of curs_row isn't necessary (conservatively estimated)
        write(terminal(s), str)
    else
        refresh_line(s)
    end
end
...

function edit_insert_newline(s::PromptState, align::Int = 0 - options(s).auto_indent)
    push_undo(s)
    buf = buffer(s)
    autoindent = align < 0
    if autoindent && ! options(s).auto_indent_tmp_off
        beg = beginofline(buf)
        align = min(something(findnext(_notspace, buf.data[beg+1:buf.size], 1), 0) - 1,
                    position(buf) - beg) # indentation must not increase
        align < 0 && (align = buf.size-beg)
    else
        align = 0
    end
    edit_insert(buf, '\n' * ' '^align)
    refresh_line(s)
    # updating s.last_newline should happen after refresh_line(s) which can take
    # an unpredictable amount of time and makes "paste detection" unreliable
    if ! options(s).auto_indent_bracketed_paste
        s.last_newline = time()
    end
end

...

function bracketed_paste(s; tabwidth=options(s).tabwidth)
    options(s).auto_indent_bracketed_paste = true
    ps = state(s, mode(s))
    input = readuntil(ps.terminal, "\e[201~")
    input = replace(input, '\r' => '\n')
    if position(buffer(s)) == 0
        indent = Base.indentation(input; tabwidth=tabwidth)[1]
        input = Base.unindent(input, indent; tabwidth=tabwidth)
    end
    return replace(input, '\t' => " "^tabwidth)
end

test/lineedit.jl was not changed from your rf/autoindentpaste version.

Tested with Windows 7+10, with terminals cmd.exe, powershell.exe and cygwin terminal.

Another special case missed. If you paste something with empty lines with multiple tabs/spaces only, it will do wrong indentation for the next line.
The fix of my code is currently compiled and after testing will be pasted here.

Above code now latest version. It accepts e.g.:

function test()

							x
                           
x
x
	y=1
	z=1
	x=1
	y=1
    	z=1
	x=1
	y=1
    
end

In the last statement “z=1” there is a hidden tab, which is not visible here, but will be copied&pasted into the REPL.