REPL, copy/paste and auto indent in julia 0.7

General Usage
1

I am referring here to this issue:

https://github.com/JuliaLang/julia/issues/25186

It is not only a cosmetic annoyance as it may seem from the screenshots in the issue above.

If you have a large code block, a few hundred lines, the REPL takes very very long to digest what has been pasted into it. It seems that the whitespaces of the already indented pasted code and the auto indent is summing up in a problematic way.

Currently I try to solve this issue in a workaround but I fail with all attempts. I am running the latest julia (0.7-dev from today, windows 10, all 64bit), tried with powershell and cmd. With windows bash, julia does not start:

root@PC:/mnt/c/Windows/System32# /mnt/c/Users/myUserName/AppData/Local/Julia-0.7.0-DEV/bin/julia.exe
ERROR: error initializing stdin in uv_pipe_open: invalid argument (EINVAL -4071)
root@PC:/mnt/c/Windows/System32#

Running cygwin mintty and starting julia from here works fine. So thats for now my workaround.

A solution to the auto indent problem has been comitted:

https://github.com/JuliaLang/julia/pull/26605/commits/771255929034ae69984cd19e69f7d0d97c985889

But some question are still open:

How can I use the new option? Is there a REPL command to switch options on/off? Is it a command line switch?

When I start julia just by double click the installed icon under windows, the default terminal, which is used is CMD, right? Is this possible to change? Or do I have to start the terminal first and run julia in this terminal by entering the path to the julia.exe on the terminals command line?

2

I will merge this PR. To de-activate the feature, you can put this in your config file:

using REPL

atreplinit() do repl
    if isdefined(repl, :options)
        repl.options = REPL.Options(auto_indent=false)
    end
end

Once the REPL is already running, you can switch the option with Base.active_repl.options.auto_indent = false.

2 Likes
3

Dang I have got to remember to do this. Currently waiting on a long function to finish pasting…

4

People experiencing the issue with pasting are invited to try a proposed fix from REPL: disable auto-indent when code is likely being pasted (fix #25186) by rfourquet · Pull Request #29129 · JuliaLang/julia · GitHub (this unfortunately requires recompiling Julia), and give feedback there on whether the fix works.

5

@rfourquet

I tried to build julia using
https://github.com/JuliaLang/julia/blob/master/README.windows.md

but failed (with rf/autoindentpaste and master same result):

$ make -j 4
make -C scratch/openblas-e8a68ef261a33568b0f0cf53e0e2287e9f12e69e/ CC=x86_64-w64-mingw32-gcc -m64 FC=x86_64-w64-mingw32-gfortran -m64 LD=x86_64-w64-mingw32-ld RANLIB=x86_64-w64-mingw32-ranlib TARGET= BINARY=64 USE_THREAD=1 GEMM_MULTITHREADING_THRESHOLD=50 NUM_THREADS=16 NO_AFFINITY=1 DYNAMIC_ARCH=1 INTERFACE64=1 SYMBOLSUFFIX=64_ LIBPREFIX=libopenblas64_ OSNAME=WINNT CROSS=1 HOSTCC=gcc CROSS_SUFFIX=x86_64-w64-mingw32- CFLAGS= -O2 -fno-asynchronous-unwind-tables FFLAGS= -O2  LDFLAGS=  MAKE_NB_JOBS=0
patching file lib/Target/X86/X86CallingConv.td
patching file lib/Target/X86/X86ISelLowering.cpp
Hunk #1 succeeded at 3067 (offset 5 lines).
Hunk #2 succeeded at 3546 (offset 5 lines).
patching file test/CodeGen/X86/win64-byval.ll
[  0%] Built target LLVMTableGen
[  0%] Built target LLVMDemangle
[  0%] Built target CREATE_LLVM_NATIVE
...
[ 82%] Built target LLVMX86Info
[ 82%] Built target LLVMX86Utils
[ 82%] Linking CXX executable ../../bin/llvm-cfi-verify.exe
[ 82%] Building CXX object lib/Target/X86/CMakeFiles/LLVMX86CodeGen.dir/X86FastISel.cpp.obj
[ 82%] Built target llvm-cfi-verify
[ 82%] Building CXX object lib/Target/X86/CMakeFiles/LLVMX86CodeGen.dir/X86ISelLowering.cpp.obj
[ 82%] Linking CXX static library ../../libLLVMX86CodeGen.a
[ 85%] Built target LLVMX86CodeGen
[ 85%] Linking CXX shared library ../../bin/LLVM.dll
../../lib/libLLVMAMDGPUCodeGen.a(AMDGPUISelDAGToDAG.cpp.obj):AMDGPUISelDAGToDAG.cpp:(.text$_ZN4llvm16SelectionDAGISel22getIncludePathForIndexEj+0x0): Mehrfachdefinition von »llvm::SelectionDAGISel::getIncludePathForIndex(unsigned int)«
../../lib/libLLVMSelectionDAG.a(SelectionDAGISel.cpp.obj):SelectionDAGISel.cpp:(.text$_ZN4llvm16SelectionDAGISel22getIncludePathForIndexEj+0x0): hier zuerst definiert
../../lib/libLLVMAMDGPUCodeGen.a(AMDGPUISelDAGToDAG.cpp.obj):AMDGPUISelDAGToDAG.cpp:(.text$_ZN4llvm16SelectionDAGISel18getPatternForIndexEj+0x0): Mehrfachdefinition von »llvm::SelectionDAGISel::getPatternForIndex(unsigned int)«
../../lib/libLLVMSelectionDAG.a(SelectionDAGISel.cpp.obj):SelectionDAGISel.cpp:(.text$_ZN4llvm16SelectionDAGISel18getPatternForIndexEj+0x0): hier zuerst definiert
collect2: Fehler: ld gab 1 als Ende-Status zurĂĽck
make[4]: *** [tools/llvm-shlib/CMakeFiles/LLVM.dir/build.make:207: bin/LLVM.dll] Fehler 1
make[3]: *** [CMakeFiles/Makefile2:16433: tools/llvm-shlib/CMakeFiles/LLVM.dir/all] Fehler 2
make[2]: *** [Makefile:150: all] Fehler 2
make[1]: *** [/home/oheil/julia/deps/llvm.mk:449: scratch/llvm-6.0.1/build_Release/build-compiled] Fehler 2
make[1]: *** Es wird auf noch nicht beendete Prozesse gewartet....
make[3]: *** [Makefile.L3:977: dtrmm_outucopy_SKYLAKEX.obj] Interrupt
make[3]: *** [Makefile.L3:968: dtrmm_ounncopy_SKYLAKEX.obj] Interrupt
make[3]: *** [Makefile.L3:971: dtrmm_olnucopy_SKYLAKEX.obj] Interrupt
make[3]: *** [Makefile.L3:974: dtrmm_olnncopy_SKYLAKEX.obj] Interrupt
make[2]: *** [Makefile:150: libs] Interrupt
make[1]: *** [/home/oheil/julia/deps/blas.mk:101: scratch/openblas-e8a68ef261a33568b0f0cf53e0e2287e9f12e69e/build-compiled] Interrupt
make: *** [Makefile:60: julia-deps] Interrupt

Any hints what I could do?

6

I was not able to compile julia 1.0 on windows in 3 different machines, 2 Win7 , 1 WIN 10, so I used julia v0.7.0.

Building 0.7 using cygwin cross build worked as described.

=> autoindent and copy&paste into REPL produces bad multiple indentations.

Now editing the 3 files as proposed in https://github.com/JuliaLang/julia/tree/rf/autoindentpaste
and rebuilding did solve the issue for small code snippets. Copy&Pasting with small code snippets produced nice proper indented code.

With larger functions it is still getting messed up. E.g. if you paste the following function into the REPL:

function xxxxxhist_from_file(hp, file, path)
    hp.history_file = file
    seek(file, 0)
    countlines = 0
    while true
        mode = :julia
        line = hist_getline(file)
        isempty(line) && break
        countlines += 1
        line[1] != '#' &&
            error(invalid_history_message(path), repr(line[1]), " at line ", countlines)
        while !isempty(line)
            m = match(r"^#\s*(\w+)\s*:\s*(.*?)\s*$", line)
            m === nothing && break
            if m.captures[1] == "mode"
                mode = Symbol(m.captures[2])
            end
            line = hist_getline(file)
            countlines += 1
        end
        isempty(line) && break
        # Make sure starts with tab
        line[1] == ' '  &&
            error(munged_history_message(path), countlines)
        line[1] != '\t' &&
            error(invalid_history_message(path), repr(line[1]), " at line ", countlines)
        lines = String[]
        while !isempty(line)
            push!(lines, chomp(line[2:end]))
            eof(file) && break
            ch = Char(Base.peek(file))
            ch == ' '  && error(munged_history_message(path), countlines)
            ch != '\t' && break
            line = hist_getline(file)
            countlines += 1
        end
        push!(hp.modes, mode)
        push!(hp.history, join(lines, '\n'))
    end
    seekend(file)
    hp.start_idx = length(hp.history)
    return hp
end

Everything is fine until the first end in the code. Afterwards autoindentation is on again.
If you mask the first and the second end, e.g. as xxxend, it works until the commend # Make sure starts with tab, and afterwards things getting messed up again.

After some more copy&pasting it seems that it is not the end or the comment, but maybe after some time autoindentation switches on again.

So the proposed solution seems to be a starter but special cases still needs to be addressed.

7

Thanks of lot for trying that branch out! Would you mind changing the time-threshold parameter like indicated in the PR, and trying again? You can do for example Base.active_repl.options.auto_indent_time_threshold = 0.01 (or 0.02) at the REPL and then paste the code to see if it helps.

8

@rfourquet

I tried now for hours but I can’t find a good solution.

First I did play around with values for auto_indent_time_threshold but it doesn’t help. It moves just the bad auto-indent to later or earlier lines. After some time, later or sooner depending on auto_indent_time_threshold, auto-indent is jumping in and produces more and more indentations.

So I tried th following in lineedit.jl:

function edit_insert(s::PromptState, c)
    if time() - s.last_newline < options(s).auto_indent_time_threshold
        # the first inserted character since last newline was inserted very fast,
        # so it must have been inserted automatically (e.g. "paste" which is not
        # recognized as so)
        buf = buffer(s)
        pos = position(buf)
        nl = findprev(c -> c == _newline, buf.data, pos)::Int
        seek(buf, nl)
        # delete auto-inserted spaces
        edit_splice!(buf, nl => pos)
        #s.last_newline = -Inf
	options(s).auto_indent = false    ####### EXPERIMENTAL CODE HERE #####
        refresh_line(s) # necessary, as the code below doesn't do it necessarily
    end
    ...
end

Which produces perfect results whatever you paste into the REPL.
But clearly

options(s).auto_indent

is not switched on again. Thats where I failed during the last hours.

My idea was, to set
options(s).auto_indent = true
when the prompt “julia>” is written on the terminal, but I failed to find the line of code where to do it.

Adding another threshold
auto_indent_on_time_threshold = 0.1
and doing

function edit_insert(s::PromptState, c)
	if time() - s.last_newline > options(s).auto_indent_on_time_threshold
		options(s).auto_indent = true
	end
       ....

brought me back to the initial problem, that auto-indent is jumping in with larger code snippets pasted into the REPL, now sooner or later depending on two option values.

Here I have to give up to find a solution. Maybe my ideas are not so bad and you are able to make them real. I am happy to test it for you.

1 Like
9

I have put the lines of code into Julia user config file as suggested by @rfourquet in his Apr 9 post above and it works fine.

10

Thanks for all the time you spent on this.

I can reproduce the problem to a certain extent (with tmux on Linux), in that when the code being pasted is longer, there are more chances that auto-indentation kicks in.

Setting auto_indent_time_threshold = 0.03 is a sweet spot for me to avoid most problems. A bigger value means that auto-indent can be temporarilly disabled even when I’m not pasting (e.g. typing ENTER and then another character quickly). With a value of 0.03, all code that I paste which fits on my screen gets correct indentation.

I don’t really understand why this “fix” doesn’t work more reliably. I have an idea or two to try to mitigate the problem, but this is even more hacky than the auto_indent_time_threshold fix, so I’m a bit relunctant…

11

OK, I found a patch which makes it much more reliable for me: compared to the PR code, in the edit_insert_newline function, exchange the order of the last two lines, i.e. change it to:

     refresh_line(s)
     align > 0 && (s.last_newline = time())

You don’t need to recompile the whole Julia to change that, you can overwrite the function with:

import REPL
import REPL.LineEdit: edit_insert, options, buffer, position, char_move_left, char_move_right, edit_splice!, push_undo, edit_insert, prompt_string, beginofline, terminal, refresh_line, PromptState, width, edit_insert_newline, _notspace, _newline

function edit_insert_newline(s::PromptState, align::Int = 0 - options(s).auto_indent)
    push_undo(s)
    buf = buffer(s)
    if align < 0
        beg = beginofline(buf)
        align = min(something(findnext(_notspace, buf.data[beg+1:buf.size], 1), 0) - 1,
                    position(buf) - beg) # indentation must not increase
        align < 0 && (align = buf.size-beg)
    end
    edit_insert(buf, '\n' * ' '^align)
    refresh_line(s)
    align > 0 && (s.last_newline = time())
end
1 Like
12

Confirmed, this works very well regarding the indentation and with the value
auto_indent_time_threshold = 0.005

But there is another cosmetic problem, which is now somehow worse than before.
If you paste code into REPL which has more lines than the terminal currently shows, the first line of the pasted code is duplicated.
E.g.:

Open a fresh Julia REPL.
Decrease the terminal window to 22 lines
Paste the following code (23 lines code + 1 extra line for last \n)

function test()
    println("1")
    println("2")
    println("3")
    println("4")
    println("5")
    println("6")
    println("7")
    println("8")
    println("9")
    println("10")
    println("11")
    println("12")
    println("13")
    println("14")
    println("15")
    println("16")
    println("17")
    println("18")
    println("19")
    println("20")
    println("21")
end

into the window and scroll up. What you see is:

julia> function test()
julia> function test()
julia> function test()
           println("1")
           println("2")
           println("3")
           println("4")
           println("5")
           println("6")
           println("7")
           println("8")
           println("9")
           println("10")
           println("11")
           println("12")
           println("13")
           println("14")
           println("15")
           println("16")
           println("17")
           println("18")
           println("19")
           println("20")
           println("21")
       end
test (generic function with 1 method)

This gets worse with more lines.

If you try the same in a REPL without the auto-indent patch, it is also a problem but not as much as with the patch. It seems to be only a display problem, the code itself is fine.

Should this be reported as a new issue? I didn’t check, maybe its already reported.

1 Like
13

I can be more precise about this new problem:

If you paste larger code into the REPL what happens is:
After some number of lines, typicaly more than the terminal window can show at once, the complete pasted code is printed newly from the beginning. This happens multiple times depending on the number of code lines pasted, until the last time, when the complete code is shown as one block.
Example:

julia> function test()
julia> function test()
julia> function test()
           println("1")
julia> function test()
           println("1")
julia> function test()
           println("1")
           println("2")
julia> function test()
           println("1")
           println("2")
julia> function test()
           println("1")
           println("2")
           println("3")
julia> function test()
           println("1")
           println("2")
           println("3")
julia> function test()
           println("1")
           println("2")
           println("3")
           println("4")
julia> function test()
           println("1")
           println("2")
           println("3")
           println("4")
julia> function test()
           println("1")
           println("2")
           println("3")
           println("4")
           println("5")
julia> function test()
           println("1")
           println("2")
           println("3")
           println("4")
           println("5")
julia> function test()
           println("1")
           println("2")
           println("3")
           println("4")
           println("5")
           println("6")
julia> function test()
           println("1")
           println("2")
           println("3")
           println("4")
           println("5")
           println("6")
julia> function test()
           println("1")
           println("2")
           println("3")
           println("4")
           println("5")
           println("6")
           println("7")
julia> function test()
           println("1")
           println("2")
           println("3")
           println("4")
           println("5")
           println("6")
           println("7")
julia> function test()
           println("1")
           println("2")
           println("3")
           println("4")
           println("5")
           println("6")
           println("7")
           println("8")
julia> function test()
           println("1")
           println("2")
           println("3")
           println("4")
           println("5")
           println("6")
           println("7")
           println("8")
julia> function test()
           println("1")
           println("2")
           println("3")
           println("4")
           println("5")
           println("6")
           println("7")
           println("8")
           println("9")
           println("10")
           println("11")
           println("12")
           println("13")
           println("14")
           println("15")
           println("16")
           println("17")
           println("18")
           println("19")
           println("20")
           println("21")
       end
test (generic function with 1 method)
14

It is the same in REPL 0.6.2, so it seems something older and not related to the auto-indent thing.

15

Thanks so much for trying that out! So will update the PR accordingly.
I also noticed the other problem you are mentioning, I’m not clear to which extent the patch here makes the matter worse, but the REPL is definitily buggy when the input spans more than a screen, and that needs to be fixed anyway.

1 Like
16

And it is probably Windows specific just like the auto-indent problem.
It is not occuring in the cygwin terminal.

17

Yes its not really worse, just different because of the mix of auto-indent problem and duplcation problem.

18

I can confirm the same behaviour as I also use windows.

And this is without the patch applied.

19

I can observe this bug on Linux too, with different terminals. I believe this has been already mentioned in github, but I don’t know if there is an issue for that, so feel free to open one if you can’t find it.

20

Added comments to existing issues:

pasting multilines:
https://github.com/JuliaLang/julia/issues/24049

cursor vanishing:
https://github.com/JuliaLang/julia/issues/26422