[ANN] Diary.jl 📔

Evey · August 17, 2020, 3:45pm

Tired of losing work? Forgot what you did a week ago? Is your ~~hair~~REPL history a mess?

Sounds like you could use a diary.

With Diary.jl, your REPL history is automatically tracked, parsed and written separately to a diary.jl file in your current active project. It’s that simple! And it does so in the background without stealing the REPL and thus affecting responsiveness.

If you do not like Diary.jl to automatically write your history to a diary file, you can disable the autocommit option. With autocommit disabled, you must manually commit your history, by communicating with Diary.jl through comments. Simply type: # diary: commit [n] where [n] is the n last number of things you typed into the REPL that you want to commit to your diary file.

Diary.jl also comes with numerous ways to configure the location of your diary files. All configuration options can be set globally by creating a ~/.julia/config/Diary.toml file, or on a per-project basis, by placing a Diary.toml file in your project root. More information on the configuration options of Diary.jl can be found in the documentation.

Always keep a diary under your pillow. Get yours at Diary.jl .

]add Diary

xiaodai · August 17, 2020, 4:08pm

i want to use it but i have this fear that i will at some point type the password into the repl for some quick and dirty testing and forget to delete then a hacker get my password…

tbeason · August 17, 2020, 4:29pm

Interesting idea. Does it completely prevent the standard REPL history when in use, or does it just additionally save what I type in a new spot? I.e. is it redirecting or a second copy

Evey · August 17, 2020, 6:50pm

Thanks a lot for both of those comments.

The standard Julia REPL saves the history in a file called ~/.julia/logs/repl_history.jl. If you accidentally type a password into the REPL and forget about it, it will live in there forever, and a potential hacker can retrieve it. So Diary.jl does not really raise the security concern there. In fact, it might do the opposite, since you will be more readily made aware of this fact by having it written out in a diary file.

Diary.jl still writes everything you type into the REPL into the repl_history.jl file (or whatever JULIA_HISTORY was set to, when Diary was loaded). To work with multiple sessions, however, Diary.jl must temporarily redirect the history to a new file first, before copying it over. What Diary.jl does is then to copy your REPL history to a temporary file, set JULIA_HISTORY to that file instead, and then watch it for changes. Any change is verbatim copied back to the original history file, to remain consistent with the formatting used there, but is afterwards parsed by Diary.jl to remove metadata, strip off trailing semi-colons and put it into the relevant diary file. This is also the reason why Diary doesn’t have to steal the REPL. Everything happens in the background by monitoring the REPL history file, not by directly reading what the user types.

As such, using Diary.jl isn’t incompatible with normal usage. In fact, you could probably use it for years and completely forget about it, if you configure Diary.jl to use a filename that is hidden by the filesystem.

crbinz · August 17, 2020, 7:27pm

What a great idea! Thanks for this!

Evey · August 17, 2020, 8:16pm

I wasn’t sure if your comment was a feature request or concern, but I think it might be useful to be able to configure Diary.jl to actually redirect, instead of keeping a second copy. So I just registered version v0.1.1 with a new option called persistent_history. It defaults to true, but if set to false, the REPL history will not be redirected to the repl_history.jl file.

Together with autocommit = false, this allows you to use Diary.jl to disable history persistence on a per-project basis. I probably wouldn’t use this myself, but I can imagine cases where you would want to do this. For instance, if you are working with sensitive data, such as in @xiaodai’s scenario, and you want to make sure that the history is cleaned up after you close the REPL.

kevbonham · August 18, 2020, 1:55pm

The difference is, I never commit that file to a repository that lives as plain text in a public repo…

Then again, I don’t think I ever enter passwords into the REPL anyway… This is cool!

Evey · August 18, 2020, 2:36pm

I would recommend adding diary.jl to .gitignore.

Thank you!

ElectronicTeaCup · August 18, 2020, 7:08pm

Wow this is a great idea! Is this only for Julia 1.5? Any chance of backward compatibility? (I haven’t made the leap to 1.5 yet)

Evey · August 18, 2020, 8:21pm

It was, but v0.1.3 (to be registered shortly) is now also compatible with v1.4! Just had to revert back to old-style keyword syntax and replace contains by occursin. Unfortunately, it can’t easily be made compatible with v1.3 and before due to relying on Pkg.project(). I’m not sure if there is a pre-1.4 way of getting the directory of the currently active project. Base.current_project() only reflects what the project was at startup.

ElectronicTeaCup · August 19, 2020, 1:32pm

Thanks a lot. I’ve got it working on 1.4 on windows. I activated an environment in a new folder adding Diary.jl which generates a Manifest.toml and Project.toml. I did some stuff in the REPL and exited, but I can’t find Diary.jl in the root folder. I went through the docs, and it seems like it should be working—but I can’t figure out what I am missing. Any ideas that I can try?

Evey · August 19, 2020, 1:39pm

Travis CI runs and passes on Windows, but seems like that doesn’t mean that it works on all Windows machines!

Try starting julia with the environment variable JULIA_DEBUG=Diary, that should print debug messages whenever Diary.jl detects a change. If it doesn’t, then I also added the option in v0.1.3 to switch to file_polling. It is a less reliable method, but may work when FileWatching.watch_file fails. See also: https://github.com/dalum/Diary.jl/issues/1

ElectronicTeaCup · August 19, 2020, 2:50pm

The debug variable doesn’t seem to show anything. I do see the following though:

julia> Diary.read_configuration()
ERROR: KeyError: key "HOME" not found
Stacktrace:
 [1] (::Base.var"#459#460")(::String) at .\env.jl:79
 [2] access_env(::Base.var"#459#460", ::String) at .\env.jl:14
 [3] getindex at .\env.jl:79 [inlined]
 [4] default_configuration() at C:\Users\asimd\.julia\packages\Diary\88OsY\src\Diary.jl:290
 [5] read_configuration(::String) at C:\Users\asimd\.julia\packages\Diary\88OsY\src\Diary.jl:251 (repeats 2 times)
 [6] top-level scope at REPL[24]:

And the same when I do ]test Diary

  Test threw exception
  Expression: Diary.parse_history(history_lines) == []
  KeyError: key "HOME" not found

fredrikekre · August 19, 2020, 2:54pm

You should replace joinpath(ENV["HOME"], ".julia", ...) with joinpath(DEPOT_PATH[1], ...).

Evey · August 19, 2020, 2:55pm

I was naïvely, I see now, assuming the environment variable HOME to be set on all systems.

Thanks!

devel-chm · January 5, 2021, 9:31pm

Trying Diary.jl for the first time:

Win10
Julia 1.5.3
Diary.jl version 0.1.4

I was able to ] add Diary from the REPL pkg prompt. Then I did using Diary from the julia prompt. However, nothing seems to happen. No diary.jl file is created. Nor is a default Diary.toml produced.

Does Diary.jl require the Diary.toml be configured before it works?

Evey · January 15, 2021, 9:29am

Thanks!

I don’t use Windows, so I can’t reproduce. Diary.toml is not created by default, you need to create it yourself, if you want to configure options. My first suggestion would be to try and switch to file-polling by putting file_polling = true in the Diary.toml file.

roflmaostc · January 15, 2021, 10:13am

I observed the same on arch Linux with Julia 1.5.3.
No diary.jl is created with default options.

Evey · January 15, 2021, 11:15am

This is surprising to me since I use Manjaro, so I would expect it to behave at least similarly on Arch. Could you turn debugging on (setting the environment variable JULIA_DEBUG=Diary) and report any output (or lack thereof) that it prints, if you do something in the REPL?

Also, do you use OhMyREPL or perhaps another package that may hijack the REPL/history file? In that case, you need to load Diary.jl after OhMyREPL, etc.

roflmaostc · January 15, 2021, 1:38pm

You mean like this? OhMyREPL and Revise are loaded during startup before Diary.

julia> using Diary

julia> ENV["JULIA_DEBUG"]=Diary
Diary

(@v1.5) pkg> activate tmp/lel/
 Activating environment at `~/tmp/lel/Project.toml`

julia> x=1+1
2

julia> 1+1
2

julia> x=2
2

julia> x+x
4

Only these files are there:

╰─➤  ls ~/tmp/lel                                                                                                                                                                                                                       [15-01-21 | 2:35:34]
Manifest.toml  Project.toml