[ANN] Diary.jl 📔

Dear Diary.jl :notebook_with_decorative_cover:,

Tired of losing work? Forgot what you did a week ago? Is your hairREPL history a mess?

Sounds like you could use a diary.

With Diary.jl, your REPL history is automatically tracked, parsed and written separately to a diary.jl file in your current active project. It’s that simple! And it does so in the background without stealing the REPL and thus affecting responsiveness.

If you do not like Diary.jl to automatically write your history to a diary file, you can disable the autocommit option. With autocommit disabled, you must manually commit your history, by communicating with Diary.jl through comments. Simply type: # diary: commit [n] where [n] is the n last number of things you typed into the REPL that you want to commit to your diary file.

Diary.jl also comes with numerous ways to configure the location of your diary files. All configuration options can be set globally by creating a ~/.julia/config/Diary.toml file, or on a per-project basis, by placing a Diary.toml file in your project root. More information on the configuration options of Diary.jl can be found in the documentation.

Always keep a diary under your pillow. Get yours at Diary.jl :notebook_with_decorative_cover:.

]add Diary

43 Likes

i want to use it but i have this fear that i will at some point type the password into the repl for some quick and dirty testing and forget to delete then a hacker get my password…

3 Likes

Interesting idea. Does it completely prevent the standard REPL history when in use, or does it just additionally save what I type in a new spot? I.e. is it redirecting or a second copy

4 Likes

Thanks a lot for both of those comments.

The standard Julia REPL saves the history in a file called ~/.julia/logs/repl_history.jl. If you accidentally type a password into the REPL and forget about it, it will live in there forever, and a potential hacker can retrieve it. So Diary.jl does not really raise the security concern there. In fact, it might do the opposite, since you will be more readily made aware of this fact by having it written out in a diary file.

Diary.jl still writes everything you type into the REPL into the repl_history.jl file (or whatever JULIA_HISTORY was set to, when Diary was loaded). To work with multiple sessions, however, Diary.jl must temporarily redirect the history to a new file first, before copying it over. What Diary.jl does is then to copy your REPL history to a temporary file, set JULIA_HISTORY to that file instead, and then watch it for changes. Any change is verbatim copied back to the original history file, to remain consistent with the formatting used there, but is afterwards parsed by Diary.jl to remove metadata, strip off trailing semi-colons and put it into the relevant diary file. This is also the reason why Diary doesn’t have to steal the REPL. Everything happens in the background by monitoring the REPL history file, not by directly reading what the user types.

As such, using Diary.jl isn’t incompatible with normal usage. In fact, you could probably use it for years and completely forget about it, if you configure Diary.jl to use a filename that is hidden by the filesystem.

3 Likes

What a great idea! Thanks for this!

1 Like

I wasn’t sure if your comment was a feature request or concern, but I think it might be useful to be able to configure Diary.jl to actually redirect, instead of keeping a second copy. So I just registered version v0.1.1 with a new option called persistent_history. It defaults to true, but if set to false, the REPL history will not be redirected to the repl_history.jl file.

Together with autocommit = false, this allows you to use Diary.jl to disable history persistence on a per-project basis. I probably wouldn’t use this myself, but I can imagine cases where you would want to do this. For instance, if you are working with sensitive data, such as in @xiaodai’s scenario, and you want to make sure that the history is cleaned up after you close the REPL.

2 Likes

The difference is, I never commit that file to a repository that lives as plain text in a public repo…

Then again, I don’t think I ever enter passwords into the REPL anyway… This is cool!

2 Likes

I would recommend adding diary.jl to .gitignore. :sweat_smile:

Thank you! :heart:

3 Likes

Wow this is a great idea! Is this only for Julia 1.5? Any chance of backward compatibility? (I haven’t made the leap to 1.5 yet)

1 Like

It was, but v0.1.3 (to be registered shortly) is now also compatible with v1.4! :tada: Just had to revert back to old-style keyword syntax and replace contains by occursin. Unfortunately, it can’t easily be made compatible with v1.3 and before due to relying on Pkg.project(). I’m not sure if there is a pre-1.4 way of getting the directory of the currently active project. Base.current_project() only reflects what the project was at startup.

3 Likes

Thanks a lot. I’ve got it working on 1.4 on windows. I activated an environment in a new folder adding Diary.jl which generates a Manifest.toml and Project.toml. I did some stuff in the REPL and exited, but I can’t find Diary.jl in the root folder. I went through the docs, and it seems like it should be working—but I can’t figure out what I am missing. Any ideas that I can try?

Travis CI runs and passes on Windows, but seems like that doesn’t mean that it works on all Windows machines! :grimacing:

Try starting julia with the environment variable JULIA_DEBUG=Diary, that should print debug messages whenever Diary.jl detects a change. If it doesn’t, then I also added the option in v0.1.3 to switch to file_polling. It is a less reliable method, but may work when FileWatching.watch_file fails. See also: https://github.com/dalum/Diary.jl/issues/1

The debug variable doesn’t seem to show anything. I do see the following though:

julia> Diary.read_configuration()
ERROR: KeyError: key "HOME" not found
Stacktrace:
 [1] (::Base.var"#459#460")(::String) at .\env.jl:79
 [2] access_env(::Base.var"#459#460", ::String) at .\env.jl:14
 [3] getindex at .\env.jl:79 [inlined]
 [4] default_configuration() at C:\Users\asimd\.julia\packages\Diary\88OsY\src\Diary.jl:290
 [5] read_configuration(::String) at C:\Users\asimd\.julia\packages\Diary\88OsY\src\Diary.jl:251 (repeats 2 times)
 [6] top-level scope at REPL[24]: 

And the same when I do ]test Diary

  Test threw exception
  Expression: Diary.parse_history(history_lines) == []
  KeyError: key "HOME" not found

You should replace joinpath(ENV["HOME"], ".julia", ...) with joinpath(DEPOT_PATH[1], ...).

2 Likes

I was naïvely, I see now, assuming the environment variable HOME to be set on all systems.

Thanks!

1 Like

Trying Diary.jl for the first time:

  • Win10
  • Julia 1.5.3
  • Diary.jl version 0.1.4

I was able to ] add Diary from the REPL pkg prompt. Then I did using Diary from the julia prompt. However, nothing seems to happen. No diary.jl file is created. Nor is a default Diary.toml produced.

Does Diary.jl require the Diary.toml be configured before it works?

3 Likes

Thanks!

I don’t use Windows, so I can’t reproduce. Diary.toml is not created by default, you need to create it yourself, if you want to configure options. My first suggestion would be to try and switch to file-polling by putting file_polling = true in the Diary.toml file.

I observed the same on arch Linux with Julia 1.5.3.
No diary.jl is created with default options.

This is surprising to me since I use Manjaro, so I would expect it to behave at least similarly on Arch. Could you turn debugging on (setting the environment variable JULIA_DEBUG=Diary) and report any output (or lack thereof) that it prints, if you do something in the REPL?

Also, do you use OhMyREPL or perhaps another package that may hijack the REPL/history file? In that case, you need to load Diary.jl after OhMyREPL, etc.

You mean like this? OhMyREPL and Revise are loaded during startup before Diary.

julia> using Diary

julia> ENV["JULIA_DEBUG"]=Diary
Diary

(@v1.5) pkg> activate tmp/lel/
 Activating environment at `~/tmp/lel/Project.toml`

julia> x=1+1
2

julia> 1+1
2

julia> x=2
2

julia> x+x
4

Only these files are there:

╰─➤  ls ~/tmp/lel                                                                                                                                                                                                                       [15-01-21 | 2:35:34]
Manifest.toml  Project.toml