DearDiary.jl: state of the project at v0.8.0

Package Announcements
, , ,
1

Hi all,

A few months on from the launch thread, DearDiary.jl just hit v0.8.0, and it feels like the right point to share where the project stands rather than just what shipped today.

The shape of the project

DearDiary.jl is a ML experiment tracker. Same job as MLflow (projects, experiments, iterations, parameters, metrics, artifacts, models), but written end-to-end in Julia. The architecture splits into Repository → Service → Routes → Client → UI layers, each with its own tests and a swappable boundary (Multitier architecture).

The big pieces

  • Bit-exact reproducibility. Every iteration can snapshot its full Manifest.toml, Julia version, and git SHA. restore(iteration_id) writes the captured environment to disk for Pkg.instantiate. with_iteration auto-snapshots driver iterations. You can re-run anything you logged, byte-for-byte
  • Model registry. First-class Model and ModelVersion entities with a NO_STAGE -> STAGING -> PRODUCTION -> ARCHIVED lifecycle. Promoting a version to PRODUCTION auto-archives the previous incumbent. Every version links back to the iteration that produced it, so a production model carries its full training lineage
  • Pluggable artifact storage. Resource bytes live in any AbstractArtifactStore backend: inline in SQLite (default), on the local filesystem, or in any S3-compatible object store (AWS S3, MinIO, Cloudflare R2). migrate_artifacts! moves rows between backends on a running database
  • Iteration lineage and lifecycle. Iterations carry a parent_iteration_id for HPO sweeps, nested CV folds, and distributed workers; an IterationStatus enum (RUNNING / SUCCEEDED / FAILED / KILLED); and an error_message field. with_iteration auto-finalises with the right status and captures exception text on failure
  • A full Julia client. connect(...), with_iteration(...) do ... end, full CRUD parity over the REST API

New in v0.8.0

An experimental embedded web dashboard. DearDiary.run() boots a small Bonito-based UI on a sibling port alongside the REST API. It walks the project tree, nests child iterations under their drivers, and renders per-iteration metric charts in pure Julia as inline SVG (no third-party JavaScript bundle, no CDN).

What’s next

MLJ Loggers integration is close. Once it lands, an MLJ pipeline will log iterations, parameters, and metrics into a running DearDiary instance without manual create_iteration / log_metrics calls. The UI roadmap stays open; experimental status means that it can keep cooking into something more polished (suggestions are totally welcome).

Try it

] add DearDiary

using DearDiary
DearDiary.run() # REST API on :9000, dashboard on :9001

Repo: GitHub - JuliaAI/DearDiary.jl: An ML experiment tracker made in Julia. · GitHub
Docs: Home · DearDiary.jl

image
image1276×1277 93.2 KB

2 Likes