# \[pre-ANN\] DataViewer.jl: explore data files with the power of Makie
**URL:** https://discourse.julialang.org/t/pre-ann-dataviewer-jl-explore-data-files-with-the-power-of-makie/105390
**Category:** Package Announcements
**Created:** [October 25, 2023, 6:11pm UTC](https://discourse.julialang.org/t/pre-ann-dataviewer-jl-explore-data-files-with-the-power-of-makie/105390 "2023-10-25T18:11:17Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@ffevotte](https://discourse.julialang.org/u/ffevotte)
#### Post date: [October 25, 2023, 6:11pm UTC](https://discourse.julialang.org/t/pre-ann-dataviewer-jl-explore-data-files-with-the-power-of-makie/105390/1 "2023-10-25T18:11:18Z")
Hello,
I’ve recently been working on [DataViewer.jl](https://github.com/triscale-innov/DataViewer.jl), a GUI that helps explore and visualize the structure of data contained in datafiles (such as HDF5, JLD2 or JSON files).
[https://raw.githubusercontent.com/triscale-innov/DataViewer.jl/gh-pages/dev/screencast.mp4](https://raw.githubusercontent.com/triscale-innov/DataViewer.jl/gh-pages/dev/screencast.mp4)
This tool focuses on understanding how data files are structured, and therefore provides by default a tree-like view of the data. When a “leaf data” is of a supported type\[1\], basic Makie-based visualizations are provided, to further understand what the data file contains.
`DataViewer.jl` can be used as a normal Julia package, whose main API is the [`DataViewer.view`](https://triscale-innov.github.io/DataViewer.jl/dev/ref/#DataViewer.view) function:
```julia-repl
julia> using DataViewer
julia> using JLD2
julia> DataViewer.view("my_file.jld2")
```
It can also be “installed” in the system, _i.e._ the [`DataViewer.install`](https://triscale-innov.github.io/DataViewer.jl/dev/ref/#DataViewer.install) function creates a launcher script\[2\] that can be called from the command line. (This is what is demoed in the screencast above)
I arguably lacked inspiration for the package name: although it is free in the General registry, a cursory look on github revealed at least 3 other Julia projects by that name. So please feel free to bikeshed on the name (or anything else!)
Implementationwise, this is based on [`JSServe.jl`](https://github.com/SimonDanisch/JSServe.jl) and [`WGLMakie`](https://github.com/MakieOrg/Makie.jl), everything being rendered in an [`Electron.jl`](https://github.com/davidanthoff/Electron.jl) window. All my gratitude goes to @sdanisch and @jules who helped me a lot!
* * *
1. Currently: a 1D 2D or 3D array of numbers, or a dictionary of numbers
2. By default, this also compiles a system image so that everything feels more snappy
---
### Author:  [@Jake](https://discourse.julialang.org/u/Jake)
#### Post date: [October 25, 2023, 6:40pm UTC](https://discourse.julialang.org/t/pre-ann-dataviewer-jl-explore-data-files-with-the-power-of-makie/105390/2 "2023-10-25T18:40:06Z")
That demo looks really nice. Without having yet tried it, does it also do:
1. Mark the curser position?
2. Better yet, mark the closest point to the curser position?
3. Allow me to transform the abscissa? i.e. k\*(n-1) where k is a constant and n is the sample number.
4. Read arrow files? And maybe even Matlab files? (The new Matlab format is a version of HDF5)
5. Save (or copy) the output graph?
Having something that competes with Matlab’s interactive graphing/data exploration facilities is really exciting.
---
### Author:  [@rafael.guerra](https://discourse.julialang.org/u/rafael.guerra)
#### Post date: [October 25, 2023, 8:23pm UTC](https://discourse.julialang.org/t/pre-ann-dataviewer-jl-explore-data-files-with-the-power-of-makie/105390/3 "2023-10-25T20:23:40Z")
Thanks and congratulations on a great job.
One question and one suggestion:
- How to increase the size of the graphic area (it doesn’t increase when we enlarge the Electron window)?
- Perhaps: `DatafileViewer.jl` ?
---
### Author:  [@jar1](https://discourse.julialang.org/u/jar1)
#### Post date: [October 25, 2023, 8:35pm UTC](https://discourse.julialang.org/t/pre-ann-dataviewer-jl-explore-data-files-with-the-power-of-makie/105390/4 "2023-10-25T20:35:53Z")
Would it be possible to put the gui in a vscode tab?
---
### Author:  [@fdekerme](https://discourse.julialang.org/u/fdekerme)
#### Post date: [October 25, 2023, 9:25pm UTC](https://discourse.julialang.org/t/pre-ann-dataviewer-jl-explore-data-files-with-the-power-of-makie/105390/5 "2023-10-25T21:25:06Z")
Nice work !
3d slice visualization could be very useful for medical images in DICOM format.
---
### Author:  [@gvdr](https://discourse.julialang.org/u/gvdr)
#### Post date: [October 26, 2023, 5:03am UTC](https://discourse.julialang.org/t/pre-ann-dataviewer-jl-explore-data-files-with-the-power-of-makie/105390/6 "2023-10-26T05:03:46Z")
Yeah, the 3D slice would be a very cool addition also to GeoStats viz functions. Is that implemented somewhere like in Makie? Gonna search for it now.
---
### Author:  [@ffevotte](https://discourse.julialang.org/u/ffevotte)
#### Post date: [October 26, 2023, 6:26am UTC](https://discourse.julialang.org/t/pre-ann-dataviewer-jl-explore-data-files-with-the-power-of-makie/105390/7 "2023-10-26T06:26:41Z")
> [@Jake](#):
>
> - Mark the curser position?
> - Better yet, mark the closest point to the curser position?
These aren’t implemented yet, but should be doable and in the spirit of the package
> [@Jake](#):
>
> - Allow me to transform the abscissa? i.e. k\*(n-1) where k is a constant and n is the sample number.
> - Save (or copy) the output graph?
Currently, `DataViewer` is more meant as a tool to quickly understand/check what’s in a data file. I.e answer questions like “did I correctly put all results in my output file?” or “does this user-provided input file contain the data I need, with the right structure?”.
It’s not really meant as a plotting tool, and I feel like adding UI to transform axes would go too far in that direction (and where do we stop? do we also want some UI to label axes? or add a title?)
Saving the output graph could be a nice, generic feature, though.
> [@Jake](#):
>
> Read arrow files? And maybe even Matlab files? (The new Matlab format is a version of HDF5)
AFAIU, Arrow files store flat tabular/columnar data à la DataFrames? For now, `DataViewer` has rather been designed to work with tree-like data structures (like Dicts of Dicts of Arrays of Dicts), but I guess we could also support DataFrame-like data structures by looking at them either as a Dictionary of columns, or an Array of rows. Once this is done, reading arrow files would be very natural.
---
### Author:  [@ffevotte](https://discourse.julialang.org/u/ffevotte)
#### Post date: [October 26, 2023, 6:32am UTC](https://discourse.julialang.org/t/pre-ann-dataviewer-jl-explore-data-files-with-the-power-of-makie/105390/8 "2023-10-26T06:32:48Z")
> [@rafael.guerra](#):
>
> Thanks and congratulations on a great job.
Thanks!
> [@rafael.guerra](#):
>
> How to increase the size of the graphic area (it doesn’t increase when we enlarge the Electron window)?
Good question, I’d like to know that as well! AFAIU, this is a limitation of WGLMakie (or maybe it’s simply that I didn’t manage to resize the WGLMakie figure?)
>
>
> I was wondering if it would be possible to auto-resize a WGLMakie figure? For e…xample, when the figure is displayed in a VSCode Plot Pane window, would it be possible to:
>
> 1. Fill the plot pane window based on the window initial size
> 2. Auto-resize the figure based on a change to the plot pane size?
>
> Basically, I'm looking for similar functionality as the PlotlyJS.jl package, which fills and resizes based on the Blink window or the VSCode Plot Pane size.
> [@rafael.guerra](#):
>
> Perhaps: `DatafileViewer.jl` ?
Yes, good idea! I like that it conveys the idea of working with datafiles rather than raw data. Maybe the command-line tool could be called `dfv` then?
---
### Author:  [@ffevotte](https://discourse.julialang.org/u/ffevotte)
#### Post date: [October 26, 2023, 6:48am UTC](https://discourse.julialang.org/t/pre-ann-dataviewer-jl-explore-data-files-with-the-power-of-makie/105390/9 "2023-10-26T06:48:56Z")
> [@jar1](#):
>
> Would it be possible to put the gui in a vscode tab?
The short answer is that it’s a perfectly sensible thing to do, but I didn’t manage to do it.
I originally thought it would be as simple as asking the JSServe app to display in VSCode, but this only partly works: all features implemented with Makie Observables work fine in this context, but navigation in the data fields doesn’t work. It might be because this is coded like a plain old web app: in the Electron window clicking on a link asks the JSServe server for the document associated to a new URL; in a VSCode pane, clicking on a link doesn’t seem to do anything…
If someone knowledgeable wants to chime in about this, I’d really appreciate it!
> [@fdekerme](#):
>
> 3d slice visualization could be very useful for medical images in DICOM format.
I’m not at all familiar with the DICOM format, but a cursory look at [DICOM.jl](https://github.com/JuliaHealth/DICOM.jl) seems to indicate that all keywords are there: tree-like data structure, Dict-like access…
It might be really easy to at least try adding support to DICOM in DataViewer.
> [@gvdr](#):
>
> Yeah, the 3D slice would be a very cool addition also to GeoStats viz functions. Is that implemented somewhere like in Makie? Gonna search for it now.
Not sure I understand: are you asking whether 3D slices are implemented in Makie? If so, there’s for example this snippet in the WGLMakie documentation, which shows how to do it:
[https://docs.makie.org/dev/explanations/backends/wglmakie/#record\_a\_statemap](https://docs.makie.org/dev/explanations/backends/wglmakie/#record_a_statemap)
---
### Author:  [@fdekerme](https://discourse.julialang.org/u/fdekerme)
#### Post date: [October 26, 2023, 7:56pm UTC](https://discourse.julialang.org/t/pre-ann-dataviewer-jl-explore-data-files-with-the-power-of-makie/105390/10 "2023-10-26T19:56:23Z")
A DICOM file consists of two parts: a header and data. The header contains information on the patient, the equipment used, image parameters and other metadata. To visualize metadata, you can see (self-promotion here 😁 ) my package [DICOMTree](https://github.com/JuliaHealth/DICOMTree.jl).
Data can take many forms, but the most common (a DICOM file from a CT scanner) is a 2D matrix of grayscale pixels, corresponding to a slice of the 3D image. To make up the complete image, there are several separate DICOM files, each containing a slice. If the image has 256 slices, there will be 256 DICOM files. Notice that this may vary for some DICOM files, in radiotherapy for instance. For example, the RT Dose (patient irradiation dose map) contains the 3D matrix directly in a single file. The RT Structure, on the other hand, contains points (coordinates in physical space) that describe the contours of certain organs/regions of interest. But the most common is the CT scanner.
Hope it was helpful 😁
---
### Author:  [@LaurentPlagne](https://discourse.julialang.org/u/LaurentPlagne)
#### Post date: [October 27, 2023, 9:00am UTC](https://discourse.julialang.org/t/pre-ann-dataviewer-jl-explore-data-files-with-the-power-of-makie/105390/11 "2023-10-27T09:00:16Z")
> [@ffevotte](#):
>
> The short answer is that it’s a perfectly sensible thing to do, but I didn’t manage to do it.
>
> I originally thought it would be as simple as asking the JSServe app to display in VSCode, but this only partly works: all features implemented with Makie Observables work fine in this context, but navigation in the data fields doesn’t work. It might be because this is coded like a plain old web app: in the Electron window clicking on a link asks the JSServe server for the document associated to a new URL; in a VSCode pane, clicking on a link doesn’t seem to do anything…
>
> If someone knowledgeable wants to chime in about this, I’d really appreciate it!
@davidanthoff and @pfitzseb may have an opinion on this.
---
### Author:  [@pfitzseb](https://discourse.julialang.org/u/pfitzseb)
#### Post date: [October 27, 2023, 9:31am UTC](https://discourse.julialang.org/t/pre-ann-dataviewer-jl-explore-data-files-with-the-power-of-makie/105390/12 "2023-10-27T09:31:44Z")
That’s expected behaviour and unlikely to change.
---
### Author:  [@sdanisch](https://discourse.julialang.org/u/sdanisch)
#### Post date: [October 27, 2023, 2:15pm UTC](https://discourse.julialang.org/t/pre-ann-dataviewer-jl-explore-data-files-with-the-power-of-makie/105390/13 "2023-10-27T14:15:44Z")
Really cool package! 🙂
> [@LaurentPlagne](#):
>
> @davidanthoff and @pfitzseb may have an opinion on this.
I think it should be possible to make links, that jus updates the APP…
But I will need to think about how to actually structure that.
You might be happy to see [SpecApi by SimonDanisch · Pull Request #3281 · MakieOrg/Makie.jl · GitHub](https://github.com/MakieOrg/Makie.jl/pull/3281), which fixed quite a few memory leaks in WGLMakie, together with [JSServe#sd/fixes](https://github.com/SimonDanisch/JSServe.jl/pull/189)!
---
### Author:  [@jar1](https://discourse.julialang.org/u/jar1)
#### Post date: [October 27, 2023, 5:30pm UTC](https://discourse.julialang.org/t/pre-ann-dataviewer-jl-explore-data-files-with-the-power-of-makie/105390/14 "2023-10-27T17:30:12Z")
Would you mind elaborating on this?
Seeing the video above I imagined VSCode Julia could someday support a user-customizable interactive gui system which would be very powerful, something like [https://gtoolkit.com/](https://gtoolkit.com/). What stands in the way of this?
---
### Author:  [@ffevotte](https://discourse.julialang.org/u/ffevotte)
#### Post date: [October 27, 2023, 8:47pm UTC](https://discourse.julialang.org/t/pre-ann-dataviewer-jl-explore-data-files-with-the-power-of-makie/105390/15 "2023-10-27T20:47:55Z")
> [@sdanisch](#):
>
> I think it should be possible to make links, that jus updates the APP…
> But I will need to think about how to actually structure that.
Glad to hear that! Please let me know if I can help.