# Rebuilding Data from Heatmaps

**URL:** <https://discourse.julialang.org/t/rebuilding-data-from-heatmaps/122291>\
**Category:** Data\
**Tags:** question\
**Created:** [November 5, 2024, 3:20pm UTC](https://discourse.julialang.org/t/rebuilding-data-from-heatmaps/122291 "2024-11-05T15:20:31Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Deceneu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/deceneu/32/217811_2.png) [@Deceneu](https://discourse.julialang.org/u/Deceneu)\
**Post date:** [November 5, 2024, 3:20pm UTC](https://discourse.julialang.org/t/rebuilding-data-from-heatmaps/122291/1 "2024-11-05T15:20:31Z")

</div>

Hello, first time posting, might be in the wrong category.

I need to extract scalar values from heatmaps saved as images. I don’t have access to the raw data, only compressed JPGs with inlayed colorbars.

I would think that finding an efficient inverse of the colormap would be the ideal way to retrieve the actual values. What would be the (most) efficient way to compute the inverse/ what would be the most computationally efficient inverse?

* * *

UPDATE: Added an example. The images are from a thermal camera.

 ![FLIR1276](https://global.discourse-cdn.com/julialang/original/3X/7/7/773a5883d496647a7d8e5056bb5f4690833269e9.jpeg)

The Minima and Maxima are present in the picture.

Running the following code

```julia
using Images, ImageShow, Plots
frame_flir = Images.load("FLIR/Misc Pics/FLIR1274.jpg")

unzip(a) = map(x->getfield.(a, x), fieldnames(eltype(a)))

plot(unzip(frame_flir[31:209, 309:312]), linecolor = frame_flir[31:209, 310], label = false, xlab = "R", ylab = "G", zlab = "B")

```

 ![plot1](https://global.discourse-cdn.com/julialang/original/3X/4/0/40dfb5ee69dbde0d28946b464dc6acb717a7ddaf.png)

```julia
plot(unzip(frame_flir[31:209, 309:312])[1], linecolor = :red, label = ["red" false false false])
plot!(unzip(frame_flir[31:209, 309:312])[2], linecolor = :green, label = ["green" false false false])
plot!(unzip(frame_flir[31:209, 309:312])[3], linecolor = :blue, label = ["blue" false false false])

```

 ![plot2](https://global.discourse-cdn.com/julialang/original/3X/8/1/8145e2811236e72b27c024b81a3cdd5b59e280ba.png)

I think this is a bit noisier than desired because of JPG compression, along the points where there are the black notches in the colorbar.

---

<div class="post-metadata">

**Author:** ![mthelm85](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mthelm85/32/224164_2.png) [@mthelm85](https://discourse.julialang.org/u/mthelm85)\
**Post date:** [November 5, 2024, 4:10pm UTC](https://discourse.julialang.org/t/rebuilding-data-from-heatmaps/122291/2 "2024-11-05T16:10:01Z")

</div>

Do you have some sample images you could provide us with? I think that would really increase your chances of getting good feedback. Do you know how the images were generated? Was a common colormap used, or was it something custom? Also, do you know how much data loss there may have been in the compression process? Finally, I assume the plotting library would have normalized the values before mapping them to the colors, so I don’t think you’ll be able to get the original values back without access to the raw data, unless you know min/max values maybe?

---

<div class="post-metadata">

**Author:** ![Deceneu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/deceneu/32/217811_2.png) [@Deceneu](https://discourse.julialang.org/u/Deceneu)\
**Post date:** [November 5, 2024, 4:54pm UTC](https://discourse.julialang.org/t/rebuilding-data-from-heatmaps/122291/4 "2024-11-05T16:54:57Z")

</div>

Hey. I have added a sample image, and a bit of code I used to analyze the colorbar present in said image. I think it’s within reason to infer the colormap using some smoothing interpolation, such as [Loess.jl](https://github.com/JuliaStats/Loess.jl). My issue is more-so with the process of obtaining the inverse map, Temperature=f(R,G,B), which, ideally, should be fast.

---

<div class="post-metadata">

**Author:** ![mthelm85](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mthelm85/32/224164_2.png) [@mthelm85](https://discourse.julialang.org/u/mthelm85)\
**Post date:** [November 6, 2024, 3:55pm UTC](https://discourse.julialang.org/t/rebuilding-data-from-heatmaps/122291/5 "2024-11-06T15:55:41Z")

</div>

Hmmm, what about building a k-d tree or ball tree with the colormap values and then finding the nearest neighbor for each pixel in the image? Once you get something (anything) working, it won’t be difficult to get advice from the community on how to speed it up.

---

<div class="post-metadata">

**Author:** ![mthelm85](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mthelm85/32/224164_2.png) [@mthelm85](https://discourse.julialang.org/u/mthelm85)\
**Post date:** [November 6, 2024, 4:54pm UTC](https://discourse.julialang.org/t/rebuilding-data-from-heatmaps/122291/6 "2024-11-06T16:54:09Z")

</div>

Here’s an attempt to just get something working with your image above:

```julia
using ColorSchemes
using Images
using NearestNeighbors

flir = Images.load(raw"C:\Users\mthel\OneDrive\Pictures\FLIR.jpeg")
colorbar = flir[31:209, 309:312]
colors = vec(mean(colorbar, dims=2))
colormap = ColorScheme(colors)
kdtree = KDTree(vcat([[red(c) green(c) blue(c)] for c in colors]...)')

function pixel_to_value(pixel, kdtree)
    query_point = [red(pixel), green(pixel), blue(pixel)]
    idx, _ = knn(kdtree, query_point, 1) 
    normalized_val = (idx[1] - 1) / 178
	return 30.0 + (17.6 - 30.0) * normalized_val
end

img_vals = map(pixel -> pixel_to_value(pixel, kdtree), flir)

# img_vals =

240×320 Matrix{Float64}:
 21.1528 21.1528 21.1528 21.1528 21.1528 … 21.5708 21.7101 21.2921 21.0135
 21.1528 21.1528 21.1528 21.1528 21.1528 21.7101 21.6404 21.2225 21.0135
 21.0831 21.0831 21.1528 21.1528 21.1528 21.5011 21.2921 21.0831 21.0135
 21.0831 21.0831 21.1528 21.1528 21.1528 20.8045 21.2921 21.2225 21.1528
 21.0831 21.0831 21.0831 21.0831 21.0831 21.6404 21.3618 21.3618 21.4315
 21.0831 21.0831 21.0831 21.0831 21.0831 … 21.7101 21.5011 21.4315 21.4315
 21.1528 21.1528 21.0831 21.0831 21.0831 21.4315 21.2225 20.8045 20.5955
  ⋮ ⋱                             
 20.4562 20.0382 19.7596 20.5955 20.1079 22.3371 22.3371 22.3371 22.2674
 20.4562 20.6652 19.7596 20.0382 20.8742 … 22.4764 22.4067 22.3371 22.3371
 20.3169 20.3169 20.7348 20.4562 20.6652 22.4764 22.4764 22.4067 22.4067
 20.3169 20.1775 20.8742 20.3169 20.1775 22.5461 22.4764 22.4067 22.4764
 20.3169 20.3169 20.3169 20.5258 20.5258 22.4764 22.4764 22.4067 22.4067
 20.4562 20.3169 20.1775 20.2472 20.0382 22.4764 22.4067 22.4764 22.4067

# convert back the other direction as a sanity check

function value_to_pixel(value, colormap)
    normalized_val = (value - 30.0) / (17.6 - 30.0)
    return get(colormap, normalized_val)
end

recovered_img = map(val -> value_to_pixel(val, colormap), img_vals)

```

Below is the original image, as well as the recovered image:

 ![FLIR1276](https://global.discourse-cdn.com/julialang/original/3X/7/7/773a5883d496647a7d8e5056bb5f4690833269e9.jpeg)  
 ![image](https://global.discourse-cdn.com/julialang/original/3X/3/1/31de0fbca124ccafb724aeb079918f4283a39054.png)

You can see just from a visual inspection that there are some problems, but it’s a start!

---

<div class="post-metadata">

**Author:** ![empet](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/empet/32/221303_2.png) [@empet](https://discourse.julialang.org/u/empet)\
**Post date:** [November 6, 2024, 10:12pm UTC](https://discourse.julialang.org/t/rebuilding-data-from-heatmaps/122291/7 "2024-11-06T22:12:36Z")

</div>

Given a heatmap as a color image of resolution (rows, cols), and fixing a number of colors, `n_colors`,  
we can perform color image quantization, using K-means, to extract the basic elements that define a heatmap:  
an array, `z_data`, of size (rows, cols), as normalized original data in the heatmap definition, and a colorscheme defined by `n_colors`.

```julia
using Clustering, Images, ColorSchemes
using Plots
plotlyjs()

function image2zvals(img::Union{Matrix{RGB{T}}, Matrix{RGBA{T}}};  
               n_colors=64, maxiter=200, tol=1e-04) where T<:Real
    
    rows, cols = size(img)
    observations = reshape(img, rows*cols)
    kmres = kmeans(observations, n_colors; maxiter = maxiter, tol=tol)
    nclust = nclusters(kmres)
    a = assignments(kmres)/nclust
    z_data = reshape(a, rows, cols)
   
    hcolors = kmres.centers # kmres.centers is the codebook
    if fieldcount(eltype(img)) == 3
        cscheme = ColorScheme([RGB(c...) for c in eachcol(hcolors)])
    else
         cscheme = ColorScheme([RGB(c[1:3]...) for c in eachcol(hcolors)]) 
    end    
    return z_data, cscheme
end 

img = load("heatmap-forum.jpeg");
zdata, cscheme = image2zvals(img; n_colors=64, maxiter=200, tol=1e-04);
#Reconstructed heatmap:
plt = heatmap(zdata[end:-1:1,:], c=cgrad(cscheme), cbar=false, size=(375, 300))

```

To avoid posting here one more image, see this notebook [https://nbviewer.org/gist/empet/6f759474f944b6f865897bebc8b5da76](https://nbviewer.org/gist/empet/6f759474f944b6f865897bebc8b5da76) displaying the heatmap recovered from the computed normalized data and the colorscheme, as well as the values of `z_data`.

The current version of Clustering.js is 0.15.7, and I worked with an older one: 0.14.4.

---

<div class="post-metadata">

**Author:** ![Deceneu](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/deceneu/32/217811_2.png) [@Deceneu](https://discourse.julialang.org/u/Deceneu)\
**Post date:** [December 6, 2024, 10:19am UTC](https://discourse.julialang.org/t/rebuilding-data-from-heatmaps/122291/8 "2024-12-06T10:19:52Z")

</div>

Sorry for the long gap between updates.

I’ve tried a number of different approaches and benchmarked the results. The particular problem I tried to optimize is reducing the compute time for a pixel-to-value computation of an array, regardless of the setup time.

1 - Brute force  
Using Loess.jl, fit the individual colors, and then sample them at an acceptable density, then compute the 3D distance between pixel color and the sampled colors. The minimum distance index gives the corresponding temperatrue.

> **Code Snippet**
>
> ```julia
> using Images, Loess, Statistics, ThreadsX
> 
> fn = "flir.jpg"
> 
> const lw, hi, δ = 18.0, 280., 0.1
> const temp_rng = range(lw, hi, step = δ)
> 
> frame_flir = Images.load(fn)
> idxs = (rows = 31:209, cols = 308:313)
> cb = frame_flir[idxs...]
> 
> const t_samp_rng = range(lw, hi, length(idxs.rows)) |> collect
> 
> unzip(a) = map(x->getfield.(a, x), fieldnames(eltype(a)))
> R, G, B = unzip(cb)
> 
> function (lm::Loess.LoessModel)(x)
> return predict(lm, x)
> end
> 
> span = 0.1 # 0.125
> 
> model_R = loess(t_samp_rng, mean(R, dims = 2) |> vec; span)
> model_G = loess(t_samp_rng, mean(G, dims = 2) |> vec; span)
> model_B = loess(t_samp_rng, mean(B, dims = 2) |> vec; span)
> 
> ### method 1, brute force
> 
> const samples = (R = model_R(temp_rng), 
> G = model_G(temp_rng), 
> B = model_B(temp_rng))
> 
> function find_temp(A, samples = samples)
> matB = map(x->(r = Float64(x.r), g = Float64(x.g), b = Float64(x.b)), A) :: Matrix{@NamedTuple{r::Float64, g::Float64, b::Float64}}
> t = ThreadsX.map(pix -> temp_rng[findmin(map((r, g, b) -> (pix.r - r)^2 + (pix.g - g)^2 + (pix.b - b)^2, samples...))[2]], matB) :: Matrix{Float64}
> return t 
> end
> 
> ```

Using BenchmarkTools.jl, average execution time is ~500ms.

```julia
julia> @benchmark find_temp(frame_flir) seconds = 10
BenchmarkTools.Trial: 21 samples with 1 evaluation.
 Range (min … max): 370.657 ms … 1.223 s ┊ GC (min … max): 53.07% … 69.91%
 Time (median): 408.187 ms ┊ GC (median): 54.02%   
 Time (mean ± σ): 507.752 ms ± 246.115 ms ┊ GC (mean ± σ): 58.57% ± 5.36%

  █▂▂ ▂
  ███▅█▁█▁▁▅▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▅▅▁▁▁▁▁▁▁▁▁▁▁▁▁▅ ▁        
  371 ms Histogram: frequency by time 1.22 s <        

 Memory estimate: 1.51 GiB, allocs estimate: 231131.

```

2 - Symbolic Regression  
If an analytic expression linking the RGB encoding to the scalar value exists, then using it would be the fastest way to convert the entire picture. Using SymbolicRegression.jl, I tried to infer an analytic form starting from both the RGB and HSV colorspaces.

> **Code Snippet**
>
> ```julia
> using Images
> import SymbolicRegression: SRRegressor
> import MLJ: machine, fit!, predict, report
> 
> fn = "flir.jpg"
> 
> const lw, hi, δ = 18.0, 280., 0.1
> const temp_rng = range(lw, hi, step = δ)
> 
> frame_flir = Images.load(fn)
> idxs = (rows = 31:209, cols = 308:313)
> cb = frame_flir[idxs...]
> 
> const t_samp_rng = range(lw, hi, length(idxs.rows)) |> collect
> 
> unzip(a) = map(x->getfield.(a, x), fieldnames(eltype(a)))
> R, G, B = unzip(cb)
> 
> ### method 2, symbolic regression
> 
> # RGB - colorspace
> inRGB = (r = vec(R), g = vec(G), b = vec(B))
> 
> # T - temperature
> inT = stack(t_samp_rng for i in 1:6) |> vec
> 
> model = SRRegressor(
> niterations = 1000,
> binary_operators = [+, -, *, /],
> unary_operators=[exp, log, cos, sin, sqrt],
> )
> 
> mach = machine(model, inRGB, inT)
> 
> fit!(mach)
> rep = report(mach)
> 
> println(rep.equations[rep.best_idx])
> 
> ```

`julia> ((log(r + 0.06514106105173974) + sin(sin(b * 2.00389950943762))) * -114.86518971882892) + 117.47144076159321`

```julia
julia> @benchmark bestf.(frame_flir)
BenchmarkTools.Trial: 2622 samples with 1 evaluation.
 Range (min … max): 1.724 ms … 13.856 ms ┊ GC (min … max): 0.00% … 86.34%
 Time (median): 1.830 ms ┊ GC (median): 0.00%
 Time (mean ± σ): 1.900 ms ± 481.131 μs ┊ GC (mean ± σ): 4.03% ± 9.53%

  ▆▃▂█▆▂
  ██████▇▅▁▅▅▃▁▁▁▁▁▃▃▄▁▁▁▁▁▁▁▃▁▁▁▁▁▃▁▁▁▁▃▁▁▁▁▁▁▁▁▁▃▁▁▁▁▅▆▇██▆ █
  1.72 ms Histogram: log(frequency) by time 3.48 ms <

 Memory estimate: 600.09 KiB, allocs estimate: 4.

```

which is a ~500x improvement, however there’s a significant error; maybe it’s possible to refine this approach, but I am no familiar enough with the underlying package.

 ![RGB - symbolic regression](https://global.discourse-cdn.com/julialang/original/3X/b/6/b69e07ce58a06275285324fe1769903445538487.png)

and HSV

> **Code Snippet**
>
> ```julia
> # HSV - colorspace
> H, S, V = convert.(HSV, cb) |> unzip
> inHSV = (h = vec(H), s = vec(S), v = vec(V))
> 
> # T - temperature
> inT = stack(t_samp_rng for i in 1:6) |> vec
> 
> model = SRRegressor(
> niterations = 1000,
> binary_operators = [+, -, *, /],
> unary_operators=[exp, log, cos, sin, sqrt],
> )
> 
> mach = machine(model, inHSV, inT)
> 
> fit!(mach)
> rep = report(mach)
> 
> println(rep.equations[rep.best_idx])
> 
> ```

with similar results, both for time to execute and error.

3 - @mthelm85’s approach using k-d tree, with some unsophisticated parallelization via ThreadsX.jl

```julia
julia> @benchmark ThreadsX.map(pixel -> pixel_to_value(pixel, kdtree), flir)
BenchmarkTools.Trial: 975 samples with 1 evaluation.                       
 Range (min … max): 3.462 ms … 26.377 ms ┊ GC (min … max): 0.00% … 6.33%
 Time (median): 5.203 ms ┊ GC (median): 29.23%        
 Time (mean ± σ): 5.106 ms ± 1.761 ms ┊ GC (mean ± σ): 19.10% ± 15.10%

  █ ▁▃
  █▃▃▃▄▃▄▃▂▂▂██▆▄▅▅▄▄▃▃▃▂▂▃▂▂▂▂▂▂▂▂▁▁▁▂▂▂▁▂▁▂▁▂▁▁▁▁▂▁▁▁▂▁▁▁▂ ▃
  3.46 ms Histogram: frequency by time 11.8 ms <

 Memory estimate: 22.39 MiB, allocs estimate: 615110.

```

evaluates in ~5ms, which is close to the analytical approach, but the error is much improved.

 ![KDTree](https://global.discourse-cdn.com/julialang/original/3X/a/7/a7c8af1d7c0f1d34aefa6944f64228f25ab45697.png).  
I will also look into the performance when feeding the complete list of points to `knn(...)`.

4 - @empet’s approach using Clustering.jl seems to be a good approach for when the colorbar is missing. The downside is that the colorscheme, and implicitly the underlying scalar is unsorted.

```julia
julia> zdata, cscheme = image2zvals(img; n_colors=128, maxiter=200, tol=1e-04);
julia> cscheme

```

 ![cscheme](https://global.discourse-cdn.com/julialang/original/3X/4/6/46885eeddb5ce96335035a2f35465aeb902c85d3.png)  
And the execution time is the longest so far at ~ 2s.

```julia
julia> @benchmark image2zvals(img, n_colors = 128, maxiter = 50, tol = 1e-2) seconds = 60        
BenchmarkTools.Trial: 32 samples with 1 evaluation.
 Range (min … max): 1.196 s … 2.685 s ┊ GC (min … max): 0.24% … 0.15%
 Time (median): 1.932 s ┊ GC (median): 0.19%
 Time (mean ± σ): 1.921 s ± 368.344 ms ┊ GC (mean ± σ): 0.17% ± 0.05%

                       █ ▃ ▃ █ █ ▃
  ▇▁▁▁▁▇▇▇▁▁▇▁▁▁▇▇▁▇▁▁▇█▁█▁█▁▁▁▁▁█▁▇▇▁▇▁█▇█▁▁▁▁▁▁▇▇▁▁▇▁▁▁▁▁▇ ▁
  1.2 s Histogram: frequency by time 2.69 s <

 Memory estimate: 47.21 MiB, allocs estimate: 335.

```