# \[ANN\] GeoStats.jl v0.36

**URL:** <https://discourse.julialang.org/t/ann-geostats-jl-v0-36/90708>\
**Category:** Package Announcements\
**Tags:** package, announcement, statistics, geo, geostatistics\
**Created:** [November 23, 2022, 1:39pm UTC](https://discourse.julialang.org/t/ann-geostats-jl-v0-36/90708 "2022-11-23T13:39:19Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![juliohm](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/juliohm/32/215266_2.png) [@juliohm](https://discourse.julialang.org/u/juliohm)\
**Post date:** [November 23, 2022, 1:39pm UTC](https://discourse.julialang.org/t/ann-geostats-jl-v0-36/90708/1 "2022-11-23T13:39:19Z")

</div>

# OVERVIEW

[GeoStats.jl](https://github.com/JuliaEarth/GeoStats.jl) v0.36 is out with major usability and performance improvements. We are one step closer to a unique experience for **geodata science** and **geostatistical learning** in pure Julia.

TL;DR this release provides **advanced geospatial pipelines and split-apply-combine** , efficient `DataFrame`-like access to geospatial data, and major performance improvements in interactive visualization with Makie.

## RELEASE NOTES

### FEATURES

- New macros `@groupby`, `@transform` and `@combine` for **geospatial** split-apply-combine
- Full integration with table transforms from [TableTransforms.jl](https://github.com/JuliaML/TableTransforms.jl) and new **geospatial** pipelines
- New `DataFrame`-like interface for geospatial data (e.g. df[rows,cols])
- Automatic selection of colorschemes from scientific types of variables
- New `GridTopology` with new adjacency topological relations
- New `RectilinearGrid` for constructing grids from x, y, z, … coordinates
- New `LaplaceSmoothing` geometric transform as a companion to `TaubinSmoothing`
- New `ProjectionPursuit` multivariate feature transform
- New geostatistical spectral clustering `GSC` algorithm
- New options for `DouglasPeucker` simplification of geometries
- New intersection algorithms between `Ray`, `Segment`, `Line`, `Triangle`, …
- New `perimeter` function and implementations of `measure`, `area`, `volume`, …
- New constructors with iterators for `PointSet`, `GeometrySet`, `Collection`, `Multi`, …

### IMPROVEMENTS

- Major visualization speedups with Makie.jl with Dehn’s 1899 triangulation from [Meshes.jl](https://github.com/JuliaGeometry/Meshes.jl)
- Refactored geospatial partition methods and fixed integration with [MLJ.jl](https://github.com/alan-turing-institute/MLJ.jl) clustering
- Refactored show methods of various types to emphasize geospatial domains
- Revived GPU support for `IQ` geostatistical simulation solver
- Updated documentation with new recommended methods for geospatial data

### BREAKING

- Drop support for [Query.jl](https://github.com/queryverse/Query.jl) in favor of [TableTransforms.jl](https://github.com/JuliaML/TableTransforms.jl)
- Drop indexable interface from `Multi` in favor of `collect`
- Drop `embeddim` and `coordtype` support for geospatial data
- Drop `ClusteringTask` in favor of `cluster` interface function
- Rename validation methods (e.g. `CrossValidation` → `KFoldValidation`)

## DEMONSTRATION

Some of the new features are demonstrated in a recent video:

[![](https://global.discourse-cdn.com/julialang/original/3X/3/0/30a9ed05c44d6b0ddd991bcd89a0c59ac9eb3db7.jpeg "Arpeggeo® - Geodata Science") ](https://www.youtube.com/watch?v=1opCT2lId88)

Below we demonstrate the features with simpler data sets.

First we load the necessary packages:

```julia
julia> using GeoStats # all-batteries included!

julia> using GeoStatsViz # except for visualization

julia> import GLMakie as Mke # choose Makie backend

```

### Geospatial split-apply-combine

Create geospatial data over 2D grid with features `a`, `b`, `c`:

```julia
julia> data = georef((a = rand(10, 10), b = rand(10, 10), c=rand(1:4, 10, 10)))
10×10 CartesianGrid{2,Float64}
  variables (rank 2)
    └─a (Float64)
    └─b (Float64)
    └─c (Int64)

```

We can easily group by `c` to produce a geospatial partition:

```julia
julia> groups = @groupby(data, :c)
4 Partition
  └─22 View{100 MeshData}
  └─29 View{100 MeshData}
  └─33 View{100 MeshData}
  └─16 View{100 MeshData}
  metadata: rows, names

```

We can then transform the groups with transforms that involve both the features and the special `geometry` column:

```julia
julia> transf = @transform(groups, :d = 2*:a + area(:geometry))
4 Partition
  └─22 View{100 MeshData}
  └─29 View{100 MeshData}
  └─33 View{100 MeshData}
  └─16 View{100 MeshData}
  metadata: rows, names

```

Finally, we can combine the results of the transformed groups:

```julia
julia> result = @combine(transf, :e = mean(:d))
4 GeometrySet{2,Float64}
  variables (rank 2)
    └─c (Float64)
    └─e (Float64)

```

Notice that, unlike in the video above, we now changed the behavior to return `Multi` geometries for the groups instead of the centroids:

```julia
julia> result.geometry
4 GeometrySet{2,Float64}
  └─22 MultiNgon{2,Float64}
  └─29 MultiNgon{2,Float64}
  └─33 MultiNgon{2,Float64}
  └─16 MultiNgon{2,Float64}

julia> viz(result.geometry, color = 1:4)

```

 ![image](https://global.discourse-cdn.com/julialang/original/3X/c/2/c22af036cead0e345c4cf5783ed3e268bfffb76a.png)

Also notice how the `geometry` column has a special `Domain` type, which is lazy in many cases avoiding the construction of expensive geometries in large datasets:

```julia
julia> data.geometry
10×10 CartesianGrid{2,Float64}
  minimum: Point(0.0, 0.0)
  maximum: Point(10.0, 10.0)
  spacing: (1.0, 1.0)

```

### Geospatial transform pipelines

We can combine feature transforms from TableTransforms.jl with geometric transforms from Meshes.jl to create truly geospatial pipelines.

For example, we can create a pipeline to standardize the coordinates of the domain, then compute the z-score of the features `a` and `b`:

```julia
julia> pipe = StdCoords() → Select(:a, :b) → ZScore()
SequentialTransform
├─ StdCoords()
├─ Select([:a, :b], nothing)
└─ ZScore(all)

```

Applying the pipeline to a `CartesianGrid` domain leads to a `SimpleMesh` domain, but the underlying topology is still a `GridTopology`:

```julia
julia> score = data |> pipe
100 SimpleMesh{2,Float64}
  variables (rank 2)
    └─a (Float64)
    └─b (Float64)

julia> topology(domain(data))
10×10 GridTopology(aperiodic, aperiodic)

julia> topology(domain(score))
10×10 GridTopology(aperiodic, aperiodic)

```

We can visualize both data sets (notice the second in the bottom left corner):

```julia
julia> viz(data, variable = :a)

julia> viz!(score, variable = :a, colorscheme = :coolwarm)

```

 ![image](https://global.discourse-cdn.com/julialang/original/3X/d/e/deebedd60d92f1afd6debecc6deabce58b1f8867.png)

## DataFrame-like interface

We now support efficient access to “rows” and “columns” of geospatial data over arbitrary domains:

```julia
julia> data[1:3,:]
3 View{10×10 CartesianGrid{2,Float64}}
  variables (rank 2)
    └─a (Float64)
    └─b (Float64)
    └─c (Int64)

julia> data[1,:]
(a = 0.4986869004101653, b = 0.9076027666401961, c = 4, geometry = Quadrangle(Point(0.0, 0.0), Point(1.0, 0.0), Point(1.0, 1.0), Point(0.0, 1.0)))

julia> data[:,:a]
100-element Vector{Float64}:
 0.4986869004101653
 0.04423514219396685
 0.7431805912560505
 0.5488350636776107
 0.1781593152690658
 0.34287338202772666
 0.01556466336416984
 0.08127752655318943
 0.2594584424431915
 0.7372470841115343
 0.5219963894546638
 0.2632569123088946
 0.5402482890222606
 0.7302507649502713
 0.30546916759280573
 0.8386743894750512
 0.5462308898202195
 0.5822888075805445
 ⋮
 0.43027481017912594
 0.38445713289998273
 0.9287552353217178
 0.7189205200160556
 0.4372832350782744
 0.8527347769479305
 0.31562905739426095
 0.6731884280856284
 0.5870119105459504
 0.5237663202982149
 0.07350755294901445
 0.7963836858702719
 0.650505284402908
 0.3696365329236847
 0.9032714797251225
 0.48907464522848165
 0.7947529092094066
 0.280343959354032

```

### Automatic selection of color schemes

If you are explicit about your scientific types, the visualization pipeline will suggest color schemes that are friendly to color-blind readers:

```julia
julia> scidata = data |> Coerce(:c => Multiclass)
10×10 CartesianGrid{2,Float64}
  variables (rank 2)
    └─a (Float64)
    └─b (Float64)
    └─c (CategoricalArrays.CategoricalValue{Int64, UInt32})

julia> viz(scidata, variable = :c)

```

 ![image](https://global.discourse-cdn.com/julialang/original/3X/1/0/106e04c1449efa7de0add8835f827ab1d588bcde.png)

And of course, all these features work with general unstructured meshes, point sets, geometry sets, and all other domain types provided by Meshes.jl.

## Acknowledgements

Thanks to all contributors (cc: Elias Carvalho (@eliascarv), Gerhard Dorn (@dorn-gerhard), Claro Henrique (@ClaroHenrique), Diana Aldana). We are finally reaching the user experience I dreamed years ago when GeoStats.jl was just a tiny research package.

Would like to support us? Leave a ⭐ on GitHub if you didn’t already: [GitHub - JuliaEarth/GeoStats.jl: An extensible framework for geospatial data science and geostatistical modeling fully written in Julia](https://github.com/JuliaEarth/GeoStats.jl)

Follow our company page on LinkedIn and Twitter for updates and job opportunities:

[https://twitter.com/arpeggeotech](https://twitter.com/arpeggeotech)

> **[Arpeggeo® | LinkedIn](https://www.linkedin.com/company/arpeggeo)**
>
> Arpeggeo® | 947 followers on LinkedIn. Advancing machine learning in geosciences | At Arpeggeo® we believe that geostatistical knowledge is key to advance machine learning technology in industries that manage geospatial resources (e.g., mining,...