[ANN] GeoStats.jl

juliohm · July 1, 2023, 4:10pm

This topic will be used for future release updates of the GeoStats.jl framework. Past topics can be found at v0.10, v0.11, v0.14, v0.18, v0.24, v0.33 and v0.36.

OVERVIEW

GeoStats.jl v0.40 is out with revised documentation using Makie.jl recipes, new algorithms for simulation of spatial point patterns, and important improvements in geometric algorithms.

This is the last release with support for Julia v1.6 (LTS), next releases will require Julia v1.9.

RELEASE NOTES

FEATURES

New Repair transforms for fixing ill-formed meshes and geometries. These repair operations are parameterized with a number (e.g. Repair{0}) that is documented here.
Parameterization of geometries meaning that now one can call geometries as segment(t) with t \in [0,1] or quad(u, v) with (u, v) \in [0,1]^2 or hex(u, v, w) with (u, v, w) \in [0,1]^3 to get points in these geometries. The function isparameterized can be used to check if an implementation is available for a given geometry.
New JarvisMarch convex hull algorithm for 2D point sets besides the existing GrahamScan algorithm. This algorithm has O(nh) complexity where n is the number of points and h is the number of points in the hull. Compare it with the GrahamScan algorithm, which has O(n^2) complexity and you can see the speed up clearly when the convex hull has a very small number of vertices compared to the original point set (see docs).
New merge operation between geometries and meshes.
New vertex function to retrieve a single vertex of a Polytope geometry or Domain.
New pointify function to extract “vertices” of geometries even when they are not expressed in terms of a finite set of vertices. This function was useful to generalize some transforms and algorithms.
New Primitive geometries such as Cone, ConeSurface, Torus (see docs).
New MultiGridPath to traverse Grid subtypes in a hierarchical order.
New Selinger simplification algorithm to simplify polygons and closed chains (see docs).
New Potrace transform to trace closed regions with holes in raster images (see docs).
New Detrend transform to remove trends from geospatial data over any domain (see docs).
New inhomogeneous PoissonProcess, InhibitionProcess and ClusterProcess (see docs)

IMPROVEMENTS

Improvements to the default triangulation algorithm for polygons, which now performs a couple of repairs to the vertices before actual triangulation. This improvement was implemented to handle real data in agriculture projects at Arpeggeo®.
Less memory allocations in Chain algorithms that are now specialized for Ring and Rope subtypes.
Various implementations of missing methods for various geometry types such as discretization, sampling, convexhull, etc.
hasintersect methods for all 0D, 1D and 2D geometries. This means that we can now easily filter a heterogeneous collection of geometries that intersect any other geometry in 2D space (see docs).
Point is now a Geometry subtype. This enabled the generalization of a few algorithms in the stack.
Segment is now a Chain which is equivalent to a Polytope{1}.
Improvement of IO methods for geometries to display more meaningful names in interactive use.
Support for progress logging in all geostatistical simulation solvers.
The new documentation now makes extensive use of the visualization stack built with Makie.jl. The visualization stack built with Plots.jl is already in maintenance mode.
Various improvements to the GeoTables.jl package that can now load and save all types of geospatial data from disk including shp, geojson, gpkg, kml using pure Julia backends whenever possible. The package also implements fixes to the different backend’s results to improve the experience of end-users who don’t know or wan’t to deal with low-level interfaces.

BREAKING

Replace passive rotations from ReferenceFrameRotations.jl by active rotations from Rotations.jl
Replace Chain concrete type by an abstract type with two subtypes Ring (for closed chains) and Rope (for open chains). This way the closedness is part of the type, and we can avoid some internal copies of vertices.
Replace Collection by GeometrySet and add support for heterogeneous collections of geometries. This means that points, segments, rings, polygons, etc. can be processed together in some algorithms.
Refactoring of IntersectionType to a reduced list of types. This code simplification enabled more flexible intersections with meshes and domain types and also improved maintainability.
Refactoring of Cylinder and CylinderSurface constructors for greater usability.
Replace uniquecoords by UniqueCoords transform.

Acknowledgements

Thanks to all contributors that made this release possible

DianaPat jwscook vickydeka lihua-cat @stla @dorn-gerhard @jackbeagley @cserteGT3 @kylebeggs @ErickChacon @mfsch @longemen3000 @hyrodium conordoherty spaette @eliascarv

Would like to support us? Leave a on GitHub if you didn’t already!

juliohm · August 8, 2023, 7:11pm

GeoStats.jl v0.42

OVERVIEW

This release comes with various performance improvements after a complete review of Meshes.jl internals to use static lists of vertices in geometries whenever possible. We reviewed many algorithms, which are now free of unnecessary memory allocations, implemented geometric optimizations, and fixed various bugs found within industrial applications.

The project now uses package extensions from Julia v1.9 to load Makie.jl recipes automatically for end users. This means that all recipe packages (e.g. GeoStatsPlots.jl and GeoStatsViz.jl) are deprecated.

Users can now start their geospatial data science workflows with:

using GeoStats

import GLMakie as Mke # choose a backend

We highly recommend reading the updated Quickstart.

FEATURES

New robust estimator of variograms by Cressie & Hawkins (see docs)
New CylindricalTrajectory domain to represent wells in geosciences (see docs)
New recipes for plotting variograms and varioplanes (see docs)

IMPROVEMENTS

Estimation solvers (Kriging, IDW, LWR) now make use of a consistent set of neighborhood search methods and domain traversal options
Estimation (a.k.a. Interpolation) with Unitful.jl objects works with all solvers now, including affine units, which are automatically converted to absolute units (e.g. Celsius → Kelvin)
All loss functions from the LossFunctions.jl submodule are reexported by default
DensityRationEstimation.jl now loads optimization backends with package extensions
Better documentation for our geometric predicates (see docs)

Acknowledgements

We would like to acknowledge all contributors and supporters of this project. It is been super fun to develop it more, and continuously apply it in industry. We fixed tons of bugs and corner cases these last months with real data, and will continue to do so.

This last week we reached 5k tests in our Meshes.jl submodule alone, and this is evidence of our commitment to provide a robust experience to our users.

Join our community channel if you would like to receive more updates:

https://julialang.zulipchat.com/#narrow/stream/276201-geostats.2Ejl

juliohm · November 6, 2023, 9:33pm

GeoStats.jl v0.47

OVERVIEW

Various new features described in the Geospatial Data Science with Julia book and other improvements to the end-user experience.

FEATURES

New GeoTables.jl implementation with better methods, including efficient show methods with PrettyTables.jl:

julia> georef((a=rand(10, 10), b=rand(Int, 10, 10)))
100×3 GeoTable over 10×10 CartesianGrid{2,Float64}
┌────────────┬──────────────────────┬─────────────────────────────────────────┐
│     a      │          b           │                geometry                 │
│ Continuous │     Categorical      │               Quadrangle                │
│ [NoUnits]  │      [NoUnits]       │                                         │
├────────────┼──────────────────────┼─────────────────────────────────────────┤
│  0.516603  │ 6789588642647594681  │ Quadrangle((0.0, 0.0), ..., (0.0, 1.0)) │
│  0.998433  │ 9150524320540539883  │ Quadrangle((1.0, 0.0), ..., (1.0, 1.0)) │
│  0.263977  │ 4931893769895542016  │ Quadrangle((2.0, 0.0), ..., (2.0, 1.0)) │
│  0.427032  │ -3491437907112294822 │ Quadrangle((3.0, 0.0), ..., (3.0, 1.0)) │
│  0.906068  │ -7822066656579038303 │ Quadrangle((4.0, 0.0), ..., (4.0, 1.0)) │
│  0.556998  │ -2408182908005467907 │ Quadrangle((5.0, 0.0), ..., (5.0, 1.0)) │
│  0.881187  │ -1985601647472655905 │ Quadrangle((6.0, 0.0), ..., (6.0, 1.0)) │
│  0.64524   │ -4982792138183876424 │ Quadrangle((7.0, 0.0), ..., (7.0, 1.0)) │
│  0.495152  │ -5114448186852198900 │ Quadrangle((8.0, 0.0), ..., (8.0, 1.0)) │
│  0.288213  │ 8738486151747287386  │ Quadrangle((9.0, 0.0), ..., (9.0, 1.0)) │
│ 0.0709234  │ -4113799560159813356 │ Quadrangle((0.0, 1.0), ..., (0.0, 2.0)) │
│  0.223479  │ 5773465236920674180  │ Quadrangle((1.0, 1.0), ..., (1.0, 2.0)) │
│  0.940171  │ -3409105571799614662 │ Quadrangle((2.0, 1.0), ..., (2.0, 2.0)) │
│  0.139222  │ 4016690596386790401  │ Quadrangle((3.0, 1.0), ..., (3.0, 2.0)) │
│  0.81553   │ 5437077184799252660  │ Quadrangle((4.0, 1.0), ..., (4.0, 2.0)) │
│  0.734611  │ -9116300046534728476 │ Quadrangle((5.0, 1.0), ..., (5.0, 2.0)) │
│  0.827469  │ -1031902277811757514 │ Quadrangle((6.0, 1.0), ..., (6.0, 2.0)) │
│  0.463852  │ -7694874758022304794 │ Quadrangle((7.0, 1.0), ..., (7.0, 2.0)) │
│  0.430009  │  383006486861661319  │ Quadrangle((8.0, 1.0), ..., (8.0, 2.0)) │
│  0.114523  │ -6621173923176021272 │ Quadrangle((9.0, 1.0), ..., (9.0, 2.0)) │
│  0.453332  │ -9063031855995266727 │ Quadrangle((0.0, 2.0), ..., (0.0, 3.0)) │
│  0.848071  │ -2619300603579320095 │ Quadrangle((1.0, 2.0), ..., (1.0, 3.0)) │
│  0.791306  │ 5963621929024569325  │ Quadrangle((2.0, 2.0), ..., (2.0, 3.0)) │
│ 0.0642176  │ 2203217482885317616  │ Quadrangle((3.0, 2.0), ..., (3.0, 3.0)) │
│  0.420784  │ -5727490695202776996 │ Quadrangle((4.0, 2.0), ..., (4.0, 3.0)) │
│  0.468374  │ 9193956341378230306  │ Quadrangle((5.0, 2.0), ..., (5.0, 3.0)) │
│  0.836219  │ 8769101217160787406  │ Quadrangle((6.0, 2.0), ..., (6.0, 3.0)) │
│  0.473154  │ 7874585605426519179  │ Quadrangle((7.0, 2.0), ..., (7.0, 3.0)) │
│  0.507466  │ -3296044975691403512 │ Quadrangle((8.0, 2.0), ..., (8.0, 3.0)) │
│  0.973035  │ 2395941252317957301  │ Quadrangle((9.0, 2.0), ..., (9.0, 3.0)) │
│  0.12081   │ 7515712133164922671  │ Quadrangle((0.0, 3.0), ..., (0.0, 4.0)) │
│     ⋮      │          ⋮           │                    ⋮                    │
└────────────┴──────────────────────┴─────────────────────────────────────────┘
                                                                69 rows omitted

New DataScienceTraits.jl in place of ScientificTypes.jl with better support for Unitful.jl, DynamicQuantities.jl and missing values.
New GeoStatsProcesses.jl with stochastic fields and point processes
New GeoStatsTransforms.jl with various advanced geostatistical transforms
New GeoIO.jl for saving/loading geotables to different file formats
New GeoArtifacts.jl for loading geotables from different databases
Support for geotables with geometry column only:

julia> georef(nothing, CartesianGrid(10,10))
100×1 GeoTable over 10×10 CartesianGrid{2,Float64}
┌─────────────────────────────────────────┐
│                geometry                 │
│               Quadrangle                │
│                                         │
├─────────────────────────────────────────┤
│ Quadrangle((0.0, 0.0), ..., (0.0, 1.0)) │
│ Quadrangle((1.0, 0.0), ..., (1.0, 1.0)) │
│ Quadrangle((2.0, 0.0), ..., (2.0, 1.0)) │
│ Quadrangle((3.0, 0.0), ..., (3.0, 1.0)) │
│ Quadrangle((4.0, 0.0), ..., (4.0, 1.0)) │
│ Quadrangle((5.0, 0.0), ..., (5.0, 1.0)) │
│ Quadrangle((6.0, 0.0), ..., (6.0, 1.0)) │
│ Quadrangle((7.0, 0.0), ..., (7.0, 1.0)) │
│ Quadrangle((8.0, 0.0), ..., (8.0, 1.0)) │
│ Quadrangle((9.0, 0.0), ..., (9.0, 1.0)) │
│ Quadrangle((0.0, 1.0), ..., (0.0, 2.0)) │
│ Quadrangle((1.0, 1.0), ..., (1.0, 2.0)) │
│ Quadrangle((2.0, 1.0), ..., (2.0, 2.0)) │
│ Quadrangle((3.0, 1.0), ..., (3.0, 2.0)) │
│ Quadrangle((4.0, 1.0), ..., (4.0, 2.0)) │
│ Quadrangle((5.0, 1.0), ..., (5.0, 2.0)) │
│ Quadrangle((6.0, 1.0), ..., (6.0, 2.0)) │
│ Quadrangle((7.0, 1.0), ..., (7.0, 2.0)) │
│ Quadrangle((8.0, 1.0), ..., (8.0, 2.0)) │
│ Quadrangle((9.0, 1.0), ..., (9.0, 2.0)) │
│ Quadrangle((0.0, 2.0), ..., (0.0, 3.0)) │
│ Quadrangle((1.0, 2.0), ..., (1.0, 3.0)) │
│ Quadrangle((2.0, 2.0), ..., (2.0, 3.0)) │
│ Quadrangle((3.0, 2.0), ..., (3.0, 3.0)) │
│ Quadrangle((4.0, 2.0), ..., (4.0, 3.0)) │
│ Quadrangle((5.0, 2.0), ..., (5.0, 3.0)) │
│ Quadrangle((6.0, 2.0), ..., (6.0, 3.0)) │
│ Quadrangle((7.0, 2.0), ..., (7.0, 3.0)) │
│ Quadrangle((8.0, 2.0), ..., (8.0, 3.0)) │
│ Quadrangle((9.0, 2.0), ..., (9.0, 3.0)) │
│ Quadrangle((0.0, 3.0), ..., (0.0, 4.0)) │
│                    ⋮                    │
└─────────────────────────────────────────┘
                            69 rows omitted

BREAKING

Deprecation of GeoClustering.jl, which now lives in GeoStatsTransforms.jl
Deprecation of PointPatterns.jl, which now lives in GeoStatsProcesses.jl
Kriging, IDW, etc. are now GeoStatsModels.jl instead of solvers
GeoTables.jl now only holds the GeoTable type, and GeoIO.jl is the new home of load/save functions

Join our community channel if you need help updating.

gvdr · November 7, 2023, 12:26am

Amazing.

Is, by any chance, anyone thinking/working on levereging the new Makie datashader for geo visualization?

juliohm · November 7, 2023, 12:43am

Not working at the moment, but any contribution is welcome.

juliohm · December 21, 2023, 9:16pm

GeoStats.jl v0.49

New exciting features, including lazy transforms on meshes. This means that we can now map large geotiff, netcdf, grib, etc files to lazy TransformedGrid with the correct coordinates from the file, without any memory allocation:

using GeoStats
using GeoIO

geotable = GeoIO.load("world.tif")

2097152×4 GeoTable over 2097152 TransformedGrid{2,Float64}
┌─────────────┬─────────────┬─────────────┬────────────────────────────────────────────────────────┐
│    BAND1    │    BAND2    │    BAND3    │                        geometry                        │
│ Categorical │ Categorical │ Categorical │                       Quadrangle                       │
│  [NoUnits]  │  [NoUnits]  │  [NoUnits]  │                                                        │
├─────────────┼─────────────┼─────────────┼────────────────────────────────────────────────────────┤
│     12      │     11      │     68      │   Quadrangle((-180.0, 90.0), ..., (-180.0, 89.8242))   │
│     12      │     11      │     68      │ Quadrangle((-179.824, 90.0), ..., (-179.824, 89.8242)) │
│     12      │     11      │     68      │ Quadrangle((-179.648, 90.0), ..., (-179.648, 89.8242)) │
│     12      │     11      │     68      │ Quadrangle((-179.473, 90.0), ..., (-179.473, 89.8242)) │
│     12      │     11      │     68      │ Quadrangle((-179.297, 90.0), ..., (-179.297, 89.8242)) │
│     12      │     11      │     68      │ Quadrangle((-179.121, 90.0), ..., (-179.121, 89.8242)) │
│      ⋮      │      ⋮      │      ⋮      │                           ⋮                            │
└─────────────┴─────────────┴─────────────┴────────────────────────────────────────────────────────┘
                                                                                2097146 rows omitted

Even though this grid has 2048 x 1024 quadrangles, we can see that the size of the geometry column is 120 bytes:

sizeof(geotable.geometry)

We can still visualize the grid efficiently with Makie:

import GLMakie as Mke

viz(geotable.geometry, color = geotable.BAND1)

This puts us one step closer to the new coordinate system infrastructure, which is planned for the first semester of 2024.

Happy new year!

juliohm · January 22, 2024, 10:23am

GeoStats.jl v0.49.2

Multi-threaded geojoin
3D intersects
viz of large Polygon sets

and other small improvements on simulation of geospatial stochastic processes (e.g. Gaussian).

juliohm · March 27, 2024, 4:56pm

GeoStats.jl v0.53

New transforms:
- Aggregate
- Transfer
- Downscale
- Upscale

Given a geotable over a grid:

X = [i/20 * cos(3π/2 * (j-1) / (30-1)) for i in 1:20, j in 1:30]
Y = [i/20 * sin(3π/2 * (j-1) / (30-1)) for i in 1:20, j in 1:30]
sgrid = StructuredGrid(X, Y)
gtb = georef((a=rand(Float64, 551), b=rand(Int, 551)), sgrid)

These transforms can be used to change the resolution, or transfer the variables to other domains with nearest neighbor interpolation:

ngtb = gtb |> Downscale(2, 2)

They work with any Grid subtype, including CartesianGrid, RectilinearGrid and StructuredGrid. Please check the docstrings to learn more.

juliohm · April 8, 2024, 4:59pm

GeoStats.jl v0.54

Various fixes and improvements to color treatment in viz and viewer.

Breaking

showfacets has been renamed to showsegments and showpoints
facetcolor has been renamed to segmentcolor and pointcolor
colorscheme has been renamed to colormap

Features

Now it is possible to pass the colorrange option to customize the color limits associated with a vector of values.
New cbar for colorbars based on custom vectors of arbitrary Julia objects, including missing values, categorical, Distributions.jl, etc.