# Any advice on Stereo Matching in Julia?

**URL:** <https://discourse.julialang.org/t/any-advice-on-stereo-matching-in-julia/53552>\
**Category:** Visualization\
**Tags:** machine-vision\
**Created:** [January 18, 2021, 5:47pm UTC](https://discourse.julialang.org/t/any-advice-on-stereo-matching-in-julia/53552 "2021-01-18T17:47:24Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Erik\_Engheim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erik_engheim/32/8122_2.png) [@Erik\_Engheim](https://discourse.julialang.org/u/Erik_Engheim)\
**Post date:** [January 18, 2021, 5:47pm UTC](https://discourse.julialang.org/t/any-advice-on-stereo-matching-in-julia/53552/1 "2021-01-18T17:47:24Z")

</div>

I am looking to do stereo matching in Julia, basically to find the depth in an image by using two images. I am at an early stage and would appreciate some advice on where to look for information, and what libraries are most relevant. I have looked at Julia Images obviously. But what else? Is OpenCV with Julia bindings a good choice or are there better libraries for this particular problem?

---

<div class="post-metadata">

**Author:** ![Palli](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/palli/32/3380_2.png) [@Palli](https://discourse.julialang.org/u/Palli)\
**Post date:** [January 18, 2021, 10:51pm UTC](https://discourse.julialang.org/t/any-advice-on-stereo-matching-in-julia/53552/2 "2021-01-18T22:51:12Z")

</div>

Based on this file, there are several algorithms, latest STEREO\_HH4 was added last August: [Update the stereo sample: · opencv/opencv@6d1f7c2 · GitHub](https://github.com/opencv/opencv/commit/6d1f7c2b1b5de024083ca1c3cf515e9d48cd9fb4#diff-adc4ede825303b1ea147841e83cc0bc1aeb147d5476aebbe3f00ceb53874453a)

While in 2015 STEREO\_3WAY was added: [Adding new HAL-accelerated MODE\_SGBM\_3WAY · opencv/opencv@aea4157 · GitHub](https://github.com/opencv/opencv/commit/aea4157340b3136e67c02a5565b152dbe5c73f2e#diff-adc4ede825303b1ea147841e83cc0bc1aeb147d5476aebbe3f00ceb53874453a)

The latest developments do not seem to be there, with many papers submitted in Dec. or Nov. last year: [The KITTI Vision Benchmark Suite](http://www.cvlibs.net/datasets/kitti/eval_scene_flow.php?benchmark=stereo)

These OpenCV algorithms seem all to be old (variants for better speed, not accuracy?), so outdated?  
Despite it explained here as [https://www.nigeltve.com/stereo-vision](https://www.nigeltve.com/stereo-vision)

> Stereo semi-global block matching (SGBM) is a strong choice when it comes to gathering depth information of the surroundings.

You could wrap/use any of the code I link to, or reimplement in Julia (I did not find any Julia-only code for this).

> **[Review of Stereo Matching Algorithms Based on Deep Learning](https://www.hindawi.com/journals/cin/2020/8562323/)**
>
> Stereo vision is a flourishing field, attracting the attention of many researchers. Recently, leveraging on the development of deep learning, stereo matching algorithms have achieved remarkable performance far exceeding traditional approaches. This...

I did start at looking at [paperswithcode.com](http://paperswithcode.com), for “stereo matching” (some paper below belong primarily to other categories), but note there are also other stereo-related sub-categories.

What I quickly found, before the links above, maybe in wrong order (or not all relevant):

> **[Papers with Code - AANet: Adaptive Aggregation Network for Efficient Stereo...](https://paperswithcode.com/paper/aanet-adaptive-aggregation-network-for)**
>
> 1 code implementation in PyTorch. Despite the remarkable progress made by learning based stereo matching algorithms, one key challenge remains unsolved. Current state-of-the-art stereo models are mostly based on costly 3D convolutions, the cubic...

> The implementation of improved version **AANet+ (stronger performance & slightly faster speed)** is also included in this repo.

> **[GitHub - HKBU-HPML/FADNet](https://github.com/HKBU-HPML/FADNet)**
>
> Contribute to HKBU-HPML/FADNet development by creating an account on GitHub.

another real-time:

> the proposed method can  
> achieve accurate depth estimation in real-time inference. Experimental  
> results demonstrated that the proposed method processed stereo image  
> pairs with resolution 1242×375 at 12-33 fps on an NVIDIA Jetson TX2  
> module and achieved competitive accuracy in depth estimation. The code  
> is available at [GitHub - JiaRenChang/RealtimeStereo: Attention-Aware Feature Aggregation for Real-time Stereo Matching on Edge Devices (ACCV, 2020)](https://github.com/JiaRenChang/RealtimeStereo)

not sure state-of-the-art, they claim, still applies, since older:

> **[GitHub - uber-research/DeepPruner: DeepPruner: Learning Efficient Stereo...](https://github.com/uber-research/DeepPruner)**
>
> DeepPruner: Learning Efficient Stereo Matching via Differentiable PatchMatch (ICCV 2019) - GitHub - uber-research/DeepPruner: DeepPruner: Learning Efficient Stereo Matching via Differentiable Patch...

> [**KITTI**](http://www.cvlibs.net/datasets/kitti/eval_scene_flow.php?benchmark=stereo): **Results competitive to SOTA, while being real-time (8x faster than SOTA). SOTA among published real-time algorithms**.

> **[GitHub - feihuzhang/GANet: GA-Net: Guided Aggregation Net for End-to-end...](https://github.com/feihuzhang/GANet)**
>
> GA-Net: Guided Aggregation Net for End-to-end Stereo Matching - GitHub - feihuzhang/GANet: GA-Net: Guided Aggregation Net for End-to-end Stereo Matching

> **[Epipolar Transformers](https://arxiv.org/abs/2005.04551)**
>
> A common approach to localize 3D human joints in a synchronized and calibrated multi-view setup consists of two-steps: (1) apply a 2D detector separately on each view to localize joints in 2D, and (2) perform robust triangulation on 2D detections...

> **[Adaptive Unimodal Cost Volume Filtering for Deep Stereo Matching](https://arxiv.org/abs/1909.03751)**
>
> State-of-the-art deep learning based stereo matching approaches treat disparity estimation as a regression problem, where loss function is directly defined on true disparities and their estimated ones. However, disparity is just a byproduct of a...

> **[GitHub - ucbdrive/hd3: Code for Hierarchical Discrete Distribution...](https://github.com/ucbdrive/hd3)**
>
> Code for Hierarchical Discrete Distribution Decomposition for Match Density Estimation (CVPR 2019) - GitHub - ucbdrive/hd3: Code for Hierarchical Discrete Distribution Decomposition for Match Densi...

> **[Hierarchical Discrete Distribution Decomposition for Match Density Estimation](https://arxiv.org/abs/1812.06264)**
>
> Explicit representations of the global match distributions of pixel-wise correspondences between pairs of images are desirable for uncertainty estimation and downstream applications. However, the computation of the match density for each pixel may be...

> Despite its simplicity, our probabilistic method achieves state-of-the-art results for both optical flow and stereo matching on established benchmarks.  
> [[1803.08669] Pyramid Stereo Matching Network](https://arxiv.org/abs/1803.08669)

See also: [JuliaImages Projects – Summer of Code](https://cn.julialang.org/jsoc/gsoc/images/index.html)

I wasn’t aware Uber was also into NLP: [GitHub - uber-research/PPLM: Plug and Play Language Model implementation. Allows to steer topic and attributes of GPT-2 models.](https://github.com/uber-research/PPLM) see blog:

[https://eng.uber.com/pplm/](https://eng.uber.com/pplm/)

NLP investigation lead me to Arxiv-nlp at: [https://transformer.huggingface.co/](https://transformer.huggingface.co/) where I tried to auto-complete: “What’s the state-of-the art stereo matching”

> **[GitHub - NoamRosenberg/autodeeplab: AutoDeeplab / auto-deeplab / AutoML for...](https://github.com/NoamRosenberg/autodeeplab)**
>
> AutoDeeplab / auto-deeplab / AutoML for semantic segmentation, implemented in Pytorch - GitHub - NoamRosenberg/autodeeplab: AutoDeeplab / auto-deeplab / AutoML for semantic segmentation, implemente...

---

<div class="post-metadata">

**Author:** ![Erik\_Engheim](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/erik_engheim/32/8122_2.png) [@Erik\_Engheim](https://discourse.julialang.org/u/Erik_Engheim)\
**Post date:** [January 19, 2021, 11:05am UTC](https://discourse.julialang.org/t/any-advice-on-stereo-matching-in-julia/53552/3 "2021-01-19T11:05:56Z")

</div>

Wow thanks! I don’t need specific help with the algorithms per say as I have a colleague, who seems to have a handle on this. The main challenge has been to uncover what Julia code already exists for this and if no whole solution exists, what sort fo building blocks or libraries are most useful beyond Images.jl