# Primer feasibility question - modelling soft tissue behavior using ML

**URL:** <https://discourse.julialang.org/t/primer-feasibility-question-modelling-soft-tissue-behavior-using-ml/76175>\
**Category:** Machine Learning\
**Created:** [February 10, 2022, 3:23pm UTC](https://discourse.julialang.org/t/primer-feasibility-question-modelling-soft-tissue-behavior-using-ml/76175 "2022-02-10T15:23:56Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![davide445](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davide445/32/33681_2.png) [@davide445](https://discourse.julialang.org/u/davide445)\
**Post date:** [February 10, 2022, 3:23pm UTC](https://discourse.julialang.org/t/primer-feasibility-question-modelling-soft-tissue-behavior-using-ml/76175/1 "2022-02-10T15:23:56Z")

</div>

Get interested in Julia since seems crossing many of the themes i’m touching lately: HPC, biophysics, GPGPU, ML, with the goal to create interactive systems.

Not a coder, I’m a solution designer that is deepening on different topics enough to understand what’s possibile and what approach is better to use (or avoid).

Normally modifying other code to test or just doing small programs building line by line to test and learn.

A past long ago as plain C scientific coder, past experience on Matlab for testing on quant finance models, recent experiences in Python using FastAI to learn about the possibilities of ML.

Didn’t like Python (just personal preference), wanted to approach HPC and GPGPU as personal experience, C++ really too steep to unlock GPU potential, Rust not really dedicated, C# not really used in this domain, reading about Julia I like his focus on HPC and elegant approach (even if I’m not sure to like Matlab-like syntax).

Wanted to ask if might be a too steep work starting with Flux/Fastai.jl/SciML to achieve something like this

 ![image](https://global.discourse-cdn.com/julialang/original/3X/5/9/5902166a64a9e4875569aba53bd5f1e09995cbd2.png)

based on [this](https://arxiv.org/abs/1904.00722) work (so an U-Net derivative), just to have an interesting goal.

---

<div class="post-metadata">

**Author:** ![ChrisRackauckas](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chrisrackauckas/32/77_2.png) [@ChrisRackauckas](https://discourse.julialang.org/u/ChrisRackauckas)\
**Post date:** [February 10, 2022, 4:23pm UTC](https://discourse.julialang.org/t/primer-feasibility-question-modelling-soft-tissue-behavior-using-ml/76175/2 "2022-02-10T16:23:18Z")

</div>

> [@davide445](#):
>
> Wanted to ask if might be a too steep work starting with Flux/Fastai.jl/SciML to achieve something like this

Looks right up our ally.

> **[GitHub - gridap/Gridap.jl: Grid-based approximation of partial differential...](https://github.com/gridap/Gridap.jl)**
>
> Grid-based approximation of partial differential equations in Julia - GitHub - gridap/Gridap.jl: Grid-based approximation of partial differential equations in Julia

This has a lot of the FEM meshing tools you need from your paper. I don’t know if GMesh is up to date specifically, but I believe so.

UNets are here:

> **[GitHub - DhairyaLGandhi/UNet.jl: Generic UNet implementation written in pure...](https://github.com/DhairyaLGandhi/UNet.jl)**
>
> Generic UNet implementation written in pure Julia, based on Flux.jl - GitHub - DhairyaLGandhi/UNet.jl: Generic UNet implementation written in pure Julia, based on Flux.jl

Connecting it to [GitHub - SciML/OperatorLearning.jl: No need to train, he's a smooth operator](https://github.com/SciML/OperatorLearning.jl) could be an interesting possibility. I’m quite interested in this field as well so feel free to follow up with an email.

---

<div class="post-metadata">

**Author:** ![KZiemian](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kziemian/32/9020_2.png) [@KZiemian](https://discourse.julialang.org/u/KZiemian)\
**Post date:** [April 7, 2022, 6:12pm UTC](https://discourse.julialang.org/t/primer-feasibility-question-modelling-soft-tissue-behavior-using-ml/76175/3 "2022-04-07T18:12:09Z")

</div>

I spend some time talking with @davide445 and below I wrote summary of his problem.

Final aim is to develop simulator of surgery navigation for educational purpose, that can be interacted with by using haptics device. It is easier to show it than explaining by words, so check this [video](https://www.youtube.com/watch?v=c4q0TbVdSEs).

For this simulator be useful it must work in real time, rendering 60 frames per second (60 FPS), with roughly 30 ms per frame. You need to ask @davide445 why this 30 ms is so important, I don’t know that. Today this can be done for simulations when the main interest is operation on [bones](https://www.youtube.com/watch?v=p4vO64Y27JE), because bones can be treated in large extent as rigid bodies (no surprise here). But making a simulation of soft tissue e.g. liver is much more challenging computationally. Again, no surprise.

Program **must** works in real time and **must** simulate cutting of soft tissue in the way that is realistic enough.

Program of simulation roughly operate in the way listed blow. @davide445, I hope I get it mostly right.

1. Take data of liver and produce it visual model.
2. Take user input.
3. Produce data input for PDEs that describe movement of the liver.
4. Send PDEs data input to solver.
5. Solve PDEs with given PDEs data input, generated PDEs data output.
6. Send PDEs data output from solver to visualisation of solver.
7. Go back to point 1.

Main question of @davide445 is: should he give Julia and SciML ecosystem as computation engine in point 5? In other words, we can assume that visualisation and I/O software is ready and we want use Julia only for computing time evolution of liver. Is this worth a try?

The goal is to find such a way to make such simulation possible under time constrain of 30 ms, while only some part of this time can be used for solving problem.

My intuition is: yes, Julia and SciML can give you as good performance in solving differential equations as you can get. If some solver is faster, you just open [GitHub issue](https://github.com/SciML) “Hey, SciML solver is slower than solver X.” and then collaboration will work until they remove this bug (this is how Chris Rackauckas describe their workflow in one talk from 2020). But there are few caveats.

First caveat. From what I understand @davide445 and his collaboration still aren’t sure if solving PDEs isn’t to big bottle neck and in the end some Machine Learning is needed to make computation fast enough. I trust SciML when you need to solve differential equations and never check they ML ability. I’m also agnostic about how good SciML is in mixing PDEs solvers with ML algorithms, I never put too much attention to neural differential equations.

Second caveat. In point 4 and 5 you need pipe Julia code with external software, this can be a big bottle neck. From what @davide445 told me, they have their own program that works entirely on GPU, where CUDA is used for communication, sharing memory and all other things. I’m not GPU guy, so I may understand him wrong and I can’t judge if would it be a problem or not.

At this stage I don’t know how much time implementing such thing in Julia would take. It can be work for 3 days or for 3 months and @davide445 need opinion someone knowledgeable is this effort worth pursuing or not? I can’t give him such answer and he need it to the end of the April at most.

---

<div class="post-metadata">

**Author:** ![KZiemian](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kziemian/32/9020_2.png) [@KZiemian](https://discourse.julialang.org/u/KZiemian)\
**Post date:** [April 7, 2022, 6:38pm UTC](https://discourse.julialang.org/t/primer-feasibility-question-modelling-soft-tissue-behavior-using-ml/76175/4 "2022-04-07T18:38:09Z")

</div>

Another question from @davide445 was: when to find skilled Julia developer? I suggested posting job offer here and on Julia Zulip, but he probably welcome all good ideas.

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [April 7, 2022, 6:39pm UTC](https://discourse.julialang.org/t/primer-feasibility-question-modelling-soft-tissue-behavior-using-ml/76175/5 "2022-04-07T18:39:55Z")

</div>

> [@KZiemian](#):
>
> It can be work for 3 days or for 3 months

… or 3 years (this doesn’t sound like a master thesis).

---

<div class="post-metadata">

**Author:** ![KZiemian](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kziemian/32/9020_2.png) [@KZiemian](https://discourse.julialang.org/u/KZiemian)\
**Post date:** [April 7, 2022, 7:40pm UTC](https://discourse.julialang.org/t/primer-feasibility-question-modelling-soft-tissue-behavior-using-ml/76175/6 "2022-04-07T19:40:35Z")

</div>

Indeed. I should add “or 3 years”.

---

<div class="post-metadata">

**Author:** ![davide445](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davide445/32/33681_2.png) [@davide445](https://discourse.julialang.org/u/davide445)\
**Post date:** [April 7, 2022, 8:00pm UTC](https://discourse.julialang.org/t/primer-feasibility-question-modelling-soft-tissue-behavior-using-ml/76175/7 "2022-04-07T20:00:57Z")

</div>

> [@KZiemian](#):
>
> surgery navigation for educational purpose

Thanks @KZiemian , just to point out a few details: surgery nativation it’s a specific kind of surgery intraoperative system, our goal is to create a pre-surgery training simulator.

> [@KZiemian](#):
>
> rendering 60 frames per second (60 FPS), with is roughly 30 ms per frame

The min goal for VR visualization is 30 fps, meaning btw every frame there are 33ms, so a reasonable “round” goal is 30ms of response time for the simulation. An ideal goal for VR visualization is to have below 20ms latency, meaning a min 50fps visualization.

> [@KZiemian](#):
>
> Take data of liver and produce it visual model.

Just to point out a liver it’s just an example of a goal, the final product can be focused on a different anatomic area. Also need to add the haptic feedback as a requirements, that need to run at 1khz update, so will probably use a separate and more coarse simulation syncronized with the visual one.

> [@KZiemian](#):
>
> It can be work for 3 days or for 3 months and @davide445 need opinion someone knowledgeable is this effort worth pursuing or not? I can’t give him such answer and he need it to the end of the April at most

The main goal is to have a 6 months POC demonstrating all the basic features and if succesful a 3 years project to deliver a working and complete solution. The POC project will start probably before end of June.

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [April 7, 2022, 8:14pm UTC](https://discourse.julialang.org/t/primer-feasibility-question-modelling-soft-tissue-behavior-using-ml/76175/8 "2022-04-07T20:14:44Z")

</div>

> [@davide445](#):
>
> The min goal for VR visualization is 30 fps, meaning btw every frame there are 33ms, so a reasonable “round” goal is 30ms of response time for the simulation. An ideal goal for VR visualization is to have below 20ms latency, meaning a min 50fps visualization.

A layman’s view: you can throw quite a bit of hardware on this problem (some cluster or tensor processing units). IMHO you should have an idea how parallelizable your algorithms will be.

---

<div class="post-metadata">

**Author:** ![KZiemian](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kziemian/32/9020_2.png) [@KZiemian](https://discourse.julialang.org/u/KZiemian)\
**Post date:** [April 7, 2022, 8:50pm UTC](https://discourse.julialang.org/t/primer-feasibility-question-modelling-soft-tissue-behavior-using-ml/76175/9 "2022-04-07T20:50:25Z")

</div>

I believe they have some problem with hardware to. About parallelization I know nothing.

---

<div class="post-metadata">

**Author:** ![goerch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/goerch/32/29122_2.png) [@goerch](https://discourse.julialang.org/u/goerch)\
**Post date:** [April 7, 2022, 8:59pm UTC](https://discourse.julialang.org/t/primer-feasibility-question-modelling-soft-tissue-behavior-using-ml/76175/10 "2022-04-07T20:59:30Z")

</div>

> [@KZiemian](#):
>
> About parallelization I know nothing.

As I recently learned: you are on the safe side of things if your algorithms are [embarrassingly parallel](https://en.wikipedia.org/wiki/Embarrassingly_parallel).

---

<div class="post-metadata">

**Author:** ![KZiemian](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kziemian/32/9020_2.png) [@KZiemian](https://discourse.julialang.org/u/KZiemian)\
**Post date:** [April 7, 2022, 9:03pm UTC](https://discourse.julialang.org/t/primer-feasibility-question-modelling-soft-tissue-behavior-using-ml/76175/11 "2022-04-07T21:03:16Z")

</div>

In our previous conversation @davide445 point to algorithm described in [_GPU Implementation of extended total Lagrangian explicit (gpuXTLED) for Surgical Incision Application_](https://etheses.whiterose.ac.uk/29375/). I don’t remember what is the role of this algorithm in the project, but it is reasonable to mention it here.

Also, even if they have they own visualization GPU software, it may be good to ask [Makie](https://makie.juliaplots.org/stable/) collaboration about few hints. They have a lot of experience with Julia and GPU visualization.

---

<div class="post-metadata">

**Author:** ![davide445](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davide445/32/33681_2.png) [@davide445](https://discourse.julialang.org/u/davide445)\
**Post date:** [April 8, 2022, 5:10am UTC](https://discourse.julialang.org/t/primer-feasibility-question-modelling-soft-tissue-behavior-using-ml/76175/12 "2022-04-08T05:10:23Z")

</div>

> [@goerch](#):
>
> A layman’s view: you can throw quite a bit of hardware on this problem (some cluster or tensor processing units)

Being a VR based interactive application there is no way to use more than a single compute node (a workstation), due the fact having a distributed system will add too much latency.  
So there will be max 4 compute accelerators (GPUs or whatever), possibly all connected into a fast low latency P2P connection (i.e. NVLink).

---

<div class="post-metadata">

**Author:** ![davide445](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davide445/32/33681_2.png) [@davide445](https://discourse.julialang.org/u/davide445)\
**Post date:** [April 8, 2022, 11:49am UTC](https://discourse.julialang.org/t/primer-feasibility-question-modelling-soft-tissue-behavior-using-ml/76175/13 "2022-04-08T11:49:45Z")

</div>

In general I’m seeinfg three possible way forward

- finding and efficient PDE numerical solver specialized for our needs and able to get the performances we need considered all the above requirements → I consider this unlikely
- find an SciML hybrid version of this PDE (so numerical + ML) that can speedup things → a possibility with more uncertains but possibly more accurate
- using an enstablished soft tissue simulation framework (there are a few of them around) to generate a suitable training set and use Julia packages “just” to design, implement, train a suitable DNN architecture to reproduce the expected behavior → probably the lower risk option, with a great doubt if will be possible to achieve the desired dynamics accuracy in realistic conditions

---

<div class="post-metadata">

**Author:** ![KZiemian](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/kziemian/32/9020_2.png) [@KZiemian](https://discourse.julialang.org/u/KZiemian)\
**Post date:** [April 8, 2022, 8:41pm UTC](https://discourse.julialang.org/t/primer-feasibility-question-modelling-soft-tissue-behavior-using-ml/76175/14 "2022-04-08T20:41:33Z")

</div>

We still far from answers, but at least we make questions and challenges easier to understand.