# Segment Anything Model for Julia?

**URL:** <https://discourse.julialang.org/t/segment-anything-model-for-julia/97257>\
**Category:** Machine Learning\
**Tags:** image-processing\
**Created:** [April 8, 2023, 8:23pm UTC](https://discourse.julialang.org/t/segment-anything-model-for-julia/97257 "2023-04-08T20:23:00Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![alex-s-gardner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alex-s-gardner/32/30210_2.png) [@alex-s-gardner](https://discourse.julialang.org/u/alex-s-gardner)\
**Post date:** [April 8, 2023, 8:23pm UTC](https://discourse.julialang.org/t/segment-anything-model-for-julia/97257/1 "2023-04-08T20:23:00Z")

</div>

Watching Meta’s Segment Anything Model (SAM) exploding online and wondering anyone is working on a port of the Python version to Julia

> **[GitHub - facebookresearch/segment-anything: The repository provides code for...](https://github.com/facebookresearch/segment-anything)**
>
> The repository provides code for running inference with the SegmentAnything Model (SAM), links for downloading the trained model checkpoints, and example notebooks that show how to use the model. -...

or if a similar model could be trained using the Julia ecosystem as the training data has been made public ([SA-1B Dataset](https://ai.facebook.com/datasets/segment-anything/))

SAM seems like a fundamental tool for many AI/image processing tasks that would be a same not to have access to.

---

<div class="post-metadata">

**Author:** ![sdanisch](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sdanisch/32/1406_2.png) [@sdanisch](https://discourse.julialang.org/u/sdanisch)\
**Post date:** [April 8, 2023, 8:43pm UTC](https://discourse.julialang.org/t/segment-anything-model-for-julia/97257/2 "2023-04-08T20:43:33Z")

</div>

Not your question, but i got it to run pretty quickly with pycall and im currently writing a few helpers and an interactive tool to segment interactively with makie 😉  
Will soon publish something …

---

<div class="post-metadata">

**Author:** ![tim.holy](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/tim.holy/32/52_2.png) [@tim.holy](https://discourse.julialang.org/u/tim.holy)\
**Post date:** [April 10, 2023, 11:39am UTC](https://discourse.julialang.org/t/segment-anything-model-for-julia/97257/3 "2023-04-10T11:39:28Z")

</div>

That does seem useful & cool. It would be great to have a Julia port since that would free us of python package management dependency hell. Since the license is Apache, a direct port seems like it might be the most straightforward way to go. I’m in the middle of other things now, but if anyone wants to give this a stab (manually, or using their favorite AI coding assistant to build the first draft) I’d be happy to review a package prototype.

---

<div class="post-metadata">

**Author:** ![RoyiAvital](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/royiavital/32/571_2.png) [@RoyiAvital](https://discourse.julialang.org/u/RoyiAvital)\
**Post date:** [April 10, 2023, 1:06pm UTC](https://discourse.julialang.org/t/segment-anything-model-for-julia/97257/4 "2023-04-10T13:06:00Z")

</div>

Do you mean the whole training cycle or just recreating the inference process with their weights?

---

<div class="post-metadata">

**Author:** ![jules](https://avatars.discourse-cdn.com/v4/letter/j/41988e/32.png) [@jules](https://discourse.julialang.org/u/jules)\
**Post date:** [April 10, 2023, 4:36pm UTC](https://discourse.julialang.org/t/segment-anything-model-for-julia/97257/5 "2023-04-10T16:36:52Z")

</div>

I think the latter. Is there an up to date way of reading pth files in Julia that contain the trained model info? I assume some manual work would be required to build out the right neural network components to mirror the pytorch setup and then initialize it with the right data from the pth checkpoint.

---

<div class="post-metadata">

**Author:** ![GunnarFarneback](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/gunnarfarneback/32/1827_2.png) [@GunnarFarneback](https://discourse.julialang.org/u/GunnarFarneback)\
**Post date:** [April 10, 2023, 4:57pm UTC](https://discourse.julialang.org/t/segment-anything-model-for-julia/97257/6 "2023-04-10T16:57:09Z")

</div>

I’ve seen [Pickle.jl](https://github.com/chengchingwen/Pickle.jl#pickletorch) recommended. It includes special support for pth files.

---

<div class="post-metadata">

**Author:** ![alex-s-gardner](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/alex-s-gardner/32/30210_2.png) [@alex-s-gardner](https://discourse.julialang.org/u/alex-s-gardner)\
**Post date:** [May 3, 2023, 2:51am UTC](https://discourse.julialang.org/t/segment-anything-model-for-julia/97257/7 "2023-05-03T02:51:58Z")

</div>

For anyone that comes looking there is now a julia wrapper for SAM:

> **[GitHub - rafaqz/SegmentAnything.jl: segment-anything wrapped for Julia](https://github.com/rafaqz/SegmentAnything.jl)**
>
> segment-anything wrapped for Julia. Contribute to rafaqz/SegmentAnything.jl development by creating an account on GitHub.

Courtesy of @sdanisch and @Raf

---

<div class="post-metadata">

**Author:** ![Raf](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/raf/32/3383_2.png) [@Raf](https://discourse.julialang.org/u/Raf)\
**Post date:** [May 3, 2023, 11:06am UTC](https://discourse.julialang.org/t/segment-anything-model-for-julia/97257/8 "2023-05-03T11:06:30Z")

</div>

Theres a bit more work still until its registered, but it works fine already.