# Guidance on POS Tagging and lemmatization approach

**URL:** <https://discourse.julialang.org/t/guidance-on-pos-tagging-and-lemmatization-approach/114344>\
**Category:** Machine Learning\
**Created:** [May 16, 2024, 7:33am UTC](https://discourse.julialang.org/t/guidance-on-pos-tagging-and-lemmatization-approach/114344 "2024-05-16T07:33:54Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Amval](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/amval/32/6217_2.png) [@Amval](https://discourse.julialang.org/u/Amval)\
**Post date:** [May 16, 2024, 7:33am UTC](https://discourse.julialang.org/t/guidance-on-pos-tagging-and-lemmatization-approach/114344/1 "2024-05-16T07:33:54Z")

</div>

Hello, I’d like to do some basic NLP tasks (POS Tagging, lemmatization) [in languages other than English.](https://discourse.julialang.org/t/postagger-on-other-languages-than-english/53385) . Or, more specifically, German.

Currently, I am using SpaCy through PyCall with the german model. This is fine, however I need to tag lot sof items and this is pretty slow. I have not been able to paralellize it due to the GIL, however I think there might be ways to do this. So I guess I have the following options:

- Keep using Python, try to use `Distributed` and `pmap` as shown here: [Run multiple python instances with pycall in different threads - #2 by cjdoris](https://discourse.julialang.org/t/run-multiple-python-instances-with-pycall-in-different-threads/78399/2) This might require some heavy restructuring in my code: I have very long chains of functions with the python calls intertwinned. I don’t know if I can just slap `@everywhere` in front of every single function.

- Try to use Transformers.jl with some state of the art model. However, I am kind of clueless as to how do to that. There is an example to encode a sentence, but not sure how to get from there to somewhere practical.

- Since I don’t really need state-of-the-art but just something workable, I could port a simple Python package that achieves these tasks. This is tedious but at least a known quantity.

Does anyone have any insights or recommendations? Some package I overlooked or a simple solution?

Thanks in advance and I hope the post is not too unfocused.