# Tokenising using TextAnalysis 0.8

**URL:** https://discourse.julialang.org/t/tokenising-using-textanalysis-0-8/119573
**Category:** Machine Learning
**Tags:** text-analysis
**Created:** [September 18, 2024, 8:44pm UTC](https://discourse.julialang.org/t/tokenising-using-textanalysis-0-8/119573 "2024-09-18T20:44:26Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![ablaom](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/ablaom/32/4889_2.png) [@ablaom](https://discourse.julialang.org/u/ablaom)
#### Post date: [September 18, 2024, 8:44pm UTC](https://discourse.julialang.org/t/tokenising-using-textanalysis-0-8/119573/1 "2024-09-18T20:44:26Z")

</div>

Thanks to the maintainers of TextAnalysis.

This works in TextAnalysis 0.7.5, but not here in 0.8.1:

```julia
julia> docs = ["Hi my name is Sam.", "How are you today?"]
2-element Vector{String}:
 "Hi my name is Sam."
 "How are you today?"

julia> tokenized_docs = TextAnalysis.tokenize.(docs)
ERROR: MethodError: no method matching tokenize(::String)

Closest candidates are:
  tokenize(::Type{S}, ::T) where {S<:Languages.Language, T<:AbstractString}
   @ TextAnalysis ~/.julia/packages/TextAnalysis/bdhyq/src/deprecations.jl:4
  tokenize(::S, ::T) where {S<:Languages.Language, T<:AbstractString}
   @ TextAnalysis ~/.julia/packages/TextAnalysis/bdhyq/src/tokenizer.jl:19

```

What’s the new way to do this under 0.8.1?

I’m trying to update the examples at MLJText.jl, which have stopped working.

---

<div class="post-metadata">

### Author: ![jlapeyre](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jlapeyre/32/4514_2.png) [@jlapeyre](https://discourse.julialang.org/u/jlapeyre)
#### Post date: [September 19, 2024, 1:21am UTC](https://discourse.julialang.org/t/tokenising-using-textanalysis-0-8/119573/2 "2024-09-19T01:21:42Z")

</div>

Looks like you need to add the `Languages` package.

```julia
julia> using TextAnalysis; using Languages;
julia> TextAnalysis.tokenize(Languages.English(), "the cat")

```