# Creating a corpus with a custom tokenizer

**URL:** https://discourse.julialang.org/t/creating-a-corpus-with-a-custom-tokenizer/82902
**Category:** General Usage
**Tags:** nlp, text-analysis
**Created:** [June 17, 2022, 2:15am UTC](https://discourse.julialang.org/t/creating-a-corpus-with-a-custom-tokenizer/82902 "2022-06-17T02:15:30Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![Jack\_N](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jack_n/32/37265_2.png) [@Jack\_N](https://discourse.julialang.org/u/Jack_N)
#### Post date: [June 17, 2022, 2:15am UTC](https://discourse.julialang.org/t/creating-a-corpus-with-a-custom-tokenizer/82902/1 "2022-06-17T02:15:30Z")

</div>

I am trying to build a corpus with a custom tokenizer in Julia but I cannot figure out how to do it. I tried using TextAnalysis.jl’s Corpus function in order to create the corpus, but it does not allow me to specify a tokenizer. I am wondering how I can make a corpus with a custom tokenizer in Julia.

Thanks,

Jack

---

<div class="post-metadata">

### Author: ![awasserman](https://avatars.discourse-cdn.com/v4/letter/a/9de0a6/32.png) [@awasserman](https://discourse.julialang.org/u/awasserman)
#### Post date: [June 17, 2022, 4:05am UTC](https://discourse.julialang.org/t/creating-a-corpus-with-a-custom-tokenizer/82902/2 "2022-06-17T04:05:09Z")

</div>

If you have a function that produces a vector of strings from an initial string, then you should be able to pass the output token vector to a [`TokenDocument`](https://docs.juliahub.com/TextAnalysis/5Mwet/0.7.3/APIReference/#TextAnalysis.TokenDocument-Tuple%7BAbstractString,TextAnalysis.DocumentMetadata%7D) directly, instead of passing the original string. Then you can just make your Corpus from a list of TokenDocuments.

```julia
token_vector = my_tokenizer_function(a_document_string)
token_doc = TokenDocument(token_vector)
corpus = Corpus([token_doc])

```