# TextAnalysis and individual TF-IDF
**URL:** https://discourse.julialang.org/t/textanalysis-and-individual-tf-idf/31798
**Category:** Specific Domains
**Created:** [December 3, 2019, 1:53pm UTC](https://discourse.julialang.org/t/textanalysis-and-individual-tf-idf/31798 "2019-12-03T13:53:54Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@rssdev10](https://discourse.julialang.org/u/rssdev10)
#### Post date: [December 3, 2019, 1:53pm UTC](https://discourse.julialang.org/t/textanalysis-and-individual-tf-idf/31798/1 "2019-12-03T13:53:54Z")
I have a case with fixed corpus of documents and one document which is a query. The task is to calculate similarity between these fixed documents and a document sent by a user. For now, we are using TextAnalysis with Corpus/DocumentMatrix and have a code something like this:
```julia
function count_tfidf(crps::Corpus, request::TokenDocument)
# take a copy of fixed corpus
local l_crps = Corpus(documents(crps) |> copy)
# add new doc into a copy for processing
push!(l_crps, request)
# merge terms to avoid recalculation for all the docs
local terms = keys(merge(lexicon(crps), ngrams(request))) |> collect
# calc full tf-idf
local doc_tf_idf = DocumentTermMatrix(l_crps, terms) |> tf_idf
# take a row with our doc
local val = doc_tf_idf[end, :]
#calculate similarity (with Distances.cosine_dist as a sample only)
return map eachrow(doc_tf_idf[1:end-1, :]) do row
cosine_dist(row, val)
end
end
```
The issue here, even when we are avoiding full lexicon update for new documents, as we are using the lexicon from a fixed corpus, the `DocumentTermMatrix` and `tf_idf` methods perform calculation for all the documents.
It would be good to have some individual tf\_idf method which can use and existing matrix and calculate weights for a requested set of terms or tokens only. And we don’t need to store there query documents or use them for recalculating lexicon/dtm of our static corpus.
Technically I see way for calculating a `DocumentTermMatrix` for that the only document if I’m creating a separate `Corpus` - [https://github.com/JuliaText/TextAnalysis.jl/blob/master/src/dtm.jl#L57](https://github.com/JuliaText/TextAnalysis.jl/blob/master/src/dtm.jl#L57). Also I see a [Features · TextAnalysis](https://juliatext.github.io/TextAnalysis.jl/dev/features/#Creating-Individual-Rows-of-a-Document-Term-Matrix-1) with following code for getting terms row for the only document from the corpus and for the given lexicon:
```julia
julia> dtv(crps[1], lexicon(crps))
1×6 Array{Int64,2}:
1 2 0 1 1 1
```
But I don’t see any way to calculate `tf_idf` or` tf_bm25` for new line only without recalculating of all weights for all documents. I cannot do it in real time but I need to create a response any case.
So, is any way to calculate TF/TF-IDF/BM25 other than reuse directly the code:
```julia
function tf_idf!(dtm::SparseMatrixCSC{T}, tfidf::SparseMatrixCSC{F}) where {T <: Real, F <: AbstractFloat}
rows = rowvals(dtm)
dtmvals = nonzeros(dtm)
tfidfvals = nonzeros(tfidf)
@assert size(dtmvals) == size(tfidfvals)
n, p = size(dtm)
# TF tells us what proportion of a document is defined by a term
words_in_documents = F.(sum(dtm, dims=2))
oneval = one(F)
# IDF tells us how rare a term is in the corpus
documents_containing_term = vec(sum(dtm .> 0, dims=1))
idf = log.(n ./ documents_containing_term)
for i = 1:p
for j in nzrange(dtm, i)
row = rows[j]
tfidfvals[j] = dtmvals[j] / max(words_in_documents[row], oneval) * idf[i]
end
end
return tfidf
end
```
from [https://github.com/JuliaText/TextAnalysis.jl/blob/master/src/tf\_idf.jl#L134](https://github.com/JuliaText/TextAnalysis.jl/blob/master/src/tf_idf.jl#L134)
---
### Author:  [@zgornel](https://discourse.julialang.org/u/zgornel)
#### Post date: [December 3, 2019, 2:33pm UTC](https://discourse.julialang.org/t/textanalysis-and-individual-tf-idf/31798/2 "2019-12-03T14:33:36Z")
Have a look at
[https://zgornel.github.io/StringAnalysis.jl/dev/examples/#Dimensionality-reduction-1](https://zgornel.github.io/StringAnalysis.jl/dev/examples/#Dimensionality-reduction-1)
and
[https://zgornel.github.io/StringAnalysis.jl/dev/examples/#Semantic-Analysis-1](https://zgornel.github.io/StringAnalysis.jl/dev/examples/#Semantic-Analysis-1)
By using the “no projection hack” you can use a random projection model to embed a query, the corpus and calculate the similarity between them.
---
### Author:  [@rssdev10](https://discourse.julialang.org/u/rssdev10)
#### Post date: [December 3, 2019, 4:30pm UTC](https://discourse.julialang.org/t/textanalysis-and-individual-tf-idf/31798/3 "2019-12-03T16:30:05Z")
Thank you. Will check it. BTW, what is the reason of implementing of the functions like TextAnalysis but as a separate project?
---
### Author:  [@zgornel](https://discourse.julialang.org/u/zgornel)
#### Post date: [December 3, 2019, 4:40pm UTC](https://discourse.julialang.org/t/textanalysis-and-individual-tf-idf/31798/4 "2019-12-03T16:40:15Z")
TextAnalysis is a big package with some issues (personal opinion ofcourse) in code quality, functionality, features and development/PR merge speed.
Basically, I needed fast deployment of a stable version, better typing, information retrieval-oriented methods, less large dependencies i.e. `Flux.jl` and cleaner API so hard forked it.