# Training a MLJ model on a large dataset
**URL:** https://discourse.julialang.org/t/training-a-mlj-model-on-a-large-dataset/73321
**Category:** Machine Learning
**Created:** [December 19, 2021, 5:05am UTC](https://discourse.julialang.org/t/training-a-mlj-model-on-a-large-dataset/73321 "2021-12-19T05:05:43Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@jlwoolf](https://discourse.julialang.org/u/jlwoolf)
#### Post date: [December 19, 2021, 5:05am UTC](https://discourse.julialang.org/t/training-a-mlj-model-on-a-large-dataset/73321/1 "2021-12-19T05:05:43Z")
Hello, I am somewhat new to Julia and the MLJ library. I am currently using a preexisting model, and would like to train it on a large dataset. However, I am running out of memory when calling fit with all the data. Each data entry is a paragraph, with embeddings for each sentence (the embedding goes as high as 1836 numbers). With each paragraph being on average 6 sentences long, each data entry is 11,000 floats, which is significant when using 100,000 processed paragraphs. I was wondering if anyone knew an effective way to train on increments of data so I can use all the data.
---
### Author:  [@cjdoris](https://discourse.julialang.org/u/cjdoris)
#### Post date: [December 19, 2021, 10:59am UTC](https://discourse.julialang.org/t/training-a-mlj-model-on-a-large-dataset/73321/2 "2021-12-19T10:59:05Z")
What kind of model are you fitting? It’s really up to the implementation if it can be trained in a streaming fashion (such as anything trained with a variant of SGD).
---
### Author:  [@ablaom](https://discourse.julialang.org/u/ablaom)
#### Post date: [December 19, 2021, 7:44pm UTC](https://discourse.julialang.org/t/training-a-mlj-model-on-a-large-dataset/73321/3 "2021-12-19T19:44:26Z")
MLJ currently requires that all data fits into memory. There is some copying that happens by default, which exists to speed-up hyper-parameter optimization. However, this can be switched off by specifying `cache=false`, as in `machine(model, X, y, cache=false)`, and in some other places, such as `TunedModel`.
The “deep learning” package Flux.jl implements a large class of gradient descent models which can be trained incrementally. You can find some basic text analysis examples at [model-zoo](https://github.com/FluxML/model-zoo). Large datasets are typically handled using [DataLoaders.jl](https://github.com/lorenzoh/DataLoaders.jl) (for standard text corpora, see also [CorpusLoaders.jl](https://github.com/JuliaText/CorpusLoaders.jl)). Flux does not provide a lot of tooling beyond building and training models (like you do get with MLJ). For that, you may want to look at [FastAI.jl](https://github.com/FluxML/FastAI.jl).
It may be that after certain reductions (eg, TF-IDF transformation, provided by [TextAnalysis.jl](https://github.com/JuliaText/TextAnalysis.jl)) your data is sufficiently reduced that it fits into memory, which would open up the possibility of using other models, such as those provided by MLJ.