# OnlineStats: StatLearn vs LinReg vs other options for linear regression on large datasets
**URL:** https://discourse.julialang.org/t/onlinestats-statlearn-vs-linreg-vs-other-options-for-linear-regression-on-large-datasets/17830
**Category:** General Usage
**Tags:** package
**Created:** [November 21, 2018, 8:20pm UTC](https://discourse.julialang.org/t/onlinestats-statlearn-vs-linreg-vs-other-options-for-linear-regression-on-large-datasets/17830 "2018-11-21T20:20:19Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@Juan](https://discourse.julialang.org/u/Juan)
#### Post date: [November 21, 2018, 8:20pm UTC](https://discourse.julialang.org/t/onlinestats-statlearn-vs-linreg-vs-other-options-for-linear-regression-on-large-datasets/17830/1 "2018-11-21T20:20:19Z")
Hello.
What’s the difference between using StatLearn and LinReg for linear regression.
```julia
fit!(StatLearn(5, MSPI()), (x, y))
```
```julia
fit!(LinReg(), (x,y))
```
both are supposed to be able to “Fit a model that is linear in the parameters.”
What method is internally used when you use LinRegBuilder?
Can I choose an algorithm different from SGD?
I’m looking for the fastest method able to fit regression models using large datasets that don’t fit on memory. What is my best option? and for logistic regression? Should I use Flux.jl instead?
OnlineStats docs have this example:
> x = randn(100\_000, 10)
> y = x \* linspace(-1, 1, 10) + randn(100\_000)
> o = StatLearn(10, .5 \* L2DistLoss(), L1Penalty(), fill(.1, 10), SGD())
> s = Series(o)
> fit!(s, x, y)
But my real data is on a csv file and it’s very big, sometimes larger than memory. How can I tell fit! to use the data directly from the disk without first loading everything on memory?
Imagine I create this data and I save it on a file called mydata.csv:
> N=30000
> x1 = repeat(1:N, outer=N)
> x2 = repeat(1:N, inner=N)
> x3 = sqrt.(repeat(1:N^2))
> x1x2 = x1 .\* x2
> gg = repeat(1:5,inner=div(N^2,5))
> y = 1 .- 2 _x1 + 3_ x2 + 0.5\*x1x2 + rand(N^2) + x3.\*rand(N^2)
> data = DataFrame(x1=x1, x2=x2, x3=x3, x1x2=x1x2, y=y,gg=gg)
> categorical!(data, :gg)
And I want to fit it as if was using the command
> fit(LinearModel, @formula(y ~ x1+x2+x3+x1x2+gg), data)
but I want to do it loading the data directly from mydata.csv,
how would you do it? It doesn’t fit on memory. I don’t know how to create a stream or however you call it.
---
### Author:  [@joshday](https://discourse.julialang.org/u/joshday)
#### Post date: [November 21, 2018, 8:35pm UTC](https://discourse.julialang.org/t/onlinestats-statlearn-vs-linreg-vs-other-options-for-linear-regression-on-large-datasets/17830/2 "2018-11-21T20:35:52Z")
- `LinReg` is exact regression.
- Everything `StatLearn` does is approximate.
- `LinRegBuilder` is a more general version of `LinReg`.
- `StatLearn` has many algorithm options: `SGD`, `ADAGRAD`, `ADAM`, `ADAMAX`, `RMSPROP`, `MSPI`, …
- `StatLearn` will be faster, `LinReg` will be more correct. For linear regression I would use `LinReg`. For logistic regression your only option (in OnlineStats) is `StatLearn(p, LogitMarginLoss())`.
- You can repeatedly `fit!` an OnlineStats object on new batches of data, but OnlineStats is agnostic on how you get your data into Julia. It doesn’t have helpers for streaming a CSV file.
- If you want to iterate through the rows of a CSV file one by one without loading it into memory, see [`CSV.File`](http://juliadata.github.io/CSV.jl/latest/).
---
### Author:  [@Juan](https://discourse.julialang.org/u/Juan)
#### Post date: [November 21, 2018, 8:41pm UTC](https://discourse.julialang.org/t/onlinestats-statlearn-vs-linreg-vs-other-options-for-linear-regression-on-large-datasets/17830/3 "2018-11-21T20:41:02Z")
but this would only fit the regression to each batch of data, I want to fit it using the whole dataset.
And what of those algorithm (SGD, ADAGRAD, ADAM, ADAMAX, RMSPROP, MSPI…) would you suggest to quickly fit a linear regression model on a large dataset?
Is there any tutorial with more examples on how to use OnlineStats for these kind of tasks and where I can read more details of all options?
---
### Author:  [@joshday](https://discourse.julialang.org/u/joshday)
#### Post date: [November 21, 2018, 8:44pm UTC](https://discourse.julialang.org/t/onlinestats-statlearn-vs-linreg-vs-other-options-for-linear-regression-on-large-datasets/17830/4 "2018-11-21T20:44:56Z")
I think you’re misunderstanding how OnlineStats works. Suppose you have a giant dataset `(x, y)` that is split into batches `(x1,y1), (x2,y2), (x3,y3)`.
The following two things will give you the same exact answer:
```julia
fit!(LinReg(), (x,y))
```
```julia
o = LinReg()
fit!(o, (x1, y1))
fit!(o, (x2, y2))
fit!(o, (x3, y3))
```
* * *
You’ll need to read about those algorithms yourself to determine what’s best for you. They’re all comparable in terms of speed.
---
### Author:  [@Juan](https://discourse.julialang.org/u/Juan)
#### Post date: [November 21, 2018, 8:48pm UTC](https://discourse.julialang.org/t/onlinestats-statlearn-vs-linreg-vs-other-options-for-linear-regression-on-large-datasets/17830/5 "2018-11-21T20:48:32Z")
OK, I’ll see how to do it.
I thought I could do it easily from OnlineStats or JuliaDB.
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [November 21, 2018, 9:29pm UTC](https://discourse.julialang.org/t/onlinestats-statlearn-vs-linreg-vs-other-options-for-linear-regression-on-large-datasets/17830/6 "2018-11-21T21:29:46Z")
> [@Juan](#):
>
> I thought I could do it easily from OnlineStats or JuliaDB.
What are you missing? CSV.File lets you load a CSV file incrementally and OnlineStats lets you fit the linear regression incrementally. Is the thing that’s missing the outer loop that calls CSV.File and passes blocks of it to OnlineStats?
---
### Author:  [@piever](https://discourse.julialang.org/u/piever)
#### Post date: [November 21, 2018, 10:00pm UTC](https://discourse.julialang.org/t/onlinestats-statlearn-vs-linreg-vs-other-options-for-linear-regression-on-large-datasets/17830/7 "2018-11-21T22:00:30Z")
> [@StefanKarpinski](#):
>
> What are you missing? CSV.File lets you load a CSV file incrementally and OnlineStats lets you fit the linear regression incrementally. Is the thing that’s missing the outer loop that calls CSV.File and passes blocks of it to OnlineStats?
Actually none of that is necessary 🙂 (at least as far as I can tell, I’ve never used this feature in real applications but from quick REPL experimentation it seems everything is in place). One can use `loadtable` to load many CSVs together, say:
```julia
t = loadtable(myfiles)
```
and then one can “reduce” a table by an online-stat, so if we want `LinReg()` we would do:
```julia
reduce(LinReg(), t, select = (:x, :y))
```
with the extra advantage that, as our table is loaded in a distributed way, the fitting can happen in parallel.
---
### Author:  [@Juan](https://discourse.julialang.org/u/Juan)
#### Post date: [November 21, 2018, 11:02pm UTC](https://discourse.julialang.org/t/onlinestats-statlearn-vs-linreg-vs-other-options-for-linear-regression-on-large-datasets/17830/8 "2018-11-21T23:02:02Z")
Is it the CSV.jl way?
I guess if I have just one file I don’t need to use the reduce.
How is it done with JuliaDB instead?