# Announcing TSFrames.jl (formerly TSx) v0.1.0: A timeseries data manipulation package based on DataFrames

**URL:** <https://discourse.julialang.org/t/announcing-tsframes-jl-formerly-tsx-v0-1-0-a-timeseries-data-manipulation-package-based-on-dataframes/91264>\
**Category:** Package Announcements\
**Tags:** package, dataframes, time-series\
**Created:** [December 5, 2022, 2:23pm UTC](https://discourse.julialang.org/t/announcing-tsframes-jl-formerly-tsx-v0-1-0-a-timeseries-data-manipulation-package-based-on-dataframes/91264 "2022-12-05T14:23:12Z")\
**Posts on this page:** 14\
**Page:** 2

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [February 14, 2023, 9:19pm UTC](https://discourse.julialang.org/t/announcing-tsframes-jl-formerly-tsx-v0-1-0-a-timeseries-data-manipulation-package-based-on-dataframes/91264/21 "2023-02-14T21:19:44Z")

</div>

Yes, `dropmissing` should probably be generic in DataAPI. TSFrames should not implement `skipmissing` in the way it currently is.

---

<div class="post-metadata">

**Author:** ![bkamins](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/bkamins/32/208538_2.png) [@bkamins](https://discourse.julialang.org/u/bkamins)\
**Post date:** [February 14, 2023, 9:34pm UTC](https://discourse.julialang.org/t/announcing-tsframes-jl-formerly-tsx-v0-1-0-a-timeseries-data-manipulation-package-based-on-dataframes/91264/22 "2023-02-14T21:34:08Z")

</div>

1. TSFrames.jl has DataFrames.jl as a depencency, so it can add a method to `dropmissing` without a problem. Preferably with the same signature `dropmissing(table::T, cols=:; view::Bool=false, disallowmissing::Bool=!view)` but if e.g. some kwargs are hard to support then this does not have to be strictly followed.
2. `skipmissing` has a different API as it RETAINS indices from the parent:

```julia
julia> x = [1, 2, missing, 4, 5, missing]
6-element Vector{Union{Missing, Int64}}:
 1
 2
  missing
 4
 5
  missing

julia> x |> skipmissing |> eachindex |> collect
4-element Vector{Int64}:
 1
 2
 4
 5

```

@chiraganand - in general my recommendation would be that if you add new features to TSFrames.jl (especially features that are not unique to the package) you could consider making a small announcement at #data channel on Slack so that people can respond to it before you make a release. This is the procedure we try to follow in DataFrames.jl and it works reasonably well (people tend to watch #data channel on Slack instead of watching all the repositories on GitHub that might interest them as on GitHub there is too much going on in parallel and it is hard to track).

---

<div class="post-metadata">

**Author:** ![chiraganand](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chiraganand/32/32787_2.png) [@chiraganand](https://discourse.julialang.org/u/chiraganand)\
**Post date:** [February 15, 2023, 6:59am UTC](https://discourse.julialang.org/t/announcing-tsframes-jl-formerly-tsx-v0-1-0-a-timeseries-data-manipulation-package-based-on-dataframes/91264/23 "2023-02-15T06:59:32Z")

</div>

Actually, TSFrames does not have a specific implementation for `skipmissing`.

---

<div class="post-metadata">

**Author:** ![chiraganand](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chiraganand/32/32787_2.png) [@chiraganand](https://discourse.julialang.org/u/chiraganand)\
**Post date:** [February 15, 2023, 7:01am UTC](https://discourse.julialang.org/t/announcing-tsframes-jl-formerly-tsx-v0-1-0-a-timeseries-data-manipulation-package-based-on-dataframes/91264/24 "2023-02-15T07:01:11Z")

</div>

> [@bkamins](#):
>
> TSFrames.jl has DataFrames.jl as a depencency, so it can add a method to `dropmissing` without a problem. Preferably with the same signature `dropmissing(table::T, cols=:; view::Bool=false, disallowmissing::Bool=!view)` but if e.g. some kwargs are hard to support then this does not have to be strictly followed.

Yes, this should be easily possible to do in TSFrames.

> [@bkamins](#):
>
> @chiraganand - in general my recommendation would be that if you add new features to TSFrames.jl (especially features that are not unique to the package) you could consider making a small announcement at #data channel on Slack so that people can respond to it before you make a release. This is the procedure we try to follow in DataFrames.jl and it works reasonably well (people tend to watch #data channel on Slack instead of watching all the repositories on GitHub that might interest them as on GitHub there is too much going on in parallel and it is hard to track).

Okay, sure.

---

<div class="post-metadata">

**Author:** ![chiraganand](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chiraganand/32/32787_2.png) [@chiraganand](https://discourse.julialang.org/u/chiraganand)\
**Post date:** [February 18, 2023, 5:43am UTC](https://discourse.julialang.org/t/announcing-tsframes-jl-formerly-tsx-v0-1-0-a-timeseries-data-manipulation-package-based-on-dataframes/91264/25 "2023-02-18T05:43:18Z")

</div>

@mdogan Have you tried using `dropmissing` with `ts.coredata`? The `TSFrame.coredata` property stores the underlying DataFrame and this is how most of the TSFrames methods implement wrappers over DataFrames functionality. Till the time TSFrames gets it’s own `dropmissing()` this could be a good way to achieve what you are trying to do.

---

<div class="post-metadata">

**Author:** ![mdogan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mdogan/32/47244_2.png) [@mdogan](https://discourse.julialang.org/u/mdogan)\
**Post date:** [February 18, 2023, 9:04am UTC](https://discourse.julialang.org/t/announcing-tsframes-jl-formerly-tsx-v0-1-0-a-timeseries-data-manipulation-package-based-on-dataframes/91264/26 "2023-02-18T09:04:09Z")

</div>

@chiraganand, thanks very much for the follow-up. Yes, that way, it works, but as I highlighted at the beginning, I will be using TSFrames within my package’s functions as a main data object, and `dropmissing` does not suit the flow of the behavior of my package.

```julia
using Dates
using TSFrames
using DataFrames

dates = Date(2021, 12, 31):Month(1):Date(2022, 07, 31);
TSLA = [352.26,312.24,missing,359.2,290.25,252.75,224.47,297.15];
NFLX = [602.44,427.14,394.52,missing,190.36,197.44,174.87,224.9];
MSFT = [336.32,310.98,298.79,308.31,277.52,271.87,missing,280.74];

# by the way, you should add the below line somewhere at the top of the TSFrames documentation :) It's easy to construct a TSFrame object, but it took quite some time to figure out this functionality.
prices_ts = TSFrame([TSLA NFLX MSFT], dates, colnames=[:TSLA, :NFLX, :MSFT])

julia> returns = pctchange(prices_ts)
8×3 TSFrame with Date Index
 Index TSLA NFLX MSFT
 Date Float64? Float64? Float64?        
──────────────────────────────────────────────────────────────
 2021-12-31 missing missing missing
 2022-01-31 -0.113609 -0.290983 -0.0753449
 2022-02-28 missing -0.0763684 -0.0391987
 2022-03-31 missing missing 0.0318618
 2022-04-30 -0.191954 missing -0.099867
 2022-05-31 -0.129199 0.0371927 -0.0203589
 2022-06-30 -0.111889 -0.114313 missing
 2022-07-31 0.323785 0.286098 missing

julia> dropmissing(returns.coredata)
2×4 DataFrame
 Row │ Index TSLA NFLX MSFT       
     │ Date Float64 Float64 Float64    
─────┼───────────────────────────────────────────────
   1 │ 2022-01-31 -0.113609 -0.290983 -0.0753449
   2 │ 2022-05-31 -0.129199 0.0371927 -0.0203589

# I force my functions to use the TSFrame object, therefore:
TSFrame(dropmissing(returns.coredata))
2×3 TSFrame with Date Index
 Index TSLA NFLX MSFT       
 Date Float64 Float64 Float64    
───────────────────────────────────────────────
 2022-01-31 -0.113609 -0.290983 -0.0753449
 2022-05-31 -0.129199 0.0371927 -0.0203589

```

At this stage, I’d like to keep the dependencies at a minimum. In the documentation, I will specify that users can drop missing values using `dropmissing` in the way described above.

`skipmissing` is normally problematic and does not work with many functions, but If I need to skip missing values, I’ll use the syntax below within my library’s functions. This is an alternative way for me to make column-wise operations and skip _ **Tables.jl** _ as a dependency.

By the way, you gave me this idea in one of your replies; I just removed Table.jl and used _ **Matrix** _ instead.

```julia
m = []
for col in collect.(skipmissing(eachcol(Matrix(ts))))
    println(col)
    push!(m, mean(skipmissing(col)))
end
m

```

Indeed, it is better than dropping the lines with missing values in terms of the analysis conducted. However, it would be great to have _ **TSFrames.jl** _ specific `dropmissing` equivalent so that users can easily drop missing values if needed before passing them as input to the functions.

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [February 18, 2023, 12:25pm UTC](https://discourse.julialang.org/t/announcing-tsframes-jl-formerly-tsx-v0-1-0-a-timeseries-data-manipulation-package-based-on-dataframes/91264/27 "2023-02-18T12:25:49Z")

</div>

> [@mdogan](#):
>
> ` collect.(skipmissing(eachcol(Matrix(ts))))`

This is still not doing what you think it is. You are missing a broadcast. Do `skipmissing.(eachcol(...)` instead.

Also, you should be able to do `eachcol` on `ts` directly. Did you try that?

---

<div class="post-metadata">

**Author:** ![mdogan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mdogan/32/47244_2.png) [@mdogan](https://discourse.julialang.org/u/mdogan)\
**Post date:** [February 18, 2023, 12:55pm UTC](https://discourse.julialang.org/t/announcing-tsframes-jl-formerly-tsx-v0-1-0-a-timeseries-data-manipulation-package-based-on-dataframes/91264/28 "2023-02-18T12:55:44Z")

</div>

> [@pdeffebach](#):
>
> Also, you should be able to do `eachcol` on `ts` directly. Did you try that?

This does not work.

```julia
julia> eachcol(prices_ts)
ERROR: MethodError: no method matching eachcol(::TSFrame)
Closest candidates are:
  eachcol(::AbstractVecOrMat) at abstractarraymath.jl:583
  eachcol(::DataFrames.AbstractDataFrame) at C:\Users\ploot\.julia\packages\DataFrames\dgZn3\src\abstractdataframe\iteration.jl:175
Stacktrace:
 [1] top-level scope
   @ REPL[1]:1

```

> [@pdeffebach](#):
>
> This is still not doing what you think it is. You are missing a broadcast. Do `skipmissing.(eachcol(...)` instead.

It does not work if I don’t use the _ **collect** _. It does not skip the `missing`. Unfortunately, `missing` is weakest point of Julia so far, I observed.

```julia
julia> mean.(skipmissing(eachcol(Matrix(prices_ts))))
3-element Vector{Missing}:
 missing
 missing
 missing

julia> mean.(collect.(skipmissing(eachcol(Matrix(prices_ts)))))
3-element Vector{Missing}:
 missing
 missing
 missing

```

This is the only way it worked:

```julia
julia> m = []
Any[]

julia> for col in collect.(skipmissing(eachcol(Matrix(prices_ts))))
           println(col)
               push!(m, mean(skipmissing(col)))
       end
Union{Missing, Float64}[352.26, 312.24, missing, 359.2, 290.25, 252.75, 224.47, 297.15]
Union{Missing, Float64}[602.44, 427.14, 394.52, missing, 190.36, 197.44, 174.87, 224.9]
Union{Missing, Float64}[336.32, 310.98, 298.79, 308.31, 277.52, 271.87, missing, 280.74]

julia> m
3-element Vector{Any}:
 298.3314285714286
 315.95285714285717
 297.78999999999996

```

---

<div class="post-metadata">

**Author:** ![pdeffebach](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/pdeffebach/32/10320_2.png) [@pdeffebach](https://discourse.julialang.org/u/pdeffebach)\
**Post date:** [February 18, 2023, 1:03pm UTC](https://discourse.julialang.org/t/announcing-tsframes-jl-formerly-tsx-v0-1-0-a-timeseries-data-manipulation-package-based-on-dataframes/91264/29 "2023-02-18T13:03:17Z")

</div>

You misinterpreted what I wrote. You are doing `skipmissing(eachcol(...))`. You should do `skipmissing.(eachcol(...))`. Notice the `.`

```julia
julia> X = [missing 2; 3 missing];

julia> for col in collect.(skipmissing.(eachcol(X)))
           println(col)
           push!(m, mean(col))
       end
[3]
[2]

```

But also, `mean` “just works” with `skipmissing`.

```julia
julia> mean.(skipmissing.(eachcol(X)))
2-element Vector{Float64}:
 3.0
 2.0

```

I understand other functions do not, though, and we are actively working on making it easier to accomplish these tasks.

---

<div class="post-metadata">

**Author:** ![mdogan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mdogan/32/47244_2.png) [@mdogan](https://discourse.julialang.org/u/mdogan)\
**Post date:** [February 18, 2023, 3:21pm UTC](https://discourse.julialang.org/t/announcing-tsframes-jl-formerly-tsx-v0-1-0-a-timeseries-data-manipulation-package-based-on-dataframes/91264/30 "2023-02-18T15:21:51Z")

</div>

@pdeffebach, Thank you so much for your answer.

> [@pdeffebach](#):
>
> . Notice the `.`

Sorry, I missed that. I tried, and the final output is the same, because, I think _**mean()**_ function was _skipping_ the `missing` not this one `skipmissing(eachcol(Matrix(prices_ts)))`.

```julia
julia> m = []
Any[]

julia> for col in collect.(skipmissing.(eachcol(Matrix(prices_ts))))
           println(col)
               push!(m, mean(col))
       end
[352.26, 312.24, 359.2, 290.25, 252.75, 224.47, 297.15]
[602.44, 427.14, 394.52, 190.36, 197.44, 174.87, 224.9]
[336.32, 310.98, 298.79, 308.31, 277.52, 271.87, 280.74]

julia> m
3-element Vector{Any}:
 298.3314285714286
 315.95285714285717
 297.78999999999996

```

```julia
julia> m = []
Any[]

julia> for col in collect.(skipmissing(eachcol(Matrix(prices_ts))))
           println(col)
           push!(m, mean(skipmissing(col)))
           end
Union{Missing, Float64}[352.26, 312.24, missing, 359.2, 290.25, 252.75, 224.47, 297.15]
Union{Missing, Float64}[602.44, 427.14, 394.52, missing, 190.36, 197.44, 174.87, 224.9]
Union{Missing, Float64}[336.32, 310.98, 298.79, 308.31, 277.52, 271.87, missing, 280.74]

julia> m
3-element Vector{Any}:
 298.3314285714286
 315.95285714285717
 297.78999999999996

```

> [@pdeffebach](#):
>
> I understand other functions do not, though, and we are actively working on making it easier to accomplish these tasks.

A big thanks for that. The problems are not because of the TSFrames package but because of how Julia works. Nevertheless, I’m looking forward to the next releases of TSFrames.

---

<div class="post-metadata">

**Author:** ![mdogan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mdogan/32/47244_2.png) [@mdogan](https://discourse.julialang.org/u/mdogan)\
**Post date:** [February 20, 2023, 2:26pm UTC](https://discourse.julialang.org/t/announcing-tsframes-jl-formerly-tsx-v0-1-0-a-timeseries-data-manipulation-package-based-on-dataframes/91264/31 "2023-02-20T14:26:43Z")

</div>

@chiraganand @pdeffebach, thanks very much for your answers so far. I wonder whether there is a TSFrame (or some other) function to obtain the **data frequency** of a **TSFrame** object.

I want to get something like **daily** if data is daily, **monthly** if data is monthly, and so on.

Thank you in advance

---

<div class="post-metadata">

**Author:** ![mdogan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/mdogan/32/47244_2.png) [@mdogan](https://discourse.julialang.org/u/mdogan)\
**Post date:** [February 27, 2023, 7:02pm UTC](https://discourse.julialang.org/t/announcing-tsframes-jl-formerly-tsx-v0-1-0-a-timeseries-data-manipulation-package-based-on-dataframes/91264/32 "2023-02-27T19:02:11Z")

</div>

> [@chiraganand](#):
>
> Can you send the link to your library? We ([XKDR Forum](https://github.com/xKDR/)) have also started work on some other finance-related packages which depend on TSFrames.jl. It would be good to know what each of us are doing to avoid overlap and collaborate, if possible.
> 
> Yes, we do plan to maintain the package going forward.

@chiraganand, I know that it’s kinda late, but I wanted to share something meaningful before sharing it with you and others. I have already released my package, **PortfolioAnalytics.jl** , which aims to be a _ **tool for quantitative portfolio analytics** _—indeed made a second release.

All functions accept only the **TSFrame** object as an input - I don’t want to rely on unmaintained packages. I’d be very happy to collaborate, and I sincerely thank you and others in xKDR for the **TSFrames** package and all your answers so far.

The package is under heavy development, but even now possible to derive _returns_, _portfolio returns_, _value at risk_, _expected shortfall_, minimum-variance or maximum-sharpe _portfolio optimization_, and a few _others_. Hopefully, the package’s functionality will improve with further releases, either minor or major. I plan to add new functionalities every month until it becomes a mature library.

[Announcing PortfolioAnalytics.jl: Tool for Quantitative Portfolio Analytics - Package Announcements / Package announcements - Julia Programming Language (julialang.org)](https://discourse.julialang.org/t/announcing-portfolioanalytics-jl-tool-for-quantitative-portfolio-analytics/95236)

[doganmehmet/PortfolioAnalytics.jl (github.com)](https://github.com/doganmehmet/PortfolioAnalytics.jl)

I am open to any feedback, feature request, and collaboration.

Thanks

---

<div class="post-metadata">

**Author:** ![chiraganand](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/chiraganand/32/32787_2.png) [@chiraganand](https://discourse.julialang.org/u/chiraganand)\
**Post date:** [February 28, 2023, 7:40am UTC](https://discourse.julialang.org/t/announcing-tsframes-jl-formerly-tsx-v0-1-0-a-timeseries-data-manipulation-package-based-on-dataframes/91264/33 "2023-02-28T07:40:36Z")

</div>

> [@mdogan](#):
>
> @chiraganand @pdeffebach, thanks very much for your answers so far. I wonder whether there is a TSFrame (or some other) function to obtain the **data frequency** of a **TSFrame** object.
> 
> I want to get something like **daily** if data is daily, **monthly** if data is monthly, and so on.

No, there is no such functionality as such. Though, there is an `isRegular()` function which tells whether observations in a series are equally spaced or not for a given unit.

---

<div class="post-metadata">

**Author:** ![Realosunboy6](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/realosunboy6/32/215064_2.png) [@Realosunboy6](https://discourse.julialang.org/u/Realosunboy6)\
**Post date:** [February 28, 2025, 5:27am UTC](https://discourse.julialang.org/t/announcing-tsframes-jl-formerly-tsx-v0-1-0-a-timeseries-data-manipulation-package-based-on-dataframes/91264/34 "2025-02-28T05:27:14Z")

</div>

![image](https://global.discourse-cdn.com/julialang/original/3X/4/3/43ddf6e7e4474b1dea20b4f3fe4cf8e0bf1c7f4a.png)  
 ![image](https://global.discourse-cdn.com/julialang/original/3X/7/4/744cc2f60d8e44ac40638c304ffa39e7481d83c8.png)  
i noticed change in value for return and port after joining(). What might be causing the issue?

[Previous page](https://discourse.julialang.org/t/announcing-tsframes-jl-formerly-tsx-v0-1-0-a-timeseries-data-manipulation-package-based-on-dataframes/91264.md?page=1)