Convert a PythonCall df to Julia df, incomplete conversion

General Usage
,
1

Hi,

I have a PythonCall result looking like:

julia> forecast
Python:
                      ds         trend    yhat_lower  ...  multiplicative_terms_lower  multiplicative_terms_upper          yhat
0    2024-04-08 06:19:00  26502.637177  26479.105747  ...                         0.0                         0.0  26483.719090
1    2024-04-08 06:20:00  26502.687072  26479.010556  ...                         0.0                         0.0  26483.783991
2    2024-04-08 06:21:00  26502.736966  26479.212692  ...                         0.0                         0.0  26483.848816
3    2024-04-08 06:22:00  26502.786860  26478.784132  ...                         0.0                         0.0  26483.913559
4    2024-04-08 06:23:00  26502.836754  26479.375907  ...                         0.0                         0.0  26483.978211
...                  ...           ...           ...  ...                         ...                         ...           ...
8636 2024-04-14 06:15:00  26524.734656  26500.998124  ...                         0.0                         0.0  26505.755963
8637 2024-04-14 06:16:00  26524.739858  26501.096483  ...                         0.0                         0.0  26505.776387
8638 2024-04-14 06:17:00  26524.745060  26500.832311  ...                         0.0                         0.0  26505.796769
8639 2024-04-14 06:18:00  26524.750261  26501.324523  ...                         0.0                         0.0  26505.817101
8640 2024-04-15 06:18:00  26532.240510  26508.760525  ...                         0.0                         0.0  26513.307350

After a Julia conversion, I got:

julia> julia_forecast = DataFrame(PyTable(forecast))
8641×16 DataFrame
  Row │ ds                                 trend    yhat_lower  yhat_upper  trend_lower  trend_upper  additive_terms  additive_terms_lower  additive_ ⋯
      │ PyArray…                           Float64  Float64     Float64     Float64      Float64      Float64         Float64               Float64   ⋯
──────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
    1 │ UInt8[0x00, 0x48, 0x42, 0x33, 0x…  26502.6     26479.1     26488.6      26502.6      26502.6        -18.9181              -18.9181            ⋯
    2 │ UInt8[0x00, 0xa0, 0x89, 0x2b, 0x…  26502.7     26479.0     26488.1      26502.7      26502.7        -18.9031              -18.9031
    3 │ UInt8[0x00, 0xf8, 0xd0, 0x23, 0x…  26502.7     26479.2     26488.4      26502.7      26502.7        -18.8881              -18.8881
    4 │ UInt8[0x00, 0x50, 0x18, 0x1c, 0x…  26502.8     26478.8     26488.6      26502.8      26502.8        -18.8733              -18.8733
    5 │ UInt8[0x00, 0xa8, 0x5f, 0x14, 0x…  26502.8     26479.4     26488.9      26502.8      26502.8        -18.8585              -18.8585            ⋯
    6 │ UInt8[0x00, 0x00, 0xa7, 0x0c, 0x…  26502.9     26479.2     26488.6      26502.9      26502.9        -18.8439              -18.8439
    7 │ UInt8[0x00, 0x58, 0xee, 0x04, 0x…  26502.9     26478.9     26488.6      26502.9      26502.9        -18.8293              -18.8293
    8 │ UInt8[0x00, 0xb0, 0x35, 0xfd, 0x…  26503.0     26479.7     26488.8      26503.0      26503.0        -18.8149              -18.8149
  ⋮   │                 ⋮                     ⋮         ⋮           ⋮            ⋮            ⋮             ⋮                  ⋮                      ⋱
 8635 │ UInt8[0x00, 0x38, 0x70, 0xc9, 0x…  26524.7     26501.0     26510.6      26524.7      26524.7        -19.0092              -19.0092            ⋯
 8636 │ UInt8[0x00, 0x90, 0xb7, 0xc1, 0x…  26524.7     26501.2     26510.7      26524.7      26524.7        -18.9939              -18.9939
 8637 │ UInt8[0x00, 0xe8, 0xfe, 0xb9, 0x…  26524.7     26501.0     26510.1      26524.7      26524.7        -18.9787              -18.9787
 8638 │ UInt8[0x00, 0x40, 0x46, 0xb2, 0x…  26524.7     26501.1     26510.8      26524.7      26524.7        -18.9635              -18.9635
 8639 │ UInt8[0x00, 0x98, 0x8d, 0xaa, 0x…  26524.7     26500.8     26510.6      26524.7      26524.7        -18.9483              -18.9483            ⋯
 8640 │ UInt8[0x00, 0xf0, 0xd4, 0xa2, 0x…  26524.8     26501.3     26510.6      26524.8      26524.8        -18.9332              -18.9332

So obviously I would like to get the first column as “2024-04-08 06:19:00” etc.
Looked at pyconvert() without much success.

Sorry for the MWE but I come into this after a lot of processes.

2

Oh yeah this is a known issue. What is forecast.dtypes?

1 Like
3 
julia> forecast.dtypes
Python:
ds                            datetime64[ns]
trend                                float64
yhat_lower                           float64
yhat_upper                           float64
trend_lower                          float64
trend_upper                          float64
additive_terms                       float64
additive_terms_lower                 float64
additive_terms_upper                 float64
daily                                float64
daily_lower                          float64
daily_upper                          float64
multiplicative_terms                 float64
multiplicative_terms_lower           float64
multiplicative_terms_upper           float64
yhat                                 float64
dtype: object
4

Thanks, I assumed it would be that. We have no conversion rules for datetime64 currently. Shouldn’t be hard to add, just hasn’t been done yet.

Unfortunately I don’t think there’s a neat solution. The easiest I think would be to convert forecast["ds"] to a list of strs some how, pyconvert this to a Vector{String}, then parse them into Dates.DateTimes, then assign this vector into the Julia dataframe.

2 Likes
5

Maybe DateTimes64 · Julia Packages can help

2 Likes
6

I was able to handle that this way:

ts_conv=[]
for ts in forecast.ds
    push!(ts_conv, string(ts))
end

ts_julia = DateTime.(ts_conv, "yyyy-mm-dd HH:MM:SS")

julia_forecast.ds = ts_julia

which gives:

julia> julia_forecast
8641×16 DataFrame
  Row │ ds                   trend    yhat_lower  yhat_upper  trend_lower  trend_upper  additive_term ⋯
      │ DateTime             Float64  Float64     Float64     Float64      Float64      Float64       ⋯
──────┼────────────────────────────────────────────────────────────────────────────────────────────────
    1 │ 2024-04-08T06:19:00  26502.6     26479.1     26488.6      26502.6      26502.6        -18.918 ⋯
    2 │ 2024-04-08T06:20:00  26502.7     26479.0     26488.1      26502.7      26502.7        -18.903
    3 │ 2024-04-08T06:21:00  26502.7     26479.2     26488.4      26502.7      26502.7        -18.888
    4 │ 2024-04-08T06:22:00  26502.8     26478.8     26488.6      26502.8      26502.8        -18.873
    5 │ 2024-04-08T06:23:00  26502.8     26479.4     26488.9      26502.8      26502.8        -18.858 ⋯
    6 │ 2024-04-08T06:24:00  26502.9     26479.2     26488.6      26502.9      26502.9        -18.843
    7 │ 2024-04-08T06:25:00  26502.9     26478.9     26488.6      26502.9      26502.9        -18.829
  ⋮   │          ⋮              ⋮         ⋮           ⋮            ⋮            ⋮             ⋮       ⋱
 8636 │ 2024-04-14T06:14:00  26524.7     26501.2     26510.7      26524.7      26524.7        -18.993
 8637 │ 2024-04-14T06:15:00  26524.7     26501.0     26510.1      26524.7      26524.7        -18.978 ⋯
 8638 │ 2024-04-14T06:16:00  26524.7     26501.1     26510.8      26524.7      26524.7        -18.963
 8639 │ 2024-04-14T06:17:00  26524.7     26500.8     26510.6      26524.7      26524.7        -18.948
 8640 │ 2024-04-14T06:18:00  26524.8     26501.3     26510.6      26524.8      26524.8        -18.933
 8641 │ 2024-04-15T06:18:00  26532.2     26508.8     26518.1      26532.2      26532.2        -18.933 ⋯

Thanks everyone.

2 Likes
7

FWIW, a slightly more efficient one-liner (fuses the loops and avoids an intermediate Vector{Any})

julia_forecast.ds = DateTime.(string.(forecast.ds), "yyyy-mm-dd HH:MM:SS")
1 Like