Simple method for converting float array to DateTime (or similar)?

New to Julia
1

I have a time series with time recorded in a vector of floats in the format yyyy,mm,dd,HH,MM,SS.ssssss; i.e.:

times = [2017 09 06 00 00 04.911360]

This comes from the MATLAB datevec() function, which outputs a vector of floats with fractional seconds.

When I work with these data in MATLAB, I often store dates as a float using the datenum() function. In Julia, I want to convert these time vectors to something like DateTime, or split into Date and Time in order to preserve microsecond accuracy. I could simply convert the date vectors into a float like datenum(), but I like the idea of using Date and Time in order to easily (I hope) bin by time windows (e.g. separate a 24-hour time series into 10-minute bins).

Here’s what I have so far:

# sample data
times = [   2017 09 06 00 00 04.911360;
            2017 09 06 04 15 55.193727;
            2017 09 06 22 55 12.256655]

# attempt to initialize Date and Time arrays
sdate = Date(times[1,1]):Day(1):Date(times[3,1])
stime = Time(times[1,4]):Second(1):Time(times[3,4])


for i = 1:size(times, 1)
    years   = times[i,1]
    months  = times[i,2]
    days    = times[i,3]
    hours   = times[i,4]
    minutes = times[i,5]
    seconds = floor(times[i,6])
    millis  = floor((times[i,6] - seconds)*1000)
    micros  = floor(((times[i,6] - seconds)*1000 - millis)*1000)

    sdate[i] = Date(years, months, days)
    stime[i] = Time(hours, minutes, seconds, millis, micros)

end

There are a number of problems here, including:

  • I don’t know how to initialize sdate and stime, so both the initializations and the loop fail.
  • Even if I get the initialization and assignment in the loop working, I have to tediously break out each argument of Date and Time.

It would be great to be able do something like:

sdate = Date.(times[:,1:3])
stime = Time.(times[:,4:6])
sdatetime = DateTime.(times)

…but of course . operates elementwise, so that doesn’t work. Any ideas?

2

What is the source of the data?
Possibly a CSV file?
If we can assume that you start with a String
DateFormat does a lot of work for you.

The microseconds are a bit trickier though…

times=["2017 09 06 00 00 04","2017 09 06 04 15 55"] 
dfmt = DateFormat("yyyy mm dd HH MM SS")
dt = DateTime.(times,dfmt)
sdate = Date.(dt)
stime = Time.(dt)
3

Actually DateFormat does support digits for the second part (up to milliseconds), but your data has more precision than that. Truncating is always an option (albeit not a preferable one).
Time has support for nanoseconds, thus the parsing would need to be done separately.

EDIT: second part of the code has a possible solution. I am still assuming that you start with a String and not a matrix as in your example.

using Dates
dfmt = DateFormat("yyyy mm dd HH MM SS.s")
DateTime("2017 09 06 00 00 04.911",dfmt) #works for me
DateTime("2017 09 06 00 00 04.911360",dfmt) #fails

#How about this:
using Dates
times=["2017 09 06 00 00 04.911360","2017 09 06 04 15 55.193727"] 
dfmt = DateFormat("yyyy mm dd HH MM SS")
times_wo_mus = map(x->x[1:findfirst(".",x)[1]-1],times)
times_mus = map(x->x[findfirst(".",x)[1]+1:end],times)

#Parse DateTime (second precision only)
dt = DateTime.(times_wo_mus,dfmt)
sdate = Date.(dt)
stime = Time.(dt)

#add Microseconds
mus = Microsecond.(parse.(Int,times_mus))
stime .= stime .+ mus
1 Like
4

The package CompoundPeriods.jl is great for this:

using Dates, CompoundPeriods

T = [   2017 09 06 00 00 04.911360;
        2017 09 06 04 15 55.193727;
        2017 09 06 22 55 12.256655]

tcp = Year.(T[:,1]) + Month.(T[:,2]) + Day.(T[:,3]) + Hour.(T[:,4]) +
      + Minute.(T[:,5]) + Nanosecond.(round.(Int,1e9*T[:,6]))

t = canonical.(tcp)

3-element Vector{CompoundPeriod}:
 2017 years, 9 months, 6 days, 4 seconds, 911 milliseconds, 360 microseconds
 2017 years, 9 months, 6 days, 4 hours, 15 minutes, 55 seconds, 193 milliseconds, 727 microseconds
 2017 years, 9 months, 6 days, 22 hours, 55 minutes, 12 seconds, 256 milliseconds, 655 microseconds
1 Like
5

@bernhard, this is very helpful. The data comes from .mat files, which I read with MAT.jl. The files are generated daily, and analyses range from single-day events to several weeks, so speed will eventually be somewhat important. The file is imported as an array of Float64, but I can convert to String with @sprintf(). Here’s a version of your solution that does truncate to millisecond precision:

using Dates

times = [   2017 09 06 00 00 04.911360;
            2017 09 06 04 15 55.193727;
            2017 09 06 22 55 12.256655]

years   = times[:,1]
months  = times[:,2]
days    = times[:,3]
hours   = times[:,4]
minutes = times[:,5]
seconds = times[:,6]

dfmt = DateFormat("yyyy mm dd HH MM SS.s")

timestring = fill("",size(times, 1),1)

for i = 1:size(times, 1)
    
    timestring[i] = @sprintf("%.0f %02.0f %02.0f %02.0f %02.0f %06.3f", years[i], months[i], days[i], hours[i], minutes[i], seconds[i])

end

dt = DateTime.(timestring,dfmt)
6

@rafael.guerra I like this one! Nice that you can easily get millisecond precision with DateTime.(t), or preserve microsecond precision with Date.(t) and Time.(t). So the full solution becomes:

using Dates, CompoundPeriods

T = [   2017 09 06 00 00 04.911360;
        2017 09 06 04 15 55.193727;
        2017 09 06 22 55 12.256655]

tcp = Year.(T[:,1]) + Month.(T[:,2]) + Day.(T[:,3]) + Hour.(T[:,4]) +
      + Minute.(T[:,5]) + Nanosecond.(round.(Int,1e9*T[:,6]))

t = canonical.(tcp)

sdt = DateTime.(t)  # millisecond precision
sdate = Date.(t)    # date only
stime = Time.(t)    # time only with microsecond precision
7

@ToddAnderson, if you like the functionality, consider starring the package repository in github, the author @JeffreySarnoff should appreciate.

2 Likes