Higher resolution DateTime/TimeStamp

Internals & Design
1

Hi,
as I understand, in Julia we have Date (with 1d resolution, and an “epoch” of Rata Dei 0001-01-01 with the good old proleptic Gregorian calendar), DateTime (with 1 ms resolution, and the same “epoch”, able to cover +/-200 m years), as well as Time (with 1 ns resolution).

I’m looking for a DateTime or TimeStamp with a bit higher resolution.
It would appear that there are two options (while staying within an Int64):

  1. something with microsecond resolution, centred at the same epoch, with +/- 200k years
  2. something with nanosecond resolution, covering only +/- 200 years, thus should probably be centred around some other epoch, e.g. the 1970 UNIX one.

(It appears that Python has a datetime with 1µs resolution, as in 1., while numpy’s datetime64 allows you to select the unit, so you could have a 1 ns resolution.)

How easy would it be to instantiate and work with a Julia DateTime with a different resolution? How much is everything predicated on the immutable UTInstant in UTMillisecond? Are there any users/libraries doing this?

(For background, I’m trying to convince a team I’m working with to consider Julia instead of Python, and tried to code something up quickly to demonstrate how easy it is, and basically stumbled at the first opportunity, trying to parse "2017-05-03 09:00:00.007935” (note date and time with micro second resolution)… :-/ )

I’d hope implementing a useful TimeStamp based on UTMicrosecond should be not too hard, though it’s unclear to me how it fits into the whole infrastructure (particularly the fancy new parser is somewhat hard to read).

A TimeStamp based on UTNanosecond would have to have a different epoch to be useful, so that would be somewhat harder.

Thanks,
Fab

PS: There was this related thread, but that’s been a long time ago… :slight_smile:
https://groups.google.com/forum/#!searchin/julia-users/datetime$20resolution|sort:relevance/julia-users/R25Mz7nWo7o/0nLMlSqAsuYJ

1 Like
2

I’ve hacked together a quick implementation of the bare bones I needed for this task - it’s really nice how quick that is possible in Julia (and how fast it works), but of course this is neither integrated into the existing infrastructure nor does it really have any functionality.

TimeStampJ bare bones implementation 
immutable TimeStampJ
    instant::Int64
end

# constructor
function TimeStampJ(y::Int64, m::Int64, d::Int64,
                  h::Int64=0, mi::Int64=0, s::Int64=0, micros::Int64=0)
    0 < m < 13 || throw(ArgumentError("Month: $m out of range (1:12)"))
    0 < d < Dates.daysinmonth(y, m) + 1 || throw(ArgumentError("Day: $d out of range (1:$(Dates.daysinmonth(y, m)))"))
    -1 < h < 24 || throw(ArgumentError("Hour: $h out of range (0:23)"))
    -1 < mi < 60 || throw(ArgumentError("Minute: $mi out of range (0:59)"))
    -1 < s < 60 || throw(ArgumentError("Second: $s out of range (0:59)"))
    -1 < micros < 1000000 || throw(ArgumentError("Microsecond: $ms out of range (0:999999)"))
    rata = micros + 1000000 * (s + 60mi + 3600h + 86400 * Dates.totaldays(y, m, d))
    return TimeStampJ(rata)
end

# prettyprint
function Base.string(t::TimeStampJ)
    s, micros = fldmod(t.instant, 1000000)
    mi, s = fldmod(s, 60)
    h, mi = fldmod(mi, 60)
    days, h = fldmod(h, 24)
    y, m, d = Dates.yearmonthday(days)

    yy = y < 0 ? @sprintf("%05i",y) : lpad(y,4,"0")
    mm = lpad(m,2,"0")
    dd = lpad(d,2,"0")
    hh = lpad(h,2,"0")
    mii = lpad(mi,2,"0")
    ss = lpad(s,2,"0")
    micross = micros == 0 ? "" : @sprintf("%.6f", micros / 1e+6)[2:end]
    return "$yy-$mm-$(dd)T$hh:$mii:$ss$micross"
end

Base.show(io::IO, x::TimeStampJ) = print(io, Base.string(x))

# parse from string "2017-05-03 09:00:00.048425"
function TimeStampJ_fromISO(dtiso::String)
    y = parse(Int,dtiso[1:4])
    m = parse(Int,dtiso[6:7])
    d = parse(Int,dtiso[9:10])
    h = parse(Int,dtiso[12:13])
    mi = parse(Int,dtiso[15:16])
    s = parse(Int,dtiso[18:19])
    micros = parse(Int,dtiso[21:26])

    return TimeStampJ(y, m, d, h, mi, s, micros)
end
3 Likes
3

Probably unrelated: TimeSeries.jl allows you to specify decimal accuracy in your DataTime types. There are also clean structures and functions for handling all sorts of calendar trickinesseses.

4

TimeSeries.jl allows you to specify decimal accuracy in your DataTime types

Oh cool, can you point me to where? From having a quick look at the documentation I didn’t find it (maybe I was looking at the wrong version; or are you referring to the pretty printing?)

1 Like
5

Darn, I glazed over it myself.
If I am not mistaken, your DateTimes will still be well ordered, even up to nanoseconds.
I am interested to find put as I am also probably going to use this soon.
If not, a PUll Request would probably be welcome over there.

6

Well, but from what I understand, Julia DateTimes do not measure nanoseconds, they measure milliseconds.

My TimeStampJ hack above measures micro seconds (with the same “zero” as DateTime, allowing +/- 200k years).

BTW, I’ve implemented a hand-rolled parser, no error checking whatsoever, but fast:

Customized timestamp parsing 
function TimeStampJ_fromISO(dtiso)::TimeStampJ
    pos = 1
    y = 0
    for k = 1:4
        c, pos = next(dtiso, pos)
        y = y * 10 + (c - '0')
    end
    
    c, pos = next(dtiso, pos)
    
    c, pos = next(dtiso, pos)
    m = (c - '0')
    c, pos = next(dtiso, pos)
    m = m*10 + (c - '0')

    c, pos = next(dtiso, pos)
    
    c, pos = next(dtiso, pos)
    d = (c - '0')
    c, pos = next(dtiso, pos)
    d = d*10 + (c - '0')

    c, pos = next(dtiso, pos)

    c, pos = next(dtiso, pos)
    h = (c - '0')
    c, pos = next(dtiso, pos)
    h = h*10 + (c - '0')

    c, pos = next(dtiso, pos)

    c, pos = next(dtiso, pos)
    mi = (c - '0')
    c, pos = next(dtiso, pos)
    mi = mi*10 + (c - '0')

    c, pos = next(dtiso, pos)

    c, pos = next(dtiso, pos)
    s = (c - '0')
    c, pos = next(dtiso, pos)
    s = s*10 + (c - '0')

    c, pos = next(dtiso, pos)

    micros = 0
    for k = 1:6
        c, pos = next(dtiso, pos)
        micros = micros * 10 + (c - '0')
    end

    return TimeStampJ(y, m, d, h, mi, s, micros)
end
7

It appears there is an implementation of a Time type in v0.6 that includes nanosecond resolution:
https://github.com/JuliaLang/julia/pull/12274

8

That seems to be there already in 0.5 - Time with ns resolution (not capturing the date).

DateTime has the date, but only millisecond resolution, from what I gather.

I could try and add a TimeStamp type with microsecond resolution, or one could go ahead and make the whole thing more general, parametrised with an offset (epoch) and multiplier (resolution).

9

@joshualeond , thanks for the link, informative. Didn’t realise this was all in flux so much!

Can I ping people here? @quinnj, @iamed2, @JeffreySarnoff, @tkelman - any feedback?

Is there a plan for

  • DateTime with microsecond resolution (+/- 200k years)
  • DateTime with nanosecond resolution (+/- 200 years) and different epoch
  • DateTime with 128 bits (many many years with huge resolution…)
  • parametrised DateTime?

Anything I can do?

Cheers
Fab

10

If you want to help with a 128 bit time package, email me directly.

Jeffrey Sarnoff

11

Hi Fab,

If you’re after a scientific timekeeping library, perhaps you’d be interested in a nascent package created by @andyferris and myself - https://github.com/FugroRoames/Chrono.jl ?

The ideas there are geared toward scientific time keeping:

  • Time is measured as a duration since the epoch of a clock.
  • Many clock types may exist; time measured by one clock may or may not be convertible to time measured by another clock, depending on whether the clocks have been synchronized by a physical experiment.
  • Duration is parameterized on both the multiple of the second in which the clock ticks (may be a rational or floating point number), and a storage type. So you can have a clock which ticks in 1/3 femtoseconds, with Int128 storage type just as easily as one ticking in days (86400s) with Int32 storage.
  • There are functions to take the difference between two UTC date times, with correct support for leap seconds.

Honestly, I don’t think we’ve quite settled on the design of Chrono.jl (in particular, regarding the nature and use of clock types, and clarifying the thinking about how the conversion between times as measured by different clocks should be handled). We also haven’t had the chance to work on it for a while, but I’d love for this - or some other package like it - to become a standard for scientific time keeping in julia.

Cheers,
~Chris

1 Like
12

Since Julia is billed as a Scientific Computing Language I’m surprised the built in time library cannot do everything your Chrono.jl sets out to do. Have you considered enhancing the built in DateTime instead of building an external library?

Bob

1 Like
13

It seems to be a common development pattern in Julia to deploy new functionality as a separate package - which can then in time be merged into the standard functionality with time.

3 Likes
14

Hi Chris,
so that would be an SI second based time (unlike the Base Date, Time, DateTime, which is based on UT seconds with 86400 seconds per day)?

That would certainly be cool to have, not sure to which extent it’s in Base already. However, not sure it’s necessary for my use case.

(EDIT: Julia Base Date, Time, DateTime uses UT seconds = 1/86400 day = measure of angle, not “TI seconds”, whatever that is, as I erroneously wrote)

15

Interesting! Would be nice to see it as the base to AstronomicalTime.jl, astronomy is perhaps the field with the most complicated time system (there are several of them and the tick of the clock depends on many different physical effects, including general relativistic ones).

16

Yes, the ticks of clocks in Chrono.jl are intended to be multiples of SI seconds. This is in contrast to Base, where last I looked a second was a somewhat ambiguously defined UT second (not the UTC realization of UT though, as DateTime can’t represent leap seconds). This choice makes DateTime quite simple to understand and very suitable for customary time keeping. However, it’s unsuitable for high precision scientific timekeeping, even ignoring the millisecond precision issue.

As to why the Chrono functionality isn’t in Base there’s two reasons. First, it’s always a good idea to prototype outside of the standard library, and we didn’t yet get far enough to make a solid proposal. Second, the kind of duration arithmetic desired for customary time systems like the Gregorian calendar is somewhat incompatible with scientific timekeeping. For example, what does adding one month to a date mean?

In a customary system representing time points as dates, one might reasonably desire that simultaneously

Date(2000,1,1) + Month(1) == Date(2000,2,1)
Date(2000,2,1) + Month(1) == Date(2000,3,1)

and this is the case in julia-0.6. However, these two months are different lengths, so Month(1) doesn’t have a well defined duration in seconds.

In a scientific system, we’d presumably want month = Dates.Second(2629746) (the long time average number of seconds in a month, ignoring leap seconds), in which case we’d have

julia> DateTime(2000,1,1) + month
2000-01-31T10:29:06

which will be confusing to most users, and arguably quite incorrect for code which needs to deal with calendars.

And all this before we’ve even got to worrying about UTC and leap seconds, let alone the physical reality of clock drift and relativistic effects :slight_smile:

@giordano here’s a fun article about the JILA Sr optical clock where the team says their clock is good enough to measure a height difference of 2cm vertically due to general relativistic time dilation. As for AstronomicalTime.jl, I’d love for something like Chrono.jl to be a useful base layer for things like this. @helgee kindly sent a PR early on (https://github.com/FugroRoames/Chrono.jl/pull/2) but we got bogged down in some conceptual difficulties, and I’m glad to see he’s just getting on with it in a separate package :slight_smile: Someday we’ll have a good Base package for this stuff though, I think.

17

LabView, another technical-computing language, uses a very handy 128-bit fixed-point type for its timestamps: http://www.ni.com/tutorial/7900/en/
Would be nice to have something similar in Julia.

18

There’s no need for everything to go into Base. Packages have no performance penalty, are easier to maintain, and make Julia lean. Something like this is a great utility package.

1 Like
19

Nothing ambiguous about it: Base’s time is in UT1, modulo time zones (which are a whole other can of worms). This is in contrast to many other language’s time systems which claim to be in UTC and then ignore leap seconds – which is contradictory, not just ambiguous.

20

The leap seconds will be handled from v0.6?

In v0.5.2 I get:

julia> DateTime("2016-12-31T23:59:59")
2016-12-31T23:59:59

julia> DateTime("2016-12-31T23:59:60")
ERROR: ArgumentError: Second: 60 out of range (0:59)
 in DateTime(::Base.Dates.Year, ::Base.Dates.Month, ::Base.Dates.Day, ::Base.Dates.Hour, ::Base.Dates.Minute, 
...

The most recent leap second was 2016-12-31T23:59:60.

1 Like