Higher resolution DateTime/TimeStamp

dates

#6

Well, but from what I understand, Julia DateTimes do not measure nanoseconds, they measure milliseconds.

My TimeStampJ hack above measures micro seconds (with the same “zero” as DateTime, allowing +/- 200k years).

BTW, I’ve implemented a hand-rolled parser, no error checking whatsoever, but fast:

Customized timestamp parsing
function TimeStampJ_fromISO(dtiso)::TimeStampJ
    pos = 1
    y = 0
    for k = 1:4
        c, pos = next(dtiso, pos)
        y = y * 10 + (c - '0')
    end
    
    c, pos = next(dtiso, pos)
    
    c, pos = next(dtiso, pos)
    m = (c - '0')
    c, pos = next(dtiso, pos)
    m = m*10 + (c - '0')

    c, pos = next(dtiso, pos)
    
    c, pos = next(dtiso, pos)
    d = (c - '0')
    c, pos = next(dtiso, pos)
    d = d*10 + (c - '0')

    c, pos = next(dtiso, pos)

    c, pos = next(dtiso, pos)
    h = (c - '0')
    c, pos = next(dtiso, pos)
    h = h*10 + (c - '0')

    c, pos = next(dtiso, pos)

    c, pos = next(dtiso, pos)
    mi = (c - '0')
    c, pos = next(dtiso, pos)
    mi = mi*10 + (c - '0')

    c, pos = next(dtiso, pos)

    c, pos = next(dtiso, pos)
    s = (c - '0')
    c, pos = next(dtiso, pos)
    s = s*10 + (c - '0')

    c, pos = next(dtiso, pos)

    micros = 0
    for k = 1:6
        c, pos = next(dtiso, pos)
        micros = micros * 10 + (c - '0')
    end

    return TimeStampJ(y, m, d, h, mi, s, micros)
end

#7

It appears there is an implementation of a Time type in v0.6 that includes nanosecond resolution:
https://github.com/JuliaLang/julia/pull/12274


#8

That seems to be there already in 0.5 - Time with ns resolution (not capturing the date).

DateTime has the date, but only millisecond resolution, from what I gather.

I could try and add a TimeStamp type with microsecond resolution, or one could go ahead and make the whole thing more general, parametrised with an offset (epoch) and multiplier (resolution).


#9

@joshualeond , thanks for the link, informative. Didn’t realise this was all in flux so much!

Can I ping people here? @quinnj, @iamed2, @JeffreySarnoff, @tkelman - any feedback?

Is there a plan for

  • DateTime with microsecond resolution (+/- 200k years)
  • DateTime with nanosecond resolution (+/- 200 years) and different epoch
  • DateTime with 128 bits (many many years with huge resolution…)
  • parametrised DateTime?

Anything I can do?

Cheers
Fab


#10

If you want to help with a 128 bit time package, email me directly.

Jeffrey Sarnoff


#11

Hi Fab,

If you’re after a scientific timekeeping library, perhaps you’d be interested in a nascent package created by @andyferris and myself - https://github.com/FugroRoames/Chrono.jl ?

The ideas there are geared toward scientific time keeping:

  • Time is measured as a duration since the epoch of a clock.
  • Many clock types may exist; time measured by one clock may or may not be convertible to time measured by another clock, depending on whether the clocks have been synchronized by a physical experiment.
  • Duration is parameterized on both the multiple of the second in which the clock ticks (may be a rational or floating point number), and a storage type. So you can have a clock which ticks in 1/3 femtoseconds, with Int128 storage type just as easily as one ticking in days (86400s) with Int32 storage.
  • There are functions to take the difference between two UTC date times, with correct support for leap seconds.

Honestly, I don’t think we’ve quite settled on the design of Chrono.jl (in particular, regarding the nature and use of clock types, and clarifying the thinking about how the conversion between times as measured by different clocks should be handled). We also haven’t had the chance to work on it for a while, but I’d love for this - or some other package like it - to become a standard for scientific time keeping in julia.

Cheers,
~Chris


Space/astro domain
#12

Since Julia is billed as a Scientific Computing Language I’m surprised the built in time library cannot do everything your Chrono.jl sets out to do. Have you considered enhancing the built in DateTime instead of building an external library?

Bob


#13

It seems to be a common development pattern in Julia to deploy new functionality as a separate package - which can then in time be merged into the standard functionality with time.


#14

Hi Chris,
so that would be an SI second based time (unlike the Base Date, Time, DateTime, which is based on UT seconds with 86400 seconds per day)?

That would certainly be cool to have, not sure to which extent it’s in Base already. However, not sure it’s necessary for my use case.

(EDIT: Julia Base Date, Time, DateTime uses UT seconds = 1/86400 day = measure of angle, not “TI seconds”, whatever that is, as I erroneously wrote)


#15

Interesting! Would be nice to see it as the base to AstronomicalTime.jl, astronomy is perhaps the field with the most complicated time system (there are several of them and the tick of the clock depends on many different physical effects, including general relativistic ones).


#16

Yes, the ticks of clocks in Chrono.jl are intended to be multiples of SI seconds. This is in contrast to Base, where last I looked a second was a somewhat ambiguously defined UT second (not the UTC realization of UT though, as DateTime can’t represent leap seconds). This choice makes DateTime quite simple to understand and very suitable for customary time keeping. However, it’s unsuitable for high precision scientific timekeeping, even ignoring the millisecond precision issue.

As to why the Chrono functionality isn’t in Base there’s two reasons. First, it’s always a good idea to prototype outside of the standard library, and we didn’t yet get far enough to make a solid proposal. Second, the kind of duration arithmetic desired for customary time systems like the Gregorian calendar is somewhat incompatible with scientific timekeeping. For example, what does adding one month to a date mean?

In a customary system representing time points as dates, one might reasonably desire that simultaneously

Date(2000,1,1) + Month(1) == Date(2000,2,1)
Date(2000,2,1) + Month(1) == Date(2000,3,1)

and this is the case in julia-0.6. However, these two months are different lengths, so Month(1) doesn’t have a well defined duration in seconds.

In a scientific system, we’d presumably want month = Dates.Second(2629746) (the long time average number of seconds in a month, ignoring leap seconds), in which case we’d have

julia> DateTime(2000,1,1) + month
2000-01-31T10:29:06

which will be confusing to most users, and arguably quite incorrect for code which needs to deal with calendars.

And all this before we’ve even got to worrying about UTC and leap seconds, let alone the physical reality of clock drift and relativistic effects :slight_smile:

@giordano here’s a fun article about the JILA Sr optical clock where the team says their clock is good enough to measure a height difference of 2cm vertically due to general relativistic time dilation. As for AstronomicalTime.jl, I’d love for something like Chrono.jl to be a useful base layer for things like this. @helgee kindly sent a PR early on (https://github.com/FugroRoames/Chrono.jl/pull/2) but we got bogged down in some conceptual difficulties, and I’m glad to see he’s just getting on with it in a separate package :slight_smile: Someday we’ll have a good Base package for this stuff though, I think.


#17

LabView, another technical-computing language, uses a very handy 128-bit fixed-point type for its timestamps: http://www.ni.com/tutorial/7900/en/
Would be nice to have something similar in Julia.


#18

There’s no need for everything to go into Base. Packages have no performance penalty, are easier to maintain, and make Julia lean. Something like this is a great utility package.


#19

Nothing ambiguous about it: Base’s time is in UT1, modulo time zones (which are a whole other can of worms). This is in contrast to many other language’s time systems which claim to be in UTC and then ignore leap seconds – which is contradictory, not just ambiguous.


Is Dates.UTInstant UT1, UT2, or UTC?
#20

The leap seconds will be handled from v0.6?

In v0.5.2 I get:

julia> DateTime("2016-12-31T23:59:59")
2016-12-31T23:59:59

julia> DateTime("2016-12-31T23:59:60")
ERROR: ArgumentError: Second: 60 out of range (0:59)
 in DateTime(::Base.Dates.Year, ::Base.Dates.Month, ::Base.Dates.Day, ::Base.Dates.Hour, ::Base.Dates.Minute, 
...

The most recent leap second was 2016-12-31T23:59:60.


#22

The latest docs say:

Anyway, the fact that Date and DateTime are based on UT seconds is a simplifying, yet honest assumption so that things like leap seconds and all their complexity can be avoided.

So it looks like they’re not supported…


#23

No, there is no plan at the moment to handle leap seconds in Base – doing so would change the meaning of time in Base, making it no longer UT1. It would also make Base time calculations depend on an external and changing data set: the leap seconds database. I think this is one of the reasons why most languages simply ignore leap seconds, even though that technically makes their notions of time incoherent if they claim to support UTC. UTC is a somewhat computationally unfortunate worst-of-all-worlds standard (I understand the motivation, however): it ticks in AI seconds – i.e. “physics seconds” – but keeps within 0.9 seconds of UT1 by occasionally adding leap seconds, so that UTC midnight remains in sync with the rotation of the earth. I believe that correct handling of UTC entails computing ΔT (the difference between Universal Time and Terrestrial Time) although it’s possible that one could do less by just keeping an up-to-date table of leap seconds. The simplest way to get Base Julia to accept UTC formatted times is to just ignore leap seconds, treating 2016-12-31T23:59:60 as another name for 2016-12-31T23:59:59. I’m not sure that’s a good idea, however, and raising an error seems like a safer option for the time being.


#24

Sorry, this doesn’t make sense. The difference between UT1 and UTC is presently about 500-600 ms, and changes all the time. This would be very confusing for users, who have 99.9% of their timestamps in UTC (plus eventual time zone offset).

According to my experiene (working with satellite data) the simplest way is to have internally a so-called day segmented time representation: nr of days since some epoch (e.g. 2000-01-01), and the nr of milli/nano/pico … seconds on that day, i.e. two integers, or one integer and a float, whatever precision you want to support. This time is UTC, where each day has 86399, 86400, or 86401 seconds. The input functions must accept a time like “…T23:59:60”, at least on the days of a leap second (can only be June 30 or December 31). E.g. sateliites, stars have positions also at such times. To return a period type between two time stamps where one or more leap seconds are in between one could still assume that each day has 86400 s and make it clear, that the result is without leap seconds. If somebody needs with leap seconds, refer to conversion to TAI, see below.

To support “exotic” users of time functions, Julia should support conversion to/from TAI and GPS time. This conversion is a simple offset, but jumps whenever a leap second is inserted which is known at least 6 months in advance (there is a historical period 1961-1968 when TAI and UT both ran at different paces, an IT nightmare, perhaps this does not need to be supported any more). To claim support of UT1 one would need to digest the weekly or so IERS data.


#25

Stephan, this discussion has been had. It makes sense. Julia doesn’t use SI seconds in base.


#26

Base’s time is in UT1

Sorry, this doesn’t make sense.

As in it is not a coherent statement or as in you think it’s a bad idea?

To claim support of UT1 one would need to digest the weekly or so IERS data.

We would only need that to convert to or from UT1, which we don’t do (in Base). All we do is provide a representation of time in which each day has precisely 86400 seconds, and whose interpretation is recommended as UT1 (modulo time zones). Since each day in UT1 has precisely 86400 seconds, this works. If you want to work with satellites and TAI, you should not be using Julia’s Base time at all. Rather, you should use a time representation that implements TAI or something derived from it (like UTC).

This would be very confusing for users, who have 99.9% of their timestamps in UTC.

Everyone claims their times are in UTC, but then 99.9% of them ignore leap seconds or don’t handle them correctly. Programming languages do this, operating systems do this, financial modeling software does this, programmers do it. It seems better to be honest about it and simply say that we do not count time in Base using SI seconds – jarring though this may be if one is accustomed fudging the difference.

If you have strong feelings about this (as it seems you do), you may want to consider writing a Julep (Julia Enhancement Proposal) with a detailed proposal for a better handling of time in Base and/or the broader Julia ecosystem. As a warning, I would be prepared for some friendly debate on the matter.