Universal Time vs UTC Time in Dates

[barrettp]

@danielwe

Specifically, the documentation states that Dates assumes Universal Time - Wikipedia , which is basically solar time, meaning that the lengths of days and seconds are nonuniform due to variations in the earth’s rotation. This is convenient because it means each day has exactly 86400 seconds and you don’t have to fuss with leap seconds in DateTime arithmetic, but it also means that even millisecond precision is a bit nonsensical, and locating events with nanosecond precision on a timeline that’s uniform across centuries is way beyond the scope of the module.

Are you sure you mean UT/UT1 and not UTC, because all computer clocks are synchronized using NTP which uses UTC as its time standard. So when you ask the computer for the date you are getting UTC, not UT1. Does that mean that DateTime converts UTC to UT1? Note that UT1 is based on Julian days, which is what astronomers use for calculations. If you want a time scale without leap seconds, then use TAI or Julian days.

Note that the international time standard community has talked about getting rid of leap seconds. The US has wanted to do this for years. One proposal to keep UTC close to UT1 is to apply a running offset that is base on a polynomial estimate of DUT1.

[danielwe]

@barrettp

Are you sure you mean UT/UT1 and not UTC

Yes, UT/UT1 and not UTC, as mentioned a couple of times in the documentation and elaborated on in a footnote: Dates · The Julia Language

@barrettp

Does that mean that DateTime converts UTC to UT1?

No, I don’t think so. The way I read the documentation, the only point of mentioning UT/UT1 is as a simplifying assumption for datetime representation and arithmetic: you just count (milli)seconds from an epoch and every 86400 seconds represents a day. That’s why I’m also saying the millisecond precision seems a bit nonsensical.

Fundamentally, the DateTime type is designed to represent the information contained in an unqualified ISO 8601 date/time, not to exactly place an event relative to a specific time standard.

[dlakelan]

@danielwe

module now needs to know about leap seconds (assuming UTC timestamps).

https://docs.julialang.org/en/v1/stdlib/Dates/#footnote-1

Footnote notes that in the time model used for DateTime there are no leap seconds. (Edit: LOL, just realized you were the one who posted that above!) Now, if the time model used in the generation of the data has leap seconds then yeah, you’ve got issues, provided that a leap second occurred within the bounds of your dataset, which, hopefully we’ll get rid of leapseconds and then won’t have to deal with it? From what I can see there haven’t been any since 2017 or so

[mihalybaci]

It would appear that the decision has already been made to end leapseconds.

`DateTime` arithmetic on `Microsecond` (or smaller) scale

@Oscar_Smith brought up an argument for why it shouldn’t error - namely, imagine using DateTime and its arithmetic as the time variable in an ODE. DateTime shouldn’t be used for time intervals (UTM though is there for that); nor then I think ODEs. DateTime is for ISO 8601 timestamps, as documented; I guess for 8601-1 (going forward, not yet documented as such I believe), as opposed to 8601-2: https://www.isotc154.org/posts/2019-08-27-introduction-to-the-new-8601/ ISO 8601 is now a family…

[danielwe]

@dlakelan

if the time model used in the generation of the data has leap seconds then yeah, you’ve got issues, provided that a leap second occurred within the bounds of your dataset

The obvious issue is that calculated intervals may be off by a second (so much for nanosecond precision), but another is that there may be times in the dataset that are not even representable by DateTime:

julia> using Dates

julia> DateTime("2016-12-31T23:59:60")
ERROR: ArgumentError: Second: 60 out of range (0:59)

[stephancb]

The documentation is incorrect. Timestamps are generated by clocks, i.e. hardware. In Julia the now() function asks the OS what time it is, and the OS reads the hardware clock. Other software has comparable functionality to generate time stamps via the OS from a clock, and these timestamps might also end up in Julia when reading databases etc…

So which time standard do all this clocks in computers, mobile/cell phones etc use? Since about 2 decades they get synchronized to each other with help of internet protocols (NTP is dominating)., and the time standard is UTC. UTC is (currently) based on the SI second and atomic clocks, not on any astronomical objects like the Earth or the Sun. In UTC a day can have 86399, 86400, or 86401 seconds, and almost all days have of course 86400 seconds.

Especially footnote 1 in the documentation is nonsense. I would be happy with something like

“Julia provides the function now() to obtain a timestamp from the OS and the constructor of DateTime accepts ISO8601 strings like 2023-08-25T21:20:46.223 for importing externally generated timestamps. Most timestamps that Julia is possibly exposed to are UTC, as almost all timestamp generating clocks in the world are synchronized to UTC. Julia does, like also most other software systems, not comply with UTC at leap seconds: For example DateTime("2016-12-31T23:59:60.5") with a valid UTC time string throws an error.”

[mbauman]

There are two separate questions here:

• Is now() actually, meaningfully UTC in a way that breaks this UT1 assumption? Implementation-wise, it grabs its value from Libc’s count of seconds since the POSIX epoch — and that does not include leap seconds in its count, explicitly assuming that there are exactly 86400 seconds per day. But even if that weren’t its implementation and it read the clock on your computer as a string, it almost certainly wouldn’t see a 23:59:60 timestamp thanks to NTP leap smear. Now I suppose the abolition of leap seconds might push this a little closer towards it being a real UTC time, but I don’t know enough about how operating systems will deal with that to speculate more.
• How should Julia handle IO with dates that include a 23:59:60 timestamp? Given that leap seconds are encoded into the tz database, this seems like a similar sort of leap smear could occur when using TimeZones.jl to parse a ZonedDateTime in the 24 hours around leap seconds to shift the times to UT1, but given the sloppiness above, how would you know you’re not double-shifting them?

All in all, this does seem like a:

simplifying, yet honest assumption

[stephancb]

Good questions.

Is now() actually, meaningfully UTC?

With NTP (normally automatically configured and up and running, you never worry about it or touch it) and standard internet connections now() is UTC (plus TZ offset) accurately to a few tens of msec. If there is a nearby accurate NTP server, like in some university LANs, it can be better, a few msec. Therefore in Julia DateTime a time code with msec resolution is a reasonable choice. It is slightly better than the typical accuracy of now(). As you mention, NTP warns ahead of leap seconds, and some OS react to this with slowing down and accelerating the system clock near the leap second. At these very rare occasions the error of now() becomes of course larger, up to about 500 msec. It is also true that the OS normally does not keep tables of leap seconds, but this does not affect the correctness of now() and comparable functionality in other software. It works anyway, too long to write out why.

In summary, now() and similar functions in Python, Matlab, MySQL, … normally return a UTC timestamp with an accuracy of at least tens of msec, often better. Practically all timestamps that a software like Julia possibly has to deal with are UTC (including timestamped astronomical data).

How should Julia handle IO with dates that include a 23:59:60 timestamp?

Throwing an error is perhaps not a big issue, as the leap seconds are so rare, and such timestamps often don’t make it to IO to Julia, because other software handling this before also errors. But it could be handled by using internally a day segmented time code (for all times!), e.g.

struct DsTimeCode  # day segmented time code, medium resolution
                   # (can represnt leap second times)
    day2000::Int32 # day2000=0  is 2000-01-01
    msec::Int32    # milliseconds of day
end

as I had suggested earlier. This allows to accept the ISO8601 string with “60” as the last second of the day, and to store the corresponding binary timestamp. It still is 64 bit in total, and comparing timestamps has very little overhead compared to a single 64 bit counter, etc. Ideally this sort of time code should have been used when UNIX and Windows were designed, but introducing it now in Julia would still have advantages.

[barrettp]

UT1 is not what you think it is. It is a timescale based on the diurnal motion of the stars, actually distant radio galaxies. It is measured twice per day. It is therefore not equal to 86400 seconds per day, because the Earth’s rotation and the motion of the Earth around the Sun varies by milliseconds per day. Per the “Explanatory Supplement to the Astronomical Almanac”, broadcast time services distribute timescales based on the redefined Coordinated Universal Time (UTC), which differs from Atomic Time (TAI) by an integer number of seconds. DUT1 is an extrapolation of the difference between UT1 minus UTC. It is transmitted in code on broadcast time signals. TAI is a practical time standard that conforms as closely as possible to the definition of the SI second. So let’s please stop using Universal Time (UT1). NTP time might be more appropriate.

[mbauman]

I think I may understand: your objections are not so much that we ignore leap seconds or that we use a non-SI-length second; it’s that we (ab)use the term UT1 to describe such assumptions?

[barrettp]

Yes, the documentation, and discussion, should not refer to UT1. For someone like me with some knowledge of time standards this is confusing. It would be good to precisely define what, if any, time standard is being used so as not to mislead potential users.

[tim.holy]

Since you have the expertise, can you submit a pull request to fix it? There may not be anyone else in Julia development who could.