The latest docs say:
Anyway, the fact that Date and DateTime are based on UT seconds is a simplifying, yet honest assumption so that things like leap seconds and all their complexity can be avoided.
So it looks like they’re not supported…
No, there is no plan at the moment to handle leap seconds in Base – doing so would change the meaning of time in Base, making it no longer UT1. It would also make Base time calculations depend on an external and changing data set: the leap seconds database. I think this is one of the reasons why most languages simply ignore leap seconds, even though that technically makes their notions of time incoherent if they claim to support UTC. UTC is a somewhat computationally unfortunate worst-of-all-worlds standard (I understand the motivation, however): it ticks in AI seconds – i.e. “physics seconds” – but keeps within 0.9 seconds of UT1 by occasionally adding leap seconds, so that UTC midnight remains in sync with the rotation of the earth. I believe that correct handling of UTC entails computing ΔT (the difference between Universal Time and Terrestrial Time) although it’s possible that one could do less by just keeping an up-to-date table of leap seconds. The simplest way to get Base Julia to accept UTC formatted times is to just ignore leap seconds, treating 2016-12-31T23:59:60 as another name for 2016-12-31T23:59:59. I’m not sure that’s a good idea, however, and raising an error seems like a safer option for the time being.
1 Like
Sorry, this doesn’t make sense. The difference between UT1 and UTC is presently about 500-600 ms, and changes all the time. This would be very confusing for users, who have 99.9% of their timestamps in UTC (plus eventual time zone offset).
According to my experiene (working with satellite data) the simplest way is to have internally a so-called day segmented time representation: nr of days since some epoch (e.g. 2000-01-01), and the nr of milli/nano/pico … seconds on that day, i.e. two integers, or one integer and a float, whatever precision you want to support. This time is UTC, where each day has 86399, 86400, or 86401 seconds. The input functions must accept a time like “…T23:59:60”, at least on the days of a leap second (can only be June 30 or December 31). E.g. sateliites, stars have positions also at such times. To return a period type between two time stamps where one or more leap seconds are in between one could still assume that each day has 86400 s and make it clear, that the result is without leap seconds. If somebody needs with leap seconds, refer to conversion to TAI, see below.
To support “exotic” users of time functions, Julia should support conversion to/from TAI and GPS time. This conversion is a simple offset, but jumps whenever a leap second is inserted which is known at least 6 months in advance (there is a historical period 1961-1968 when TAI and UT both ran at different paces, an IT nightmare, perhaps this does not need to be supported any more). To claim support of UT1 one would need to digest the weekly or so IERS data.
Stephan, this discussion has been had. It makes sense. Julia doesn’t use SI seconds in base.
Base’s time is in UT1
Sorry, this doesn’t make sense.
As in it is not a coherent statement or as in you think it’s a bad idea?
To claim support of UT1 one would need to digest the weekly or so IERS data.
We would only need that to convert to or from UT1, which we don’t do (in Base). All we do is provide a representation of time in which each day has precisely 86400 seconds, and whose interpretation is recommended as UT1 (modulo time zones). Since each day in UT1 has precisely 86400 seconds, this works. If you want to work with satellites and TAI, you should not be using Julia’s Base time at all. Rather, you should use a time representation that implements TAI or something derived from it (like UTC).
This would be very confusing for users, who have 99.9% of their timestamps in UTC.
Everyone claims their times are in UTC, but then 99.9% of them ignore leap seconds or don’t handle them correctly. Programming languages do this, operating systems do this, financial modeling software does this, programmers do it. It seems better to be honest about it and simply say that we do not count time in Base using SI seconds – jarring though this may be if one is accustomed fudging the difference.
If you have strong feelings about this (as it seems you do), you may want to consider writing a Julep (Julia Enhancement Proposal) with a detailed proposal for a better handling of time in Base and/or the broader Julia ecosystem. As a warning, I would be prepared for some friendly debate on the matter.
3 Likes
Base’s time is in UT1, modulo time zones
You’re right, it’s been documented as UT1 since the DateTime module was first merged before version 0.4.
Using UT1 is a good choice for time points and it simplifies the underlying date time arithmetic considerably. It does have the interesting side effect that Durations are not expressed in SI seconds.
UTC itself does not ignore leap seconds, 2016-12-31T23:59:59.314Z is a valid UTC time according to ISO 8601. Almost all software systems (databases, etc.) cannot handle the leap seconds and assume that each day has 86400 s. Julia is not worse in this respect, but I have ideas how to make it better without throwing the existing DateTime completely over board:
Internally times inside leap seconds should be representable, in order to be able to time stamp real events. This is difficult, if you count the nr of seconds (or milliseconds…) from a certain epoch, like the POSIX time in seconds from 1970-01-01. Then the leap seconds collide with the assumption that every day has 86400. If the internal representation is day segmented (my previous post), no problem to represent timestamps inside leap seconds. Conversion back to date/time strings is not more complicated than with single counters, rather easier. Performance hit and additional memory is probably negligible.
Conversion to TAI, GPS can be handled in a packages, probably not so many users need it.
UT1 has a definition connected to the Earth rotation, and it is measured and changes all the time relative to UTC. Whenever the difference exceeds about 1.5 s, a leap second is inserted into UTC, to bring the difference down again. So yes, UT1 doesn’t have leap seconds, but it also really doesn’t have years, months, days. From an IT viewpoint it is just a table of differences to UTC.
Ok, I see now that I should send this Julep, …
1 Like
@stephancb, did you see the various packages mentioned earlier in the thread? Chronos.jl, AstronomicalTime.jl?
UTC itself does not ignore leap seconds, 2016-12-31T23:59:59.314Z is a valid UTC time according to ISO 8601.
Leap seconds only exist in UTC – they are a feature of how it bridges UT and TAI.
Almost all software systems (databases, etc.) cannot handle the leap seconds and assume that each day has 86400 s. Julia is not worse in this respect
Yes, precisely – most software systems implement a UT time scheme and then claim to implement UTC. This is incoherent. Instead, Julia Base explicitly claims to implement UT time with seconds that are defined in terms of the Earth’s rotation. This is both pragmatic (no table of leap seconds) and honest (no bogus claims of implementing UTC).
UT1 has a definition connected to the Earth rotation, and it is measured and changes all the time relative to UTC. Whenever the difference exceeds about 1.5 s, a leap second is inserted into UTC, to bring the difference down again. So yes, UT1 doesn’t have leap seconds, but it also really doesn’t have years, months, days. From an IT viewpoint it is just a table of differences to UTC.
Claiming that UT doesn’t have days, months or years seems backwards since UT is based on the rotation of the Earth, which is precisely where those concepts are natural. Connecting TT and TAI with days, months and years is what requires all the gymnastics with leap seconds. You’re also assuming that one needs to work on UTC and can only get to UT1 by using a table of differences. That’s not true, however, one can just as well work in UT1 consistently and only convert to UTC/TAI using a table of differences if one happens to really need SI seconds. Using UT seconds in Base Julia is essentially an assertion that most users do not need SI seconds and are better served by working with seconds that are 1/86400 of a day. POSIX does not have leap seconds; most operating systems don’t do leap seconds and will never give you leap seconds; and most computer hardware won’t either. If you’re getting time from satellite systems, then yes, you will have real UTC values and need to handle leap seconds. In that case, you should, as I said, not be using Base’s time types.
2 Likes
Great discussion here; a few notes from when we originally developed and merged the Dates module into Base Julia
-
There was originally support for leap seconds, meaning
DateTimeoriginally supported UTC, but after lengthy consideration & debate (similar to this one), it was decided that the majority of data analysis & timestamp use-cases were in fact accustomed to UT1 representations, without support for leap seconds. In order to provide compatibility, along with a drastically simpler implementation in terms of arithmetic and not needing to provide leap second database updates, Base.Dates.DateTime represents UT1 with millisecond resolution, 84,600 seconds to a day. -
In the
Base.Datesmodule, it was constructed so that external packages could extend and interop w/ the current Base definitions ofDateandDateTime. TheTimeZones.jlis an excellent example of a package that provides an additionalTimeTypecalled aZonedDateTimethat supports timezone aware timestamps, arithmetic, etc. TheDatesmodule was certainly designed to be extensible and allow appropriate alternative representations, given the simplified approach taken in Base. -
Other notes of interest in the design of the
Base.Datesmodule are theDates.Instant,Dates.UTInstant, andDates.Calendartypes. TheDateTimetype itself is defined in Base using theUTInstantrepresentation, but that doesn’t preclude the definition and use of aUTCInstantorTAIInstantInstantsubtype. Similarly withCalendar,Base.Datescurrently assumes the proleptic gregorian calendar representation when calculating years & months back thru history, but that doesn’t preclude someone defining their ownCalendarsubtype and providing separate logic. -
In terms of resolution, it was again a decision weighing the tradeoff between simple, clear code in Base that addressed the majority of use-cases, while leaving things extensible for packages to provide additional functionality.
Millisecondresolution has the nice property of being able to represent a “good enough” range of dates with full millisecond precision (i.e.-146138511-01-01T00:00:00-146138512-12-31T23:59:59). Additional precision cuts down the possible range of values by orders of magnitude and would require additional bits (than the current Int64).
Hopefully that gives a little background on the Dates module design and some of the tradeoffs, design decisions that were made. I pointed out that certain use-cases were deliberately not implemented (timezones, leap seconds, > millisecond resolution), but the design was (hopefully) made to be extensible enough to allow packages to do so.
1 Like
Not really. With each leap second the difference between UTC and TAI increases, it is now exactly 37 s. The leap seconds are inserted into UTC in order to keep the difference between UT1 and UTC within about +/- 900 ms (not 1.5 s as I wrote earlier). See here for a graph.
Well, not pragmatic from the users viewpoint. In 99.9 % of all cases, the time stamp that is to be entered into the Julia system is meant to be UTC, e.g. when obtained from the computer clock, that is synchornized with the NTP protocol, or a GPS receiver or so. If the user took the claim seriously, then she would have to go to the weekly IERS Bulletin A, available by ftp here, and check what is the difference between UT1 and UTC on that day, convert the UTC time stamp to UT1 for input into Julia DateTime functions. Likewise when getting out a time stamp from Julia, the user would have to assume, it is UT1 and convert to the desired UTC, again by consulting the IERS tables. Nobody does this of course in practice, I just describe what the claim that Julia times stamps are UT1 would imply for a meticulous user.
“seconds that are defined in terms of the Earth’s rotation” are not defined. The SI second is defined in terms of atomic oscillations. UTC, TAI, and GPS increase their seconds in intervals of exactly SI seconds, but not UT1 The difference between UT1 and UTC on each day is given by the IERS in SI seconds, that’s all they provide (and a direction of the Earth’s rotation axis, not relevant for time). To get a “UT1 second” one could take two UTC times, add the respective DUT1s (difference between UT1 and UTC) and calculate the nr of SI seconds between the two UT1 times divided by nr of SI seconds between the UTC times. The result is a length of the “UT1 second” in units of SI seconds, but it would depend on the two times. The Earth’s rotation period changes as a result of big hurricanes, monsun, Earth quakes, things happening at the core/mantle boundary etc, it is not a good basis for defining a second.
On my computer I get
julia> now()
2017-05-17T21:49:59.523
which was obviously UTC taken from the computer clock synchronized to UTC (within a few ms). If it were UT1 which is right now slightly more more than 500 ms off, it should have been 2017-05-17T21:50:00.xxx. If I type
julia> t=DateTime(now())
the variable t has quite obviously the time stamp also internally in UTC as no lookup of the time difference between UT1 and UTC has been done inside Julia. Therefore the claim that Julia time is in UT1 is factually not correct on computers with clocks synchronized to UTC, i.e. almost all of them.
I think that the DateTime documentation needs to be updated in this respect, and I can try to formulate a suggestion.
Yes, presently most computer clocks will never return time stamps within leap seconds, they are programmed to jump over them. But this may get improved in the future and data acquisition that can handle time stamps in leap seconds exists now. It is not a big deal, just one second of data possibly lost every few years because of software not accepting the time stamps, but Julia would have a chance to handle this more correctly. I’ll try to see how the Julep works to provide more details of my suggestion.
I wonder where you get this number from. My expectation would be that the majority of users just don’t know enough about the intricacies of leap seconds and time standards, and don’t have a strong reason to care about the difference either way.
1 Like
A possible review of the Date and Time documention:
Delete “…on the UT second [1].” including the complete footnote and replace by “…on UTC [1].” and the footnote
[1] Since 1972 UTC is the worldwide time standard. UTC advances at the same rate as TAI (Temps Atomique International) which itself is based on the SI second. Time stamping systems have become increasingly accurate in syncing to UTC. Thanks to the NTP protocol in combination with modern OS kernels the system time of computers and internet connected devices is accurate to at least a few ms, often better. GPS/GNSS and radio broadcasts (LORAN, DCF77 etc) provide high precision timing in numerous data acquisition . The vast majority of absolute time stamps in the world are UTC, and DateTime is normally used to handle UTC time stamps.
A section on leap seconds should be inserted:
Leap seconds
The ITU decides from time to time, on average roughly every 18 months, to insert a leap second into UTC. The leap seconds have been difficult to handle in technical computing (including Julia), but the reason is that UTC is so prevented from drifting too far away from a time based on the Earth’s rotation, nowadays called UT1. There have been suggestions how to handle this differently, but the ITU has for now decided to continue with the insertion of leap seconds as needed at least until 2023.
In ISO 8601 a time stamp within a leap seconds would for example be “2016-12-31T23:59:60.4”. A system call for a time stamp within a leap second, for example using the Julia now() function, should never return such time stamps, if NTP is configured and the OS not too old, but rather a time stamp just outside the leap second. Time stampig based on other clocks (separate oscillators, GPS, radio broadcasts) could in principle produce them, or the user can (for fun?) just type them. Time stamps in leap seconds are not accepted by DateTime constructors, an error is thrown. The user would need to filter out data with such time stamps or restamp them, before the constructor is called.
----- end of section Leap seconds
One can certainlly argue here about the wording, style,spelling etc., but the technical contents are, I think, facts.
Oh, yes, to answer the original question, yes, millisecond accuracy in DateTime seems not enough, see above statements about normally achieved timing accuracy.
For suggested revisions to the manual, you should simply open a pull request against https://github.com/JuliaLang/julia/blob/master/doc/src/manual/dates.md . That will allow all the specific changes to be discussed within the github system.
AstronomicalTime.md seems to be pretty empty right now, so I cannot say
much about it.
Chronos.jl looks fine, but I cannot see that it has anything about UT1,
which is the issue here.
/Stephan
The README is surely empty, but the code is not. Timescales that should be eventually supported are:
- UTC
- UT1
- TAI
- TT
- TCG
- TCB
- TDB
Sorry to re-raise an old topic. I personally do think there is value in having the standard library DateTime implementation have flexible and arbitrary resolution. Looking at the code, couldn’t this be done it a quite simple and non-breaking way by changing the definition of DateTime to struct DateTime{I <: Instant}... and having the default implementation (returned from the default constructors / parsers etc) be DateTime{UTInstant{Millisecond}}? This is similar behaviour to numpy’s datetime64[x] where x can be s, ms, us, ns etc., but defaults to ns when constructed by most constructors. I think if done correctly this could be achieved with relatively little extra code complexity, and should also be non-breaking and not affect performance at all.
1 Like
A PR demonstrating this may be the quickest way to convince people.
Yes, of course - at this stage I was more just trying to initiate a discussion / see if there were any very obvious flaws with the idea that would get identified much more quickly by people who have no doubt spent much more time with the codebase than I have.
2 Likes
Good idea! I wish it was this simple, but if you make DateTime parametric it becomes a non-concrete type (when spelled without the type parameters). So user code which expects it to be a concrete type (backed by the bits of an Int64) will suddenly become really slow. For example, consider code which constructs a Vector{DateTime}() and then fills it with a loop. Or any struct containing a DateTime, which suddenly gets a pointer in it rather than an inlined Int64.
3 Likes