Hi, as I understand, in Julia we have Date (with 1d resolution, and an “epoch” of Rata Dei 0001-01-01 with the good old proleptic Gregorian calendar), DateTime (with 1 ms resolution, and the same “epoch”, able to cover +/-200 m years), as well as Time (with 1 ns resolution). I’m looking for a DateTime or TimeStamp with a bit higher resolution. It would appear that there are two options (while staying within an Int64): something with microsecond resolution, centred at the same epoch, with +/- 200k years something with nanosecond resolution, covering only +/- 200 years, thus should probably be centred around some other epoch, e.g. the 1970 UNIX one. (It appears that Python has a datetime with 1µs resolution, as in 1., while numpy’s datetime64 allows you to select the unit, so you could have a 1 ns resolution.) How easy would it be to instantiate and work with a Julia DateTime with a different resolution? How much is everything predicated on the immutable UTInstant in UTMillisecond? Are there any users/libraries doing this? (For background, I’m trying to convince a team I’m working with to consider Julia instead of Python, and tried to code something up quickly to demonstrate how easy it is, and basically stumbled at the first opportunity, trying to parse "2017-05-03 09:00:00.007935” (note date and time with micro second resolution)… :-/ ) I’d hope implementing a useful TimeStamp based on UTMicrosecond should be not too hard, though it’s unclear to me how it fits into the whole infrastructure (particularly the fancy new parser is somewhat hard to read). A TimeStamp based on UTNanosecond would have to have a different epoch to be useful, so that would be somewhat harder. Thanks, Fab PS: There was this related thread, but that’s been a long time ago… :slight_smile: https://groups.google.com/forum/#!searchin/julia-users/datetime$20resolution|sort:relevance/julia-users/R25Mz7nWo7o/0nLMlSqAsuYJ

Base’s time is in UT1, modulo time zones You’re right, it’s been documented as UT1 since the DateTime module was first merged before version 0.4. Using UT1 is a good choice for time points and it simplifies the underlying date time arithmetic considerably. It does have the interesting side effect that Durations are not expressed in SI seconds.

UTC itself does not ignore leap seconds, 2016-12-31T23:59:59.314Z is a valid UTC time according to ISO 8601. Almost all software systems (databases, etc.) cannot handle the leap seconds and assume that each day has 86400 s. Julia is not worse in this respect, but I have ideas how to make it better without throwing the existing DateTime completely over board: Internally times inside leap seconds should be representable, in order to be able to time stamp real events. This is difficult, if you count the nr of seconds (or milliseconds…) from a certain epoch, like the POSIX time in seconds from 1970-01-01. Then the leap seconds collide with the assumption that every day has 86400. If the internal representation is day segmented (my previous post), no problem to represent timestamps inside leap seconds. Conversion back to date/time strings is not more complicated than with single counters, rather easier. Performance hit and additional memory is probably negligible. Conversion to TAI, GPS can be handled in a packages, probably not so many users need it. UT1 has a definition connected to the Earth rotation, and it is measured and changes all the time relative to UTC. Whenever the difference exceeds about 1.5 s, a leap second is inserted into UTC, to bring the difference down again. So yes, UT1 doesn’t have leap seconds, but it also really doesn’t have years, months, days. From an IT viewpoint it is just a table of differences to UTC. Ok, I see now that I should send this Julep, …

@stephancb, did you see the various packages mentioned earlier in the thread? Chronos.jl, AstronomicalTime.jl?

UTC itself does not ignore leap seconds, 2016-12-31T23:59:59.314Z is a valid UTC time according to ISO 8601. Leap seconds only exist in UTC – they are a feature of how it bridges UT and TAI. Almost all software systems (databases, etc.) cannot handle the leap seconds and assume that each day has 86400 s. Julia is not worse in this respect Yes, precisely – most software systems implement a UT time scheme and then claim to implement UTC. This is incoherent. Instead, Julia Base explicitly claims to implement UT time with seconds that are defined in terms of the Earth’s rotation. This is both pragmatic (no table of leap seconds) and honest (no bogus claims of implementing UTC). UT1 has a definition connected to the Earth rotation, and it is measured and changes all the time relative to UTC. Whenever the difference exceeds about 1.5 s, a leap second is inserted into UTC, to bring the difference down again. So yes, UT1 doesn’t have leap seconds, but it also really doesn’t have years, months, days. From an IT viewpoint it is just a table of differences to UTC. Claiming that UT doesn’t have days, months or years seems backwards since UT is based on the rotation of the Earth, which is precisely where those concepts are natural. Connecting TT and TAI with days, months and years is what requires all the gymnastics with leap seconds. You’re also assuming that one needs to work on UTC and can only get to UT1 by using a table of differences. That’s not true, however, one can just as well work in UT1 consistently and only convert to UTC/TAI using a table of differences if one happens to really need SI seconds. Using UT seconds in Base Julia is essentially an assertion that most users do not need SI seconds and are better served by working with seconds that are 1/86400 of a day. POSIX does not have leap seconds; most operating systems don’t do leap seconds and will never give you leap seconds; and most computer hardware won’t either. If you’re getting time from satellite systems, then yes, you will have real UTC values and need to handle leap seconds. In that case, you should, as I said, not be using Base’s time types.

Great discussion here; a few notes from when we originally developed and merged the Dates module into Base Julia There was originally support for leap seconds, meaning DateTime originally supported UTC, but after lengthy consideration & debate (similar to this one), it was decided that the majority of data analysis & timestamp use-cases were in fact accustomed to UT1 representations, without support for leap seconds. In order to provide compatibility, along with a drastically simpler implementation in terms of arithmetic and not needing to provide leap second database updates, Base.Dates.DateTime represents UT1 with millisecond resolution, 84,600 seconds to a day. In the Base.Dates module, it was constructed so that external packages could extend and interop w/ the current Base definitions of Date and DateTime. The TimeZones.jl is an excellent example of a package that provides an additional TimeType called a ZonedDateTime that supports timezone aware timestamps, arithmetic, etc. The Dates module was certainly designed to be extensible and allow appropriate alternative representations, given the simplified approach taken in Base. Other notes of interest in the design of the Base.Dates module are the Dates.Instant, Dates.UTInstant, and Dates.Calendar types. The DateTime type itself is defined in Base using the UTInstant representation, but that doesn’t preclude the definition and use of a UTCInstant or TAIInstant Instant subtype. Similarly with Calendar, Base.Dates currently assumes the proleptic gregorian calendar representation when calculating years & months back thru history, but that doesn’t preclude someone defining their own Calendar subtype and providing separate logic. In terms of resolution, it was again a decision weighing the tradeoff between simple, clear code in Base that addressed the majority of use-cases, while leaving things extensible for packages to provide additional functionality. Millisecond resolution has the nice property of being able to represent a “good enough” range of dates with full millisecond precision (i.e. -146138511-01-01T00:00:00 - 146138512-12-31T23:59:59). Additional precision cuts down the possible range of values by orders of magnitude and would require additional bits (than the current Int64). Hopefully that gives a little background on the Dates module design and some of the tradeoffs, design decisions that were made. I pointed out that certain use-cases were deliberately not implemented (timezones, leap seconds, > millisecond resolution), but the design was (hopefully) made to be extensible enough to allow packages to do so.

[image] StefanKarpinski: Leap seconds only exist in UTC – they are a feature of how it bridges UT and TAI. Not really. With each leap second the difference between UTC and TAI increases, it is now exactly 37 s. The leap seconds are inserted into UTC in order to keep the difference between UT1 and UTC within about +/- 900 ms (not 1.5 s as I wrote earlier). See here for a graph. [image] StefanKarpinski: Julia Base explicitly claims to implement UT time with seconds that are defined in terms of the Earth’s rotation. This is both pragmatic (no table of leap seconds) and honest (no bogus claims of implementing UTC). Well, not pragmatic from the users viewpoint. In 99.9 % of all cases, the time stamp that is to be entered into the Julia system is meant to be UTC, e.g. when obtained from the computer clock, that is synchornized with the NTP protocol, or a GPS receiver or so. If the user took the claim seriously, then she would have to go to the weekly IERS Bulletin A, available by ftp here, and check what is the difference between UT1 and UTC on that day, convert the UTC time stamp to UT1 for input into Julia DateTime functions. Likewise when getting out a time stamp from Julia, the user would have to assume, it is UT1 and convert to the desired UTC, again by consulting the IERS tables. Nobody does this of course in practice, I just describe what the claim that Julia times stamps are UT1 would imply for a meticulous user. “seconds that are defined in terms of the Earth’s rotation” are not defined. The SI second is defined in terms of atomic oscillations. UTC, TAI, and GPS increase their seconds in intervals of exactly SI seconds, but not UT1 The difference between UT1 and UTC on each day is given by the IERS in SI seconds, that’s all they provide (and a direction of the Earth’s rotation axis, not relevant for time). To get a “UT1 second” one could take two UTC times, add the respective DUT1s (difference between UT1 and UTC) and calculate the nr of SI seconds between the two UT1 times divided by nr of SI seconds between the UTC times. The result is a length of the “UT1 second” in units of SI seconds, but it would depend on the two times. The Earth’s rotation period changes as a result of big hurricanes, monsun, Earth quakes, things happening at the core/mantle boundary etc, it is not a good basis for defining a second. On my computer I get julia> now() 2017-05-17T21:49:59.523 which was obviously UTC taken from the computer clock synchronized to UTC (within a few ms). If it were UT1 which is right now slightly more more than 500 ms off, it should have been 2017-05-17T21:50:00.xxx. If I type julia> t=DateTime(now()) the variable t has quite obviously the time stamp also internally in UTC as no lookup of the time difference between UT1 and UTC has been done inside Julia. Therefore the claim that Julia time is in UT1 is factually not correct on computers with clocks synchronized to UTC, i.e. almost all of them. I think that the DateTime documentation needs to be updated in this respect, and I can try to formulate a suggestion. Yes, presently most computer clocks will never return time stamps within leap seconds, they are programmed to jump over them. But this may get improved in the future and data acquisition that can handle time stamps in leap seconds exists now. It is not a big deal, just one second of data possibly lost every few years because of software not accepting the time stamps, but Julia would have a chance to handle this more correctly. I’ll try to see how the Julep works to provide more details of my suggestion.

[image] stephancb: In 99.9 % of all cases, the time stamp that is to be entered into the Julia system is meant to be UTC I wonder where you get this number from. My expectation would be that the majority of users just don’t know enough about the intricacies of leap seconds and time standards, and don’t have a strong reason to care about the difference either way.

A possible review of the Date and Time documention: Delete “…on the UT second [1].” including the complete footnote and replace by “…on UTC [1].” and the footnote [1] Since 1972 UTC is the worldwide time standard. UTC advances at the same rate as TAI (Temps Atomique International) which itself is based on the SI second. Time stamping systems have become increasingly accurate in syncing to UTC. Thanks to the NTP protocol in combination with modern OS kernels the system time of computers and internet connected devices is accurate to at least a few ms, often better. GPS/GNSS and radio broadcasts (LORAN, DCF77 etc) provide high precision timing in numerous data acquisition . The vast majority of absolute time stamps in the world are UTC, and DateTime is normally used to handle UTC time stamps. A section on leap seconds should be inserted: Leap seconds The ITU decides from time to time, on average roughly every 18 months, to insert a leap second into UTC. The leap seconds have been difficult to handle in technical computing (including Julia), but the reason is that UTC is so prevented from drifting too far away from a time based on the Earth’s rotation, nowadays called UT1. There have been suggestions how to handle this differently, but the ITU has for now decided to continue with the insertion of leap seconds as needed at least until 2023. In ISO 8601 a time stamp within a leap seconds would for example be “2016-12-31T23:59:60.4”. A system call for a time stamp within a leap second, for example using the Julia now() function, should never return such time stamps, if NTP is configured and the OS not too old, but rather a time stamp just outside the leap second. Time stampig based on other clocks (separate oscillators, GPS, radio broadcasts) could in principle produce them, or the user can (for fun?) just type them. Time stamps in leap seconds are not accepted by DateTime constructors, an error is thrown. The user would need to filter out data with such time stamps or restamp them, before the constructor is called. ----- end of section Leap seconds One can certainlly argue here about the wording, style,spelling etc., but the technical contents are, I think, facts. Oh, yes, to answer the original question, yes, millisecond accuracy in DateTime seems not enough, see above statements about normally achieved timing accuracy.

For suggested revisions to the manual, you should simply open a pull request against https://github.com/JuliaLang/julia/blob/master/doc/src/manual/dates.md . That will allow all the specific changes to be discussed within the github system.

AstronomicalTime.md seems to be pretty empty right now, so I cannot say much about it. Chronos.jl looks fine, but I cannot see that it has anything about UT1, which is the issue here. /Stephan

Higher resolution DateTime/TimeStamp

Internals & Design

yakir12 February 6, 2019, 10:00am 45

angtft

Timestamp String with one-second resolution: fastest implementation?

Topic		Replies	Views
Universal Time vs UTC Time in Dates General Usage dates	11	1940	August 26, 2023
`DateTime` arithmetic on `Microsecond` (or smaller) scale Internals & Design dates , datetime	99	5230	August 26, 2023
TimesDates.jl dev for v2 Specific Domains dates , time , internals	44	3127	September 14, 2023
What will happen to Julia's DateTime type General Usage datetime	7	364	October 16, 2026
[ANN] TimesDates.jl: times & zones with nanoseconds Community package , announcement , dates	3	1324	April 18, 2022

Higher resolution DateTime/TimeStamp

Related topics