DataTables : convert Nullable{String}("14:00:00") to Dates.Minute

question

#1

Hi,

I have a DataTable with two columns of time and I want to compute the time duration between these 2 columns. The problem I have is to compute a duration with Nullable type.

 julia> a, b
(Nullable{String}("14:00:00"),Nullable{String}("15:15:00"))

julia> b-a
ERROR: MethodError: no method matching -(::String, ::String)
 in -(::Nullable{String}, ::Nullable{String}) at /home/fred/.julia/v0.5/NullableArrays/src/operators.jl:132


julia> convert(Dates.Minute,a)
ERROR: MethodError: Cannot `convert` an object of type Nullable{String} to an object of type Base.Dates.Minute
This may have arisen from a call to the constructor Base.Dates.Minute(...),
since type constructors fall back to convert methods.

julia> Dates.DateTime(a,"HH:MM")
ERROR: MethodError: Cannot `convert` an object of type Nullable{String} to an object of type Int64
This may have arisen from a call to the constructor Int64(...),
since type constructors fall back to convert methods.
 in DateTime(::Nullable{String}, ::String) at ./dates/types.jl:116

Thanks for your help !


#2

You probably want something like:

a, b = (Nullable{String}("14:00:00"),Nullable{String}("15:15:00"))

Base.Dates.Time(str::String) = Dates.Time(map(x->parse(Int, x), split(str, ':'))...)
difflift(a, b) = (isnull(a) || isnull(b)) ? Nullable{Dates.Nanosecond}() : Nullable{Dates.Nanosecond}(b - a)

aT, bT = Dates.Time(get(a)), Dates.Time(get(b))
difflift(aT, bT)

The idea here is you’re doing “lifting” on the nullable arguments in case either is actually null. Also, this is currently using 0.6 code where the Dates.Time type has been added. You could also use a DateTime without too much trouble since you’re just looking for the difference.


#3

@quinnj Thank you very much for your answer. I did not imagine it was so complex to compute a time difference :astonished:

I am currently using Julia 5.0 and I have this result :

julia> aT, bT = DateTime(get(a)), DateTime(get(b))
ERROR: ArgumentError: Delimiter mismatch. Couldn't find first delimiter, "-", in date string
 in parse(::String, ::Base.Dates.DateFormat) at ./dates/io.jl:169
 in DateTime(::String) at ./dates/io.jl:268

So the solution is only possible with Julia 0.6 ?


#4

On Julia 0.6, the .( and .- operators will automatically lift arguments for you, so this will be enough:

julia> a, b = (Nullable{String}("14:00:00"),Nullable{String}("15:15:00"))
(Nullable{String}("14:00:00"), Nullable{String}("15:15:00"))

julia> Base.Dates.Time(str::String) = Dates.Time(map(x->parse(Int, x), split(str, ':'))...)

julia> a2 = Dates.Time.(a)
Nullable{Base.Dates.Time}(14:00:00)

julia> b2 = Dates.Time.(b)
Nullable{Base.Dates.Time}(15:15:00)

julia> b2 .- a2
Nullable{Base.Dates.Nanosecond}(4500000000000 nanoseconds)

@quinnj Looks like we need parsing methods to create Time objects from strings?


#5

On Julia v0.5:

julia> Dates.DateTime(get(b), "HH:MM:SS") - Dates.DateTime(get(a), "HH:MM:SS")  
4500000 milliseconds

although if you look the date is Jan 1st 1 AD/CE, which is a bit distracting… :slight_smile:


#6

Thank you !

@nalimilan
I tried on julia 0.6 and I still have a problem to convert nanoseconds to minutes for example

julia> Base.Dates.Time(str::String) = Dates.Time(map(x->parse(Int, x), split(str, ':'))...)

julia> a2 = Dates.Time.(a)
Nullable{Base.Dates.Time}(14:00:00)

julia> b2 = Dates.Time.(b)
Nullable{Base.Dates.Time}(15:15:00)

julia> b2 .- a2
Nullable{Base.Dates.Nanosecond}(4500000000000 nanoseconds)

julia> Int(b2 .- a2)
ERROR: MethodError: Cannot `convert` an object of type Nullable{Base.Dates.Nanosecond} to an object of type Int64
This may have arisen from a call to the constructor Int64(...),
since type constructors fall back to convert methods.
Stacktrace:
 [1] Int64(::Nullable{Base.Dates.Nanosecond}) at ./sysimg.jl:24

julia> c=b2 .- a2
Nullable{Base.Dates.Nanosecond}(4500000000000 nanoseconds)

julia> c/1e3
ERROR: MethodError: no method matching /(::Nullable{Base.Dates.Nanosecond}, ::Float64)
Closest candidates are:
  /(::Float64, ::Float64) at float.jl:377
  /(::BigFloat, ::Union{Float16, Float32, Float64}) at mpfr.jl:343
  /(::Complex, ::Real) at complex.jl:268
  ...

with the solution of @cormullion it finaly works better on Julia 0.5 because all operations on times are possible

 julia> a, b = (Nullable{String}("14:00:00"),Nullable{String}("15:15:00"))
(Nullable{String}("14:00:00"),Nullable{String}("15:15:00"))

julia> (Nullable{String}("14:00:00"), Nullable{String}("15:15:00"))
(Nullable{String}("14:00:00"),Nullable{String}("15:15:00"))

julia> Dates.DateTime(get(b), "HH:MM:SS") - Dates.DateTime(get(a), "HH:MM:SS") 
4500000 milliseconds

julia> c = Dates.DateTime(get(b), "HH:MM:SS") - Dates.DateTime(get(a), "HH:MM:SS")
4500000 milliseconds

julia> c*60
270000000 milliseconds

julia> Int(c)/1e3
4500.0

julia> Int(c)/1e3/60
75.0

#7

Everything which works in Julia 0.5 also works on 0.6. The question is, do you want to extract the value from the Nullable wrapper immediately (which implies that no null values are possible): then use get first. OTOH, if you need to support missing values, then keep the Nullable, but then you need to use .(, .-, etc. for all operations.


#8

Thank you very much @nalimilan ! I understand better the role of the get function.
Since, to my knowledge, this useful function does not appear in the DataTables documentation, a short explanation how to extract the value from the Nullable wrapper would be very useful in the section “The Nullable Type”.

julia> c = b2 .- a2
Nullable{Base.Dates.Nanosecond}(4500000000000 nanoseconds)

julia> d = get(c)
4500000000000 nanoseconds

julia> Int(d)
4500000000000

julia> Int(d)/1e9
4500.0

#9

Yeah, that section of the DataTables manual could be improved. It should also point to the relevant section of the Julia manual. Would you make a pull request to change this?


#10

@nalimilan I tried to make a pull request (for the first time :slight_smile:) and I did not succeed. I did not find the “documentation” section of DataTables.jl on GitHub.
Do I have to fork the whole JuliaData/DataTables.jl directory to make a pull request ? Then on the “Compare changes” I don’t know what to choose. Sorry for asking those questions, but it is unclear of that works :flushed:


#11

It’s here: https://github.com/JuliaData/DataTables.jl/blob/master/docs/src/man/getting_started.md You can click on the small pen button to open a pull request directly from GitHub.


#12

Thank you @nalimilan, I did it, I hope it is correct :wink:


#13

Sorry, but I can’t see it…


#14

@nalimilan I forgot to validate, now it appears :slight_smile: