# Parse, don't validate

**URL:** <https://discourse.julialang.org/t/parse-dont-validate/63618>\
**Category:** General Usage\
**Tags:** parsing\
**Created:** [June 26, 2021, 7:05pm UTC](https://discourse.julialang.org/t/parse-dont-validate/63618 "2021-06-26T19:05:26Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![rikh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rikh/32/204104_2.png) [@rikh](https://discourse.julialang.org/u/rikh)\
**Post date:** [June 26, 2021, 7:05pm UTC](https://discourse.julialang.org/t/parse-dont-validate/63618/1 "2021-06-26T19:05:26Z")

</div>

The [Parse, don’t validate](https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/) blog post came by on Hacker News again. As far as I understand, this is advertising to use type checking instead of checking the object each time. Are there valuable lessons to be learned in this post in the Julia context? For example, would it be a good idea to use a `NonEmpty` type?

---

<div class="post-metadata">

**Author:** ![jzr](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@jzr](https://discourse.julialang.org/u/jzr)\
**Post date:** [June 26, 2021, 10:21pm UTC](https://discourse.julialang.org/t/parse-dont-validate/63618/2 "2021-06-26T22:21:40Z")

</div>

_Parse, don’t validate_ is a great idea, and Julia code will benefit from doing it. In a typed dynamic language, there is a question of how to implement the idea. E.g.:

- Double down on types, use things like [SumTypes.jl](https://github.com/MasonProtter/SumTypes.jl),
- Double down on dynamism, use things like Clojure’s [Spec](https://clojure.org/guides/spec),
- Use [traits](https://github.com/schlichtanders/WhereTraits.jl).

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [June 27, 2021, 12:17am UTC](https://discourse.julialang.org/t/parse-dont-validate/63618/3 "2021-06-27T00:17:33Z")

</div>

I think there’s already a lot that you can do in the spirit of “parse, don’t validate” using Julia’s type system. Of course what you end up with when there are bugs is `MethodError`s instead of static compilation errors.

One thing that I sometimes forget is that if I’m writing code that processes data that I’ve created myself inside my program, then I don’t need to validate it, since I’m the one who created it and I know what shape it’s in. 🙂

So I think the advice in “parse, don’t validate” is focused mostly on processing input data. But some of the advice applies generally, like this one:

> Use a data structure that makes illegal states unrepresentable.

Sometimes you see a function like this:

```julia
function foo(; flag1, flag2)
    if flag1 && flag2
        1
    elseif flag1 && !flag2
        2
    elseif !flag1 && flag2
        3
    else
        error("flag1 and flag2 cannot both be false")
    end
end        

```

But you could just make the 4th state illegal by using the type system:

```julia
# Hopefully there are more natural names that you can use
# in your real application.
struct Flag1Flag2 end
struct Flag1NotFlag2 end
struct NotFlag1Flag2 end

foo(::Flag1Flag2) = 1
foo(::Flag1NotFlag2) = 2
foo(::NotFlag1Flag2) = 3

```

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [June 27, 2021, 12:36am UTC](https://discourse.julialang.org/t/parse-dont-validate/63618/4 "2021-06-27T00:36:47Z")

</div>

Another example that I think is related. It’s nice to avoid struct fields with `Union{Nothing, T}` types, if possible. So instead of this,

```julia
abstract type AbstractPerson end

struct Person <: AbstractPerson
    name::Union{Nothing, String}
    age::Int
end

```

you could do this:

```julia
abstract type AbstractPerson end

struct Person <: AbstractPerson
    name::String
    age::Int
end

struct Anonymous <: AbstractPerson
    age::Int
end

```

---

<div class="post-metadata">

**Author:** ![CameronBieganek](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/cameronbieganek/32/6915_2.png) [@CameronBieganek](https://discourse.julialang.org/u/CameronBieganek)\
**Post date:** [June 28, 2021, 8:24pm UTC](https://discourse.julialang.org/t/parse-dont-validate/63618/6 "2021-06-28T20:24:10Z")

</div>

Parse, don’t validate… with type checking by [JET.jl](https://github.com/aviatesk/JET.jl):

### File

```julia
# parse_dont_validate.jl

struct NonEmpty{T}
    head::T
    tail::Vector{T}
end

head(x::NonEmpty) = x.head

function foo()
    x = rand(3)
    head(x)
end

```

### JET

Scroll to the bottom to see the type error:

```julia
julia> report_file("parse_dont_validate.jl"; analyze_from_definitions=true)
[toplevel-info] virtualized the context of Main (took 0.001 sec)
[toplevel-info] entered into parse_dont_validate.jl
[toplevel-info] exited from parse_dont_validate.jl (took 0.004 sec)
[toplevel-info] analyzing from top-level definitions ... 4/4
═════ 3 possible errors found ═════
┌ @ parse_dont_validate.jl:11 rand(3)
│┌ @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Random/src/Random.jl:277 Random.rand(Random.Float64, Random.Dims(dims))
││┌ @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Random/src/Random.jl:289 Random.default_rng()
│││┌ @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Random/src/RNGs.jl:370 Random.default_rng(Base.getproperty(Random.Threads, :threadid)())
││││┌ @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Random/src/RNGs.jl:376 Random.MersenneTwister()
│││││┌ @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Random/src/RNGs.jl:147 #self#(Random.nothing)
││││││┌ @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Random/src/RNGs.jl:147 Random.seed!(Random.MersenneTwister(Core.apply_type(Random.Vector, Random.UInt32)(), Random.DSFMT_state()), seed)
│││││││┌ @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Random/src/Random.jl:426 Random.seed!(rng)
││││││││┌ @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Random/src/RNGs.jl:362 Random.make_seed()
│││││││││┌ @ /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.6/Random/src/RNGs.jl:326 Random.read(Random.pipeline(Base.cmd_gen(Core.tuple(Core.tuple("ifconfig"))), Base.cmd_gen(Core.tuple(Core.tuple("sha1sum")))), Random.String)
││││││││││┌ @ process.jl:421 Base.read(cmd)
│││││││││││┌ @ process.jl:410 Base.open(cmd, "r", Base.devnull)
││││││││││││┌ @ process.jl:339 Core.kwfunc(Base.open)(Core.apply_type(Core.NamedTuple, (:read, :write))(Core.tuple(true, true)), Base.open, cmds, stdio)
│││││││││││││┌ @ process.jl:361 Base.#open#646(write, read, _3, cmds, stdio)
││││││││││││││┌ @ process.jl:365 Base._spawn(cmds, Base.getindex(Base.Any, in, out, Base.stderr))
│││││││││││││││┌ @ process.jl:119 Base.setup_stdios(#639, stdios)
││││││││││││││││┌ @ process.jl:196 f(open_io)
│││││││││││││││││┌ @ process.jl:120 Base._spawn(Core.getfield(#self#, :cmds), stdios, Base.ProcessChain())
││││││││││││││││││┌ @ process.jl:151 Base._spawn(Base.getproperty(cmds, :b), stdios_right, chain)
│││││││││││││││││││┌ @ process.jl:181 Base._spawn_primitive(Base.getindex(Base.getproperty(cmd, :exec), 1), cmd, stdios)
││││││││││││││││││││┌ @ process.jl:99 Base.repr(cmd)
│││││││││││││││││││││┌ @ strings/io.jl:219 Base.#repr#386(Base.nothing, #self#, x)
││││││││││││││││││││││┌ @ strings/io.jl:219 Core.kwfunc(Base.sprint)(Core.apply_type(Core.NamedTuple, (:context,))(Core.tuple(context)), Base.sprint, Base.show, x)
│││││││││││││││││││││││┌ @ strings/io.jl:101 Base.#sprint#385(Core.tuple(context, sizehint, _3, f), args...)
││││││││││││││││││││││││┌ @ strings/io.jl:105 f(Core.tuple(s), args...)
│││││││││││││││││││││││││┌ @ cmd.jl:116 Base.map(#620, Base.getproperty(cmd, :exec))
││││││││││││││││││││││││││┌ @ abstractarray.jl:2294 Base.collect_similar(A, Base.Generator(f, A))
│││││││││││││││││││││││││││┌ @ array.jl:606 Base._collect(cont, itr, Base.IteratorEltype(itr), Base.IteratorSize(itr))
││││││││││││││││││││││││││││┌ @ array.jl:691 Base.iterate(itr)
│││││││││││││││││││││││││││││┌ @ generator.jl:47 Base.getproperty(g, :f)(Base.getindex(y, 1))
││││││││││││││││││││││││││││││┌ @ cmd.jl:117 Core.kwfunc(Base.sprint)(Core.apply_type(Core.NamedTuple, (:context,))(Core.tuple(Core.getfield(#self#, :io))), Base.sprint, #621)
│││││││││││││││││││││││││││││││┌ @ strings/io.jl:101 Base.#sprint#385(Core.tuple(context, sizehint, _3, f), args...)
││││││││││││││││││││││││││││││││┌ @ strings/io.jl:103 f(Core.tuple(Base.IOContext(s, context)), args...)
│││││││││││││││││││││││││││││││││┌ @ cmd.jl:118 Base.with_output_color(#622, :underline, io)
││││││││││││││││││││││││││││││││││┌ @ util.jl:71 Base.#with_output_color#814(Core.tuple(false, #self#, f, color, io), args...)
│││││││││││││││││││││││││││││││││││┌ @ util.jl:85 Base.split(str, '\n')
││││││││││││││││││││││││││││││││││││┌ @ strings/util.jl:411 Base.#split#375(0, true, #self#, str, splitter)
│││││││││││││││││││││││││││││││││││││┌ @ strings/util.jl:411 Base._split(str, Base.isequal(splitter), limit, keepempty, _7)
││││││││││││││││││││││││││││││││││││││┌ @ strings/util.jl:421 Base.first(r)
│││││││││││││││││││││││││││││││││││││││┌ @ abstractarray.jl:386 Base.iterate(itr)
││││││││││││││││││││││││││││││││││││││││ no matching method found for call signature (Tuple{typeof(iterate), Nothing}): Base.iterate(itr::Nothing)
│││││││││││││││││││││││││││││││││││││││└────────────────────────
││││││││││││││││││││││││││││││││││││││┌ @ strings/util.jl:421 Base.last(r)
│││││││││││││││││││││││││││││││││││││││┌ @ abstractarray.jl:437 Base.lastindex(a)
││││││││││││││││││││││││││││││││││││││││ no matching method found for call signature (Tuple{typeof(lastindex), Nothing}): Base.lastindex(a::Nothing)
│││││││││││││││││││││││││││││││││││││││└────────────────────────
┌ @ parse_dont_validate.jl:12 head(x)
│ no matching method found for call signature (Tuple{typeof(head), Vector{Float64}}): head(x::Vector{Float64})
└─────────────────────────────
(included_files = Set(["parse_dont_validate.jl"]), any_reported = true)

```

---

<div class="post-metadata">

**Author:** ![rikh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rikh/32/204104_2.png) [@rikh](https://discourse.julialang.org/u/rikh)\
**Post date:** [June 29, 2021, 8:49pm UTC](https://discourse.julialang.org/t/parse-dont-validate/63618/7 "2021-06-29T20:49:01Z")

</div>

Thank you both for your thoughts and Cameron for your great examples!

I’ve been thinking about it some more and don’t see much benefit (but I am open to be convinced otherwise). In essence, the point of parse don’t validate, as I understand, is to get feedback more quickly. In a perfect situation, syntax highlighting would give an error like in your last example, which is much quicker than compiling, say, Python and seeing the output. However, given that Julia has a quick evaluation going on with Revise, I doubt that the efforts put in properly using types is worth it. But, I might, of course, be completely wrong

---

<div class="post-metadata">

**Author:** ![jzr](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@jzr](https://discourse.julialang.org/u/jzr)\
**Post date:** [June 29, 2021, 9:18pm UTC](https://discourse.julialang.org/t/parse-dont-validate/63618/8 "2021-06-29T21:18:05Z")

</div>

That is only part of the benefit.

- dispatch on very specific properties of objects, good for performance: [“Algorithm efficiency comes from problem information”](https://www.stochasticlifestyle.com/algorithm-efficiency-comes-problem-information/)
- no redundant checks, because problem information is embedded in the type: If I parse an input into a NonEmptyVector{Int64}, then I don’t have to keep checking the empty case whenever I do something with it. I can just dispatch on the type. Good for performance.
- no redundant checks (less code) good for readability
- very specific types in method signatures, good for readability
- if I accidentally change or delete a necessary prerequisite check, the program will error loudly instead of silently assuming that I already checked a property and returning incorrect results

---

<div class="post-metadata">

**Author:** ![jessymilare](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/jessymilare/32/13750_2.png) [@jessymilare](https://discourse.julialang.org/u/jessymilare)\
**Post date:** [June 30, 2021, 12:57am UTC](https://discourse.julialang.org/t/parse-dont-validate/63618/9 "2021-06-30T00:57:24Z")

</div>

Another example: strings. Package `StrBase.jl` contains validated string types. That is handy because you don’t need to manually handle invalid byte codes, which also makes it faster in most cases.

---

<div class="post-metadata">

**Author:** ![rikh](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rikh/32/204104_2.png) [@rikh](https://discourse.julialang.org/u/rikh)\
**Post date:** [June 30, 2021, 7:49am UTC](https://discourse.julialang.org/t/parse-dont-validate/63618/10 "2021-06-30T07:49:52Z")

</div>

Ok, I’m getting more convinced. So, if you would do this for DataFrames, one would do the following? Define types such as `NonMissing{DataFrame}`, `Sorted{DataFrame}` and overload all kinds of methods to handle these new types such as

```julia
vcat(a::NonMissing{DataFrame}, b::DataFrame) = vcat(DataFrame(a), b)
vcat(a::NonMissing{DataFrame}, b::NonMissing{DataFrame}) = NonMissing{DataFrame}(vcat(a, b))

filter(f, df::NonMissing{DataFrame}) = NonMissing{DataFrame}(filter(f, df::DataFrame))

[...]

```

---

<div class="post-metadata">

**Author:** ![jzr](https://avatars.discourse-cdn.com/v4/letter/j/eb9ed0/32.png) [@jzr](https://discourse.julialang.org/u/jzr)\
**Post date:** [June 30, 2021, 5:59pm UTC](https://discourse.julialang.org/t/parse-dont-validate/63618/11 "2021-06-30T17:59:21Z")

</div>

That seems like a feasible strategy. That said, it looks like AbstractDataFrame has a [pretty large API](https://dataframes.juliadata.org/stable/lib/types/#DataFrames.AbstractDataFrame) that’s specific to that type. It might be easier to implement this on the smaller Tables.jl API with SplitApplyCombine.jl, and maybe using one of the other table types.

---

<div class="post-metadata">

**Author:** ![aplavin](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/aplavin/32/222056_2.png) [@aplavin](https://discourse.julialang.org/u/aplavin)\
**Post date:** [June 30, 2021, 7:36pm UTC](https://discourse.julialang.org/t/parse-dont-validate/63618/12 "2021-06-30T19:36:28Z")

</div>

Problem-specific types are often useful in this regard: e.g., not a `Table`, but a `MeasurementList`, or `Users`. Convert from a plain `Table` immediately after reading from a file, potentially throwing errors in the process, and only use the “parsed” value afterwards.