Did Julia community do something to improve its correctness?

[CameronBieganek]

@Sukera

My point is that no interface should ever have the notion of “optionality”; as soon as you’re talking about that, you have a larger interface that ought to have its own name, as well as being able to restrict dispatch on its own.

I agree with the general philosophy of RequiredInterfaces.jl that interfaces ought to be represented by abstract types, and more specific interfaces (with additional methods) are represented by subtypes. But, echoing @gdalle, it seems hard to put it into practice if we don’t have multiple inheritance.

Also, there are also some difficulties in 100% following that philosophy, even if we had multiple inheritance. Consider this function:

foo(x::Any) = convert(Int, x)

To 100% adhere to the RequiredInterfaces.jl philosophy, we would need to replace the argument annotation for x with an abstract type ConvertibleToInt. And we would also need ConvertibleToString, ConvertibleToFloat64, etc for other functions like foo. Though I suppose it could be parametric, e.g. ConvertibleTo{Int}.

Maybe I’m overinterpreting the philosophy in About Interfaces and you’re ok with not requiring argument annotations to be 100% correct all the time (in the ideal world where we have multiple inheritance).

(Do Haskell and Rust have type classes or traits for convertibility?)

[CameronBieganek]

@Mason

inherits all the requirements of AbstractArray

Of course Base itself has optional methods in the AbstractArray interface, so subtyping AbstractArray is not sufficient to communicate to the consumer which methods have been implemented…

[Mason]

Which is part of why AbstractArray is a bad interface

[CameronBieganek]

I sort of feel like solving our interfaces issue is more important than having small binaries…

[CameronBieganek]

It would be kind of cool if I could annotate a function argument like this,

foo(x::Iterable{Iterable{Table{Int,String}}} = ...

which expects an iterator of iterators of tables.

[Sukera]

@CameronBieganek

Maybe I’m overinterpreting the philosophy in About Interfaces and you’re ok with not requiring argument annotations to be 100% correct all the time (in the ideal world where we have multiple inheritance).

Right, requiring those annotations would make Julia squarely statically typed. That’s not really where I want to take this - I’m fine with having the declared (or rather, dispatched on) signature be wider than the signature you’d have to write if you wanted to encompass all possible requirements (though it should still be possible to do so, if one wishes). It’s nevertheless important to know this ConvertibleTo{Int} requirement for a user of foo, because that’s what is actually required for the call to succeed (in a sense, that ought to be something the compiler can infer for you though, allowing you/an external tool to check the “inferred requirements” against the “required guarantees” of the declared signature).

And you’re also right that the model doesn’t quite work unless there is some form of multiple abstract subtyping; the same is true of any traits based solution though, in particular when it comes to modifying code/adding new traits to an interface. You don’t want to end up in a situation where some user of e.g. the GraphInterface type from above (apologies for highjacking the example, it’s just really illustrative) extends the interface with a new subinterface (resulting in a new trait), passes that into an existing function and have it break due to the existing function having if/elseif based traitdispatch and not knowing about the new trait. There are solutions to that of course (introducing an explicit trait function dispatch layer), but they’re really unwieldy if the function hasn’t been written with that in mind (in some situations impossible without a refactor of the existing code), and on top of that make following/implementing the interface more difficult because you now also have to worry about this additional meta-interface for your interface. That’s super complicated, and should already have been handled by regular dispatch!

@CameronBieganek

(Do Haskell and Rust have type classes or traits for convertibility?)

Good question! I can’t think of them OTOH, but that means nothing of course. There’s a whole world of implicit requirements of arbitrary functions that you could express as a type requirement, but neither regular Haskell nor Rust have that. If you want to look into it, any literature on bidirectional typing, contracts, pre-/postconditions or such is bound to be plentiful.

[mstewart]

@CameronBieganek

(Do Haskell and Rust have type classes or traits for convertibility?)

I don’t know Rust. For Haskell, the answer is “no.” It’s not impossible to have a type class for fully arbitrary conversions, but it definitely would go against the grain of idiomatic Haskell and would be a pain to use. If you wanted a general conversion type class it might look something like

class Convert a b where
  convert :: a -> b

By itself this breaks type inference. Neither a nor b determine the other type. So you would have to put type annotations in appropriate places. To get type inference working again, multiparameter type classes like this usually include functional dependencies establishing that the type a determines the type b (or b determines a) or use type families which are type level functions that can compute the return type:

class Convert a where
  convert :: a -> (Converted a)

Those things make sense for some type classes, but not conversion in which you might want to convert a value of type a into values of several different other types. Conversion functions are typically attached to more specific type classes and with more limited polymorphism. And none of this is in the Haskell 2010 standard, although at this point Haskell 2010 + common ghc extensions is what most people actually use.

I had some personal LAPACK/BLAS bindings for Haskell that I used for some years and early on in that period I actually did write a type class like the above at some point. After using it for a while I realized it wasn’t so nice to use and I had made a newbie mistake in considering it.

I do miss having type classes clearly defining every interface and even having the compiler infer and tell me which interfaces I used. But you have to add in a lot of ghc extensions to try to get even close to the level of flexibility you have in Julia. And while ghc can product fast code, some of the extensions that give more type flexibility break optimizations. Struggling with that was what actually got me to try Julia. Tracking down type instabilities is easier.

[Raf]

I’m slowly putting together the requirements of the main base interfaces here if anyone is interested:

add some more interfaces to BaseInterfaces.jl

They will be imperfect but its worth trying to formalise this mess. Buying in to these interfaces would provide some of the traits we are currently missing.

[CameronBieganek]

For those interested, I’ve opened a separate thread about the feasibility of adding multiple inheritance to Julia:

Feasibility of adding multiple inheritance to Julia

The topic of interfaces came up again in one of the correctness threads. My minimally informed opinion is that the best solution would be to add multiple inheritance to the language. How feasible would it be to add multiple inheritance to Julia? I’m sure it would be quite a bit of work, but my question is not about the amount of work required. What I am wondering is how feasible it is to add multiple inheritance from a language design perspective.

[Ajay_Shah]

As part of our work on improving GLM, one piece that we did was to utilise a group of test cases from NIST, GitHub - xKDR/NISTTests.jl: Measuring the precision of simple statistical calculations

I’m not sure there should be a grand big separate testing project; there should be better engineering at the level of each project.

[Palli]

@ParadaCarleton

I often encounter problems with @inbounds

There’s a “fix” for that, I call it the “Java solution”: You run (or don’t use them locally):

julia --check-bounds=yes

Yes, it doesn’t fix broken code, but it prevents work results. You are never supposed to access out of bounds, so this should hurt work for all code, only maybe hurt performance. It’s great for development and testing.

If you don’t use OffsetArrays.jl you are not likely to need this, and even if you shouldn’t be it could expose edge-case bugs in the ecosystem. It does not ensure you’re immune to bugs if you do without this, but if you test and it works, should give you some confidence in the code you run.

Actually @inbounds may be going away, it’s been discussed that --check-bounds=no will be disabled in the future (since it can actually slow down), and by the same logic I think @inbounds will at some point be made a no-op too. There are ways to get full speed (no emitted bounds check code) with out this already. And when that’s not possible, I believe this is (always?) a potential security risk. I mean if you index by calculated amount from user or file input, then would be a risk and should never be done. If you use @inbounds then, it’s on you a bug in your code. Same as in C/C++ and the fast libraries that e.g. Python relies on, built on such.

[For some strange reason, I was searching for your post (I intended to answer sooner) and it didn’t work. Searching for inbounds found “your” post #151 (which is strange, I was just adding it) not this #106 that I’m seemingly responding to. Seems I found a bug in the (not written in Julia!) Discourse software. Could this be explained by chunked of the thread split off? Also searching for @inbounds which is probably intentionally done by the software (originally not made for Julua), but not the best for Julia…]

[Palli]

@ChrisRackauckas

I agree JuliaStats has many issues, but “Julia” is not just “JuliaStats”.

Right. Hopefully it will get fixed. If it or some package is a known problem for 4 years, then simply don’t use it (or part of it)? Or help fix it? If you complain, “but I need to do statistics”, then no, not with Julia, there are other options.

I’m thinking, in this case specifically. Let’s say Python is better for all of statistics, or some parts, regarding bugs or missing features, then you can simply use it? And I’m not saying abandon Julia, but how problematic is it in practice to use Python for just that or whatever bad package there is, with PythonCall.jl? Is it even slower?

@PetrKryslUCSD

don’t take it the wrong way, but what exactly are you expecting to gain here when all your needs are met by Python and R?

Exactly. And I’m not saying we don’t need packages fixed in Julia, and shouldn’t prefer Julia as the superior language, in general. Just that there’s no need for either or.

[sbuercklin]

Re, “Do Haskell and rust have type classes or traits for convertibility?”

and

Good question! I can’t think of them OTOH, but that means nothing of course. There’s a whole world of implicit requirements of arbitrary functions that you could express as a type requirement, but neither regular Haskell nor Rust have that. If you want to look into it, any literature on bidirectional typing, contracts, pre-/postconditions or such is bound to be plentiful.

I might be misunderstanding your point here, but if I understand correctly this absolutely does exist in Rust. The following code will refuse to compile if you don’t tell the compiler how to convert Baz into i32, and the Rust tooling can infer this at write-time. This can’t be applied directly to the types, but in Rust you don’t really work with types; you bound things on the traits that you need, and you implement these traits to arrive at the behavior you need.

struct Baz {}

impl From<Baz> for i32 {
    fn from(_b: Baz) -> Self {
        return 1
    }
}

fn test_fn(b: Baz) -> i32 {
    return b.into() // Compiler infers it needs to convert into i32 from return type
}

So no, it’s not technically a requirement of a type I implement, but rather a requirement of traits that are necessary to implement IF I want to use Baz in such a way that I convert it to an int. That, in my mind, captures what we actually need: the specific behaviors for the application at hand.

Traits in Rust allow you to specify default implementations for certain trait functions, as well as mandatory methods to implement to support a trait. The tooling around the language is very good at inferring these implementations/requirements and it is the biggest breath of fresh air I’ve gotten coming from Julia.

[Sukera]

@sbuercklin

I might be misunderstanding your point here, but if I understand correctly this absolutely does exist in Rust.

Yes, I think you misunderstood - I was referring to “expressing implicit requirements” that neither Haskell nor Rust has, not a conversion trait/typeclass (which the From trait indeed fits exactly). You can require a certain trait in a function signature in Rust, but that doesn’t automatically require all possible implicit requirements that may throw an error later on, once a runtime value is passed in & checked. As far as I’m aware, that is limited to dependent type systems so far.

[Palli]

@Sukera

The larger point of “has the community overall done something to address this (in a systemic manner)?”

This seems helpful, but I’m not sure. There is no “community overall” overall? It’s not one ecosystem, rather many or many individual packages. Yes, many shared/reused by others. I’m thinking what could actually be done in a “systemic manner”? In open source that is, rather closed/for a company, e.g. for MATLAB.

What does it mean that it needs to be systematic? Do you mean some tools need improving, or Julia as a language needs to be better? I think we could actually take some good ideas from Mojo (regarding speed/freeing early), from Clojure (persistent data structures, basically already available; do more immutability, though not all users want to be restricted to that/pure functional), from Rust (borrow checker, would need to be opt-in), from Unison (fixing/avoiding SemVer…), but I’m not sure what can be done language-wise without either totally changing the language, or it would be optional features, so would not help ensure fixing all of “the ecosystem”. And isn’t it pretty good already?

All software has bugs, and it’s a process, and not all packages will ever be fixed even mostly.

JavaScript/node.js has most packages of any ecosystem with just over 2.5 million (I doubt all good), not sure what happened there, why did about a million disappear, was it all garbage?

http://www.modulecounts.com/

We have a 3rd of the Clojore/Clojar ecosystem by numbers. A language I believe to be good, and would want to compare with. We have a bit more than 51% of the size of the R/CRAN (for some strange reason it has currently “-1/day” average growth…) and 23% or Perl5/CPAN, that could also be compared, except that nobody can really check that many anyway.

I know how to count Julia packages, have (Ruby) code for it, and opened an issue in case anyone wants to help get the Julia package count n there:

Julia module count doesn't work, here's file for new count

[sbuercklin]

OP was asking about traits for convertibility, which is From. Can you clarify what you mean by implicit requirements?

I generally do not want my language to be implicit. Rust in fact seems to lean incredibly hard on being explicit, to the point where you can require many trait bounds to guarantee that all methods within a function exist and signatures can get quite ugly.

You can implement a generic/dynamic-dispatch with dyn T where T is a set of traits you require everything that slots in, but even so the compiler enforces that all methods called within the function are captured by the (potentially composite) trait T. Likewise, where you call that function will refuse to compile if the trait bounds are not satisfied. In my experience the compiler is rigorous about this.

I would like to understand the class of error you’re describing, but as I’m not an expert on language/compiler design, it seems there’s a fundamental piece here that I am missing on what you mean by implicit requirements.

For a concrete exampe: is it an explicit, implicit, or non-requirement that getindex(::AbstractArray{T}, ::ValidIndexType) return an instance of T? In Julia, this is a non-requirement (the compiler doesn’t enforce this at all). Most people would expect this to be the case, so I would call it an implicit requirement. There may be a more rigorous/domain specific definition of implicit requirement, but it also seems that there are very few explicit requirements in Julia insofar as code will compile and runtime error instead.

[adienes]

@sbuercklin

so I would call it an implicit requirement.

not even, if you are using StaticArrays

[sbuercklin]

The space of AbstractArrays is much larger than StaticArrays. If I write all my methods with StaticArrays in the method signatures, yes, the language tooling is very good about inferring intermediate types and it’s easy to sort out guarantees. The point of AbstractArray is that I write code which works for well behaved arrays (like SA) as well as (what I would call) adversarial arrays, like StarWarsArrays.jl, although they are adversarial in indexing rather than element getindex.

[Sukera]

@sbuercklin

OP was asking about traits for convertibility, which is From. Can you clarify what you mean by implicit requirements?

@sbuercklin

For a concrete exampe: is it an explicit, implicit, or non-requirement that getindex(::AbstractArray{T}, ::ValidIndexType) return an instance of T? In Julia, this is a non-requirement (the compiler doesn’t enforce this at all). Most people would expect this to be the case, so I would call it an implicit requirement. There may be a more rigorous/domain specific definition of implicit requirement, but it also seems that there are very few explicit requirements in Julia insofar as code will compile and runtime error instead.

Yes, your example is one such “implicit requirement” - things we have to/the code has to manually check at some point through some means, because the requirement we need at runtime is not guaranteed by the type. Another implicit requirement is that the second argument is inbounds for your given AbstractArray.

For a perhaps simpler example:

function double(x::Int)
    x < 0 && throw(ArgumentError("$x is not non-negative"))
    return 2*x
end

This function only ever works when x >= 0, which is an implicit requirement of double such that calling it returns without throwing a (runtime) error. Int alone certainly doesn’t guarantee this - UInt would, but perhaps having Int here is intentional due to some other requirement? Also, this function may have some docstring that clarifies this requirement, or it may not; my point is that it would be nice to “extract” these kinds of implicit requirements that are imposed on x automatically, even without being able/having to specify that x is a Int and also that x >= 0 must hold.

This kind of stuff probably computationally very expensive to check for larger programs though, especially when it comes to more complicated functions & interactions Research here is all pretty new…

[Tarny_GG_Channie]

I believe inbound is here to stay, because it’s an opt-in thing that gives you fine-grained control over something when you need. Besides, bound checking might be differently implemented on different types, so there is this inbound. One key aspect of Julia is that Julia is a language that allows users to opt into controlling aspects of the code themselves.