Julia needs a unified industrial standard to guide the API designs.
For example, we have X operator in Pauli algebra. Although the entity is same, it can be defined independently with different names (such as XGate and PauliX) in different packages. When one uses two of such packages at the same time, these packages do not share methods!
Unlike OOD languages, Julia types/methods are designed to be open, methods defined on the same entity should be able to be used at the same time.
Here, I suggest to build an official package AbstractJulia.jl, which includes abstract types and methods from various fields, users can submit PR to make it better.
For example, it contains an abstract type like AbstractPauliX. People can either subtype it or dispatch methods on it, just like extending Base packages.
Here is an example to show how developers can benefit. Cliffords.jl is a small and neat package, but sadly, I won’t use its types for developing my own packages, just because it is small and its Pauli gates do not meet some requirements in my projects. To use Cliffords.jl, I have to make some dirty copies out of it. If some of its methods are defined on standard abstract type tree in AbstractJulia, I can extend AbstractJulia instead without copying.
I am going open a repo for this if JuliaLang is not going to have one, how do you like this idea?
Different packages may implement the same abstract concept differently.
Which is a good thing — the implementations may not (and usually are not) compatible, unless the developers make an effort to do that.
There is no need to do this in a single package. Domain-specific packages can define abstract interfaces, when that is necessary.
If you otherwise like the package and believe it could work for you with minor modifications, the first thing I would consider in this case is opening an issue and then a pull request. Usually maintainers are open to extensions if that is otherwise compatible with the goals of the package.
I don’t know what that means, but making a Git branch could be a good basis for a PR.
I think you are missing important points about open source development and version control in general and Julia packages in particular. Perhaps you should observe how these things are done in Julia for a while. There is no body enforcing or maintaining “industrial standards”, it is loose and friendly cooperation between well-intentioned people, and works fine in practice.
I think this is a good idea, but AbstractJulia is probably too broad a scope for a single package. Maybe better to have one package of abstract types and functions per domain. That way, each package can be maintained by people who are familiar with that domain.
Such packages already exist for several domains. For example, there’s MathOptInterface for optimization solvers, etc.
Maybe you could create a PauliAlgebraInterface abstract package, and see if there’s interest among developers to use it?
Edit: I think in practice, the people who develop the abstract packages are always the same people who develop at least one of the packages that implement the functionality. This is one way to ensure that the abstract types and functions make sense, and it would be very hard for a maintainer of an AbstractJulia package to have that kind of experience.
Something that could be good is an automated split into “header files” and implementation.
The way this would work is that Pkg automatically splits each FooPkg into AbstractFooPkg and ImplementationFooPkg, such that AbstractFooPkg contains all structure and function declarations. If forward declarations are ever implemented then we would just need them, and could forbid eval from declaring new type-names or function-names.
Desired result: Currently many packages are missing a “header file” package; so they become a dependency (and their binary dependencies as well!) when I want to extend them for interoperability. Or the “header files” are incomplete.
If this gets automated, then we will always have the julia analogue of header files, and they will never be incomplete.
Only problem: Sometimes structure definitions and function declarations are procedurally generated and evaled. Also, the way inner constructors currently work is not good for that.
edit: Essential property would be that header files cannot contain executable code, and use of malicious header files is always safe if you do not load the implementation. That’s for people like @anon94023334 who are very conscious of dependencies.
You can decide whether “grow” your types in a “standard” tree or not. Sometimes, we want to grow types on the same tree to avoid chaos. Let’s take AbstractArray type as an example.
In python, the numpy array ecosystem is strong. It is partly because numpy array has become the only standard for scientific programming, people are happy to use numpy array as a standard platform for package development. On the other side, C++ has many Arrays/Matrix types, its libraries are still in chaos. Julia provides an official Array types in Base, which is good for scientific programming. But we should also notice the types in base are limited. Making type tree/methods in Base extensible is what I meant in this post.
Agreed, but I agree more on @Liso 's point of view
The label official is important for unifying conventions. Please also notice this comment
Let’s see Jutho’s package KrylovKit.jl, in the index of its doc, it mentions several packages that inspired his package
I appreciate Jutho’s effort in unifying interfaces. But in this process, it must includes “dirty copying” that making the contributions from authors of original libraries invisible.
You still didn’t tell me what “dirty copying” is.
It is also unclear what it (whatever it is) “makes invisible”. Did other people contribute to this library and remained unrecognized? I get the opposite impression: this seems like a package with a well-designed interface, that was just nice to list other related packages that inspired it, which is good practice.
dirty copy (in this context): copy someone else’s implementations with minor changes, due to the disagree on convention. Here “dirty” emphasizes the side effect of copying codes, which always double the effort for maintaining the piece of code, meanwhile making the contributors unclear.
Jutho’s package for unifying interfaces is a good practice, I acknowledge this is an improper example to illustrate dirty copy, since KrylovKit did much more than changing conventions. In this case, copying is probably not avoidable. But in the example of Clifford algebra, dirty copy can be avoided if people have consensus about types. Although its types are not desired, its implementation is neat. Most importantly, many of its algebras can be defined on abstract types.
People are inspired by code from others. This is a good thing, and is essential to open source. Sometimes common patterns can be abstracted to a single implementation, sometimes this isn’t worth it.
Nothing prevents you from defining a package for some abstract interfaces, registering it, then making PRs to other packages so they will use it.
It does not have to be “official” in any sense; package authors would still need to be convinced to use it, which is the hard part.
This issue has been discussed in multiple threads already, most famously
There are a few issues with the whole function merging thing, one of my solution is the ForceImport package
which helps with selective managing of the local merging of namespaces (at least for Base methods). Now, I’m thinking I could extend this package to work with a specified package, not only Base methods.
Here’s my opinionated view: Julia is not MATLAB, so the community makes “guidelines” instead of “standards”. And good guidelines evolve from the convergence of good practices. In general you shouldn’t expect other people who write their own packages (mostly to solve their own problems) to tailor exactly to your needs, unless you join the conversation, state your needs, and possibly contribute to their packages. But over time, some packages may become the pillars of other packages and their APIs may slowly converge. That said, the currently fragmentary package ecosystem (compared to numpy/scipy) sometimes does make it difficult to interop between packages, but so can be said for a mature (but more fragmentary than Python’s) package ecosystem like R’s.
On the “interface”, I think the abstract function/type example @1115 showed would be easily done in OCaml. The AbstractJulia.jl you proposed would work like a collection of functors to extend the meta-language of Julia. But Julia is Lisp-like rather than ML-like. Given that Julia has a macro system already, is this AbstractJulia.jl really necessary? (more importantly, are these constructs orthogonal?) I don’t think enforcing a standard of API design in the language itself would be widely useful. Even if so, the burden should not be on the language or standard library developers, but on the package developers.
That sounds undemocratic. Could you please elaborate on “industrial standard”, especially on the concept of “industrial”? My stand is, unless necessary, one keep minimal level of standards thereby leaving maximal freedom to individual developers. If two groups of developers want to collaborate, they talk and align their codes. In order to standardize anything, you need to be THE leader of that industry.
We could probably look on python for inspiration. They have PEPs for standardization and PEP 249 – Python Database API Specification v2.0 is example that official standard for interface could be useful.
You could define different database API in python. Nobody will undemocratic enforce you to use that specification. (you could read “Package writers are encouraged to use this version of the specification as basis for new interfaces.”)
Julia’s multiple dispatch and JIT allow/require stronger interconnection between packages.
We have actually some interface standards (AbstractArray, AbstractString for example) where liberty to replace it with your owns is some kind of only theoretical, isn’t it?
I think that constructive discussion about interfaces and about platform which could help define them and spread them could be fine.
In my Julia practice, I spent quite some time in sorting out the abstract types and their dependencies. That task depends on, which I realized later, the understanding of the entire project. Such understanding is personal, and it evolves along with the project as well. I have difficulties even in talking to my past. I wished that I would design the architecture only once and fill up the empty functions later! But that is surreal, at least to me, because I take advices from my colleagues and constantly change the type dependencies. I also need to do significant modifications on types when I am optimizing the code. It was a painful experience to restructure the type-dependence tree. It requires a lot of changes to the function interfaces. I guess these things would also appear when intertwining two packages, and that is perhaps why we need “standards”.
However, I doubt that in my project, if the type-dependency is imposed, or there are restrictions on it, whether I am still able to make progress in that project. The restrictions may wipe out possibilities at very early stage, and as the project goes on, I may never put it back. Thus I oppose to standards which are unrelated to the project a priori.
These are very vague words … but I am not proven an experienced developer, I can’t say anything better than this
