# Industrial standards for Julia packages
**URL:** https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300
**Category:** Internals & Design
**Created:** [October 14, 2018, 5:15am UTC](https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300 "2018-10-14T05:15:02Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@1115](https://discourse.julialang.org/u/1115)
#### Post date: [October 14, 2018, 5:15am UTC](https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300/1 "2018-10-14T05:15:02Z")
Julia needs a unified industrial standard to guide the API designs.
For example, we have `X` operator in Pauli algebra. Although the **entity** is same, it can be defined independently with different names (such as `XGate` and `PauliX`) in different packages. When one uses two of such packages at the same time, these packages do not share methods!
Unlike OOD languages, Julia types/methods are designed to be open, methods defined on the same **entity** should be able to be used at the same time.
Here, I suggest to build an official package `AbstractJulia.jl`, which includes **abstract** types and methods from various fields, users can submit PR to make it better.
For example, it contains an abstract type like `AbstractPauliX`. People can either subtype it or dispatch methods on it, just like extending Base packages.
Here is an example to show how developers can benefit.
[Cliffords.jl](https://github.com/BBN-Q/Cliffords.jl) is a small and neat package, but sadly, I won’t use its types for developing my own packages, just because it is small and its Pauli gates do not meet some requirements in my projects. To use `Cliffords.jl`, I have to make some dirty copies out of it. If some of its methods are defined on standard abstract type tree in `AbstractJulia`, I can extend `AbstractJulia` instead without copying.
I am going open a repo for this if JuliaLang is not going to have one, how do you like this idea?
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [October 14, 2018, 5:45am UTC](https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300/2 "2018-10-14T05:45:43Z")
> [@1115](#):
>
> Although the **entity** is same, it can be defined independently with different names (such as `XGate` and `PauliX` ) in different packages.
Different packages may implement the same abstract concept differently.
> [@1115](#):
>
> When one uses two of such packages at the same time, these packages do not share methods!
Which is a good thing — the implementations may not (and usually are not) compatible, unless the developers make an effort to do that.
> [@1115](#):
>
> official package `AbstractJulia.jl` , which includes **abstract** types and methods from various fields
There is no need to do this in a single package. Domain-specific packages can define abstract interfaces, when that is necessary.
> [@1115](#):
>
> [Cliffords.jl](https://github.com/BBN-Q/Cliffords.jl) is a small and neat package, but sadly, I won’t use its types for developing my own packages, just because it is small and its Pauli gates do not meet some requirements in my projects.
If you otherwise like the package and believe it could work for you with minor modifications, the first thing I would consider in this case is opening an issue and then a pull request. Usually maintainers are open to extensions if that is otherwise compatible with the goals of the package.
> [@1115](#):
>
> I have to make some dirty copies out of it.
I don’t know what that means, but making a Git branch could be a good basis for a PR.
> [@1115](#):
>
> I am going open a repo for this if JuliaLang is not going to have one, how do you like this idea?
I think you are missing important points about open source development and version control in general and Julia packages in particular. Perhaps you should observe how these things are done in Julia for a while. There is no body enforcing or maintaining “industrial standards”, it is loose and friendly cooperation between well-intentioned people, and works fine in practice.
---
### Author:  [@Per](https://discourse.julialang.org/u/Per)
#### Post date: [October 14, 2018, 8:29am UTC](https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300/3 "2018-10-14T08:29:40Z")
I think this is a good idea, but `AbstractJulia` is probably too broad a scope for a single package. Maybe better to have one package of abstract types and functions per domain. That way, each package can be maintained by people who are familiar with that domain.
Such packages already exist for several domains. For example, there’s `MathOptInterface` for optimization solvers, etc.
Maybe you could create a `PauliAlgebraInterface` abstract package, and see if there’s interest among developers to use it?
Edit: I think in practice, the people who develop the abstract packages are always the same people who develop at least one of the packages that implement the functionality. This is one way to ensure that the abstract types and functions make sense, and it would be very hard for a maintainer of an `AbstractJulia` package to have that kind of experience.
---
### Author:  [@johnh](https://discourse.julialang.org/u/johnh)
#### Post date: [October 14, 2018, 8:37am UTC](https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300/4 "2018-10-14T08:37:13Z")
I know I am showing my ignorance but what is an abstract interface?
Please explain for a bear of little brain.
---
### Author:  [@Liso](https://discourse.julialang.org/u/Liso)
#### Post date: [October 14, 2018, 9:30am UTC](https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300/5 "2018-10-14T09:30:03Z")
> [@Per](#):
>
> Such packages already exist for several domains. For example, there’s `MathOptInterface` for optimization solvers, etc.
>
> Maybe you could create a `PauliAlgebraInterface` abstract package, and see if there’s interest among developers to use it?
`AbstractJulia` could be list for these interfaces…
Some notes:
If MOI has been designed to replace [MathProgBase](https://github.com/JuliaOpt/MathProgBase.jl) (see: [http://www.juliaopt.org/MathOptInterface.jl/latest/apimanual.html](http://www.juliaopt.org/MathOptInterface.jl/latest/apimanual.html)), then it seems it is difficult evolve such interfaces and revolution is sometimes simpler.
If I understand it well there are two industry “standards” for optimization solvers interface.
I like idea to help create strong ecosystem with supporting and spreading interfaces. Some kind of registration for these interfaces could be good! 🙂
So maybe you ( @1115 ) could start with repository (something similar what [GitHub - svaksha/Julia.jl: Curated decibans of Julia programming language.](https://github.com/svaksha/Julia.jl) is doing (or help them to maintain decibans))
I think we could start without standardization committee in this moment. 😉
---
### Author:  [@foobar\_lv2](https://discourse.julialang.org/u/foobar_lv2)
#### Post date: [October 14, 2018, 2:08pm UTC](https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300/6 "2018-10-14T14:08:11Z")
Something that could be good is an automated split into “header files” and implementation.
The way this would work is that `Pkg` automatically splits each `FooPkg` into `AbstractFooPkg` and `ImplementationFooPkg`, such that `AbstractFooPkg` contains all structure and function declarations. If forward declarations are ever implemented then we would just need them, and could forbid eval from declaring new type-names or function-names.
Desired result: Currently many packages are missing a “header file” package; so they become a dependency (and their binary dependencies as well!) when I want to extend them for interoperability. Or the “header files” are incomplete.
If this gets automated, then we will always have the julia analogue of header files, and they will never be incomplete.
Only problem: Sometimes structure definitions and function declarations are procedurally generated and evaled. Also, the way inner constructors currently work is not good for that.
edit: Essential property would be that header files cannot contain executable code, and use of malicious header files is always safe if you do not load the implementation. That’s for people like @anon94023334 who are very conscious of dependencies.
---
### Author:  [@1115](https://discourse.julialang.org/u/1115)
#### Post date: [October 14, 2018, 2:08pm UTC](https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300/7 "2018-10-14T14:08:37Z")
Thanks for your comments.
> [@Tamas\_Papp](#):
>
> Different packages may implement the same abstract concept differently.
>
> There is no body enforcing or maintaining “industrial standards”, it is loose and friendly cooperation between well-intentioned people, and works fine in practice.
You can decide whether “grow” your types in a “standard” tree or not. Sometimes, we want to grow types on the same tree to avoid chaos. Let’s take `AbstractArray` type as an example.
In python, the `numpy array` ecosystem is strong. It is partly because numpy array has become the only standard for scientific programming, people are happy to use numpy array as a standard platform for package development. On the other side, `C++` has many `Arrays/Matrix` types, its libraries are still in chaos. Julia provides an official `Array` types in `Base`, which is good for scientific programming. But we should also notice the types in base are limited. Making type tree/methods in Base extensible is what I meant in this post.
> [@Tamas\_Papp](#):
>
> There is no need to do this in a single package. Domain-specific packages can define abstract interfaces, when that is necessary.
Agreed, but I agree more on @Liso 's point of view
> [@Liso](#):
>
> `AbstractJulia` could be list for these interfaces…
The label `official` is important for unifying conventions. Please also notice this comment
> [@Per](#):
>
> Edit: I think in practice, the people who develop the abstract packages are always the same people who develop at least one of the packages that implement the functionality. This is one way to ensure that the abstract types and functions make sense, and it would be very hard for a maintainer of an `AbstractJulia` package to have that kind of experience.
> [@Tamas\_Papp](#):
>
> > I have to make some dirty copies out of it.
>
> I don’t know what that means, but making a Git branch could be a good basis for a PR.
Let’s see Jutho’s package [KrylovKit.jl](https://github.com/Jutho/KrylovKit.jl), in the index of its doc, it mentions several packages that inspired his package
>
I appreciate Jutho’s effort in unifying interfaces. But in this process, it must includes “dirty copying” that making the contributions from authors of original libraries invisible.
---
### Author:  [@1115](https://discourse.julialang.org/u/1115)
#### Post date: [October 14, 2018, 2:22pm UTC](https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300/8 "2018-10-14T14:22:30Z")
Interfaces are always abstract, so abstract interface here means interface…
e.g.
### Abstract Function
```julia
""""
foo(x) -> Vector
Illustrate interface.
""""
function foo end
```
### Abstract Types
```julia-auto
"""
PauliX
Abstract Pauli X interface.
"""
abstract type PauliX end
```
In summary, it contains `Labels` for abstract function/types, and documentation.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [October 14, 2018, 2:48pm UTC](https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300/9 "2018-10-14T14:48:19Z")
> [@1115](#):
>
> I appreciate Jutho’s effort in unifying interfaces. But in this process, it must includes “dirty copying” that making the contributions from authors of original libraries invisible.
You still didn’t tell me what “dirty copying” is.
It is also unclear what it (whatever it is) “makes invisible”. Did other people contribute to this library and remained unrecognized? I get the opposite impression: this seems like a package with a well-designed interface, that was just nice to list other related packages that inspired it, which is good practice.
---
### Author:  [@tamasgal](https://discourse.julialang.org/u/tamasgal)
#### Post date: [October 14, 2018, 2:49pm UTC](https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300/10 "2018-10-14T14:49:40Z")
> [@1115](#):
>
> I am going open a repo for this if JuliaLang is not going to have one, how do you like this idea?
---
### Author:  [@1115](https://discourse.julialang.org/u/1115)
#### Post date: [October 14, 2018, 3:27pm UTC](https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300/11 "2018-10-14T15:27:59Z")
`dirty copy` (in this context): copy someone else’s implementations with minor changes, due to the disagree on convention. Here “dirty” emphasizes the side effect of copying codes, which always double the effort for maintaining the piece of code, meanwhile making the contributors unclear.
Jutho’s package for unifying interfaces is a good practice, I acknowledge this is an improper example to illustrate dirty copy, since `KrylovKit` did much more than changing conventions. In this case, copying is probably not avoidable. But in the example of Clifford algebra, `dirty copy` can be avoided if people have consensus about types. Although its types are not desired, its implementation is neat. Most importantly, many of its algebras can be defined on abstract types.
---
### Author:  [@Tamas\_Papp](https://discourse.julialang.org/u/Tamas_Papp)
#### Post date: [October 14, 2018, 3:48pm UTC](https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300/12 "2018-10-14T15:48:00Z")
> [@1115](#):
>
> `dirty copy` can be avoided if people have consensus about types
People are inspired by code from others. This is a good thing, and is essential to open source. Sometimes common patterns can be abstracted to a single implementation, sometimes this isn’t worth it.
Nothing prevents you from defining a package for some abstract interfaces, registering it, then making PRs to other packages so they will use it.
It does not have to be “official” in any sense; package authors would still need to be convinced to use it, which is the hard part.
---
### Author:  [@chakravala](https://discourse.julialang.org/u/chakravala)
#### Post date: [October 14, 2018, 5:59pm UTC](https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300/13 "2018-10-14T17:59:38Z")
This issue has been discussed in multiple threads already, most famously
> [@Function name conflict: ADL / function merging?](https://discourse.julialang.org/t/function-name-conflict-adl-function-merging/10335):
>
> Is there any reason why Julia needs me to fully qualify the function name when the argument types are clear and there shouldn’t be any problem with multiple dispatch? Case 1: load JuliaDB before defining Foo julia\> using JuliaDB julia\> stack( stack(t::D) where D\<:Union{IndexedTables.NDSparse, IndexedTables.NextTable} in IndexedTables at /Users/tomkwong/.julia/v0.6/IndexedTables/src/reshape.jl:32 stack(t::D, by; select, variable, value) where D\<:Union{IndexedTables.NDSparse, IndexedTables.Next…
There are a few issues with the whole function merging thing, one of my solution is the `ForceImport` package
> **[GitHub - chakravala/ForceImport.jl: Macro that force imports conflicting...](https://github.com/chakravala/ForceImport.jl)**
>
> Macro that force imports conflicting methods in modules - GitHub - chakravala/ForceImport.jl: Macro that force imports conflicting methods in modules
which helps with selective managing of the local merging of namespaces (at least for `Base` methods). Now, I’m thinking I could extend this package to work with a specified package, not only `Base` methods.
Here is an explanation of how it is used
> [@Operations on expressions](https://discourse.julialang.org/t/operations-on-expressions/13666/12):
>
> The principle which all @force users must keep in mind is this: the goal is to extend Base methods locally without affecting the global method table by type piracy, and to be able to import them into another package to have the same local effect. In order to avoid the type piracy, one must define a new local complementary n-ary method that will fall back on the base method with Any arguments. Then there will be a tiered alternative dispatch layer within the local package scope that redirects to …
---
### Author:  [@geoalchimista](https://discourse.julialang.org/u/geoalchimista)
#### Post date: [October 16, 2018, 5:38am UTC](https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300/14 "2018-10-16T05:38:50Z")
Here’s my opinionated view: Julia is not MATLAB, so the community makes “guidelines” instead of “standards”. And good guidelines evolve from the convergence of good practices. In general you shouldn’t expect other people who write their own packages (mostly to solve their own problems) to tailor exactly to your needs, unless you join the conversation, state your needs, and possibly contribute to their packages. But over time, some packages may become the pillars of other packages and their APIs may slowly converge. That said, the currently fragmentary package ecosystem (compared to numpy/scipy) sometimes does make it difficult to interop between packages, but so can be said for a mature (but more fragmentary than Python’s) package ecosystem like R’s.
On the “interface”, I think the abstract function/type example @1115 showed would be easily done in OCaml. The `AbstractJulia.jl` you proposed would work like a collection of functors to extend the meta-language of Julia. But Julia is Lisp-like rather than ML-like. Given that Julia has a macro system already, is this `AbstractJulia.jl` really necessary? (more importantly, are these constructs _orthogonal_?) I don’t think enforcing a standard of API design in the language itself would be widely useful. Even if so, the burden should not be on the language or standard library developers, but on the package developers.
---
### Author:  [@Lian\_Yunlong](https://discourse.julialang.org/u/Lian_Yunlong)
#### Post date: [October 16, 2018, 2:20pm UTC](https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300/15 "2018-10-16T14:20:28Z")
That sounds undemocratic. Could you please elaborate on “industrial standard”, especially on the concept of “industrial”? My stand is, unless necessary, one keep minimal level of standards thereby leaving maximal freedom to individual developers. If two groups of developers want to collaborate, they talk and align their codes. In order to standardize anything, you need to be THE leader of that industry.
---
### Author:  [@Lian\_Yunlong](https://discourse.julialang.org/u/Lian_Yunlong)
#### Post date: [October 16, 2018, 2:23pm UTC](https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300/16 "2018-10-16T14:23:03Z")
A good example of democracy is a throughout discussion like this;
[Taking transposes seriously](https://github.com/JuliaLang/julia/issues/4774)
---
### Author:  [@Liso](https://discourse.julialang.org/u/Liso)
#### Post date: [October 16, 2018, 2:58pm UTC](https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300/17 "2018-10-16T14:58:05Z")
We could probably look on python for inspiration. They have PEPs for standardization and [PEP 249 – Python Database API Specification v2.0](https://www.python.org/dev/peps/pep-0249/) is example that official standard for interface could be useful.
You could define different database API in python. Nobody will undemocratic enforce you to use that specification. (you could read “_Package writers are encouraged to use this version of the specification as basis for new interfaces._”)
Julia’s multiple dispatch and JIT allow/require stronger interconnection between packages.
We have actually some interface standards (AbstractArray, AbstractString for example) where liberty to replace it with your owns is some kind of only theoretical, isn’t it?
I think that constructive discussion about interfaces and about platform which could help define them and spread them could be fine.
---
### Author:  [@1115](https://discourse.julialang.org/u/1115)
#### Post date: [October 16, 2018, 3:09pm UTC](https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300/18 "2018-10-16T15:09:25Z")
Good suggestion, building a platform for discussion & making standards is much more practical than my `AbstractJulia` proposal.
> [@Liso](#):
>
> Julia’s multiple dispatch and JIT allow/require stronger interconnection between packages.
Also, this is a proper summary for what I want to emphasis, and is also the point that many people miss.
---
### Author:  [@StefanKarpinski](https://discourse.julialang.org/u/StefanKarpinski)
#### Post date: [October 16, 2018, 3:36pm UTC](https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300/19 "2018-10-16T15:36:57Z")
> [@Liso](#):
>
> We could probably look on python for inspiration.
Perhaps a good model but certainly not a particularly democratic one:
> **[Benevolent dictator for life](https://en.wikipedia.org/wiki/Benevolent_dictator_for_life)**
>
> Benevolent dictator for life (BDFL) is a title given to a small number of open-source software development leaders, typically project founders who retain the final say in disputes or arguments within the community. The phrase originated in 1995 with reference to Guido van Rossum, creator of the Python programming language.
> Shortly after Van Rossum joined the Corporation for National Research Initiatives, the term appeared in a follow-up mail by Ken Manheimer to a meeting trying to create a semi-...
---
### Author:  [@Lian\_Yunlong](https://discourse.julialang.org/u/Lian_Yunlong)
#### Post date: [October 16, 2018, 3:57pm UTC](https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300/20 "2018-10-16T15:57:14Z")
In my Julia practice, I spent quite some time in sorting out the abstract types and their dependencies. That task depends on, which I realized later, the understanding of the entire project. Such understanding is personal, and it evolves along with the project as well. I have difficulties even in talking to my past. I wished that I would design the architecture only once and fill up the empty functions later! But that is surreal, at least to me, because I take advices from my colleagues and constantly change the type dependencies. I also need to do significant modifications on types when I am optimizing the code. It was a painful experience to restructure the type-dependence tree. It requires a lot of changes to the function interfaces. I guess these things would also appear when intertwining two packages, and that is perhaps why we need “standards”.
However, I doubt that in my project, if the type-dependency is imposed, or there are restrictions on it, whether I am still able to make progress in that project. The restrictions may wipe out possibilities at very early stage, and as the project goes on, I may never put it back. Thus I oppose to standards which are unrelated to the project a priori.
These are very vague words … but I am not proven an experienced developer, I can’t say anything better than this
[Next page](https://discourse.julialang.org/t/industrial-standards-for-julia-packages/16300.md?page=2)