The very fruitful discussion on fixing package fragmentation started from a remark I made on Slack about graphs, which is why I’m asking the community for ideas and support.
In my opinion, the most important things we can do (by order of priority) are:
- List existing packages to allow comparison
- Redesign the core interface to make it more usable
- Ensure reliability of the main packages for the long term
What follows is a brief summary of each item, but feel free to suggest more! Sorry in advance for the long post.
1. Existing packages
At this stage, the JuliaGraphs organization is recognized as the main place to look for graph-related Julia code. But that doesn’t mean it is the only place you can find such code! Incoming users are often confused about the competing standards, and there are easy fixes.
My proposal would be to add external packages to the JuliaGraphs website, so I’ve started a list. Please help me complete it, and tag your friends who develop graphs packages in secret!
For the purposes of this list, I’m excluding packages related to reading / writing / plotting graphs, as well as optimization algorithms, to concentrate only on data structures.
- “Organization”: does it belong to a GitHub organization?
- “Maintainer”: who is listed first on the GitHub contributors list?
- “Focus”: what does the package do?
- “Comp(atible)”: does it have Graphs.jl in its dependencies?
- “Reg(istered)”: does it belong to the general registry?
- “Act(ive)”: has there been at least one commit to main in 2023 (I know it’s arbitrary)?
Last update of the list: 31-05-2023
|SimpleGraphConverter.jl||-||scheinerman||Conversion to and from Graphs.jl|
|Laplacians.jl||-||danspielman||Graphs as sparse matrices|
|MetaGraphsNext.jl||JuliaGraphs||gdalle||Vertex labels, vertex and edge metadata|
|MetaGraphs.jl||JuliaGraphs||mbesancon||Vertex labels, vertex and edge metadata|
|AttributeGraphs.jl||UniStuttgart-IKR||filchristou||Vertex and edge metadata|
|DataGraphs.jl||-||mtfishman||Vertex and edge metadata|
|MetaDataGraphs.jl||-||gdalle||Vertex labels, vertex and edge metadata|
|SimpleValueGraphs.jl||-||simonschoelly||Vertex and edge metadata|
|Erdos.jl||-||CarloLucibello||Vertex and edge metadata|
|Graft.jl||-||Pranav Thulasiram Bhat||Vertex and edge metadata|
|StorageGraphs.jl||-||SebastianM-C||Vertex and edge metadata|
|LabelledGraphs.jl||IITiS PAN||Konrad Jałowiecki||Vertex labels, vertex and edge metadata|
|StenoGraphs.jl||-||aaronpeikert||Vertex and edge metadata|
|SMDGraphs.jl||JuliaSpaceMissionDesign||Michele Ceresoli||Vertex labels / metadata|
|SpatialGraphs.jl||Circuitscape||vlandau||Spatial locations for vertices|
|EmbeddedGraphs.jl||PIK-ICoNe||Paul Schultz||Graphs in 2D spaces|
|AtomGraphs.jl||Chemellia||rkurchin||Molecules as graphs|
|TextGraphs.jl||-||fargolo||Text as graphs|
|Multigraphs.jl||QuantumBFS||ChenZhao44||Edges with multiplicity|
|WrappedMultiGraphs.jl||UniStuttgart-IKR||filchristou||Edges with multiplicity|
|HyperGraphs.jl||-||Léo Diaz||Graphs with higher-order edges|
|SimpleHypergraphs.jl||-||pszufe||Graphs with higher-order edges|
|MultiLayerGraphs.jl||JuliaGraphs||InPhyT||Several stacked layers|
|SimpleGraphRepresentations.jl||-||scheinerman||Graphs with specific structure|
|SpecialGraphs.jl||JuliaGraphs||mbesancon||Structure encoded in type|
|DirectedAcyclicGraphs.jl||-||guyvdb||Directed graphs without loops|
|PeriodicGraphs.jl||-||Liozou||Repeating graph structures|
|GridGraphs.jl||-||gdalle||Graphs on a 2D grid|
Efficiency / safety
|StaticGraphs.jl||JuliaGraphs||Seth Bromberger||Immutable graphs|
|ImplicitGraphs.jl||-||scheinerman||Dynamically generated graphs|
|BoundedDegreeGraphs.jl||-||SteffenPL||No allocations with limited neighbors|
|VertexSafeGraphs.jl||-||mbesancon||Stable vertex indices|
|SuiteSparseGraphBLAS.jl||JuliaSparse||Wimmerer||Linear algebra for graphs|
|MatrixNetworks.jl||JuliaGraphs||nassarhuda||Graphs as matrices|
|LightGraphs.jl||-||Seth Bromberger||Predecessor of Graphs.jl|
|OldGraphs.jl||JuliaAttic||Dahua Lin||Old package with the same name as Graphs.jl|
A few remarks
On the positive side:
- I’m amazed at the productivity and creativity of this community
- A sizeable portion abides by the Graphs.jl interface
On the negative side:
- There are plenty of registered but inactive packages, which means good discoverable names go to waste
- The proliferation of standards, especially for metadata, is concerning
2. Core interface
If we want the Graphs.jl interface to be even more widely accepted, we have to make it better. This starts with a more generic approach, which would make it easier to handle metadata or multigraphs. The question of vertex and edge indexing is therefore central.
You’re more than welcome to contribute to the discussion, either here or on GitHub.
A few relevant issues
- Common interface for graphs with metadata · Issue #35 · JuliaGraphs/Graphs.jl · GitHub
- [Port] Proposal: relax restriction on vertex IDs · Issue #57 · JuliaGraphs/Graphs.jl · GitHub
- Implicit graph structure · Issue #109 · JuliaGraphs/Graphs.jl · GitHub
- Working with edge indices · Issue #127 · JuliaGraphs/Graphs.jl · GitHub
- Roadmap for Graphs.jl 2.0 · Issue #128 · JuliaGraphs/Graphs.jl · GitHub
- Split out a GraphsBase package? · Issue #135 · JuliaGraphs/Graphs.jl · GitHub
- Discussion : API for Graphs.jl 2.0 · Issue #146 · JuliaGraphs/Graphs.jl · GitHub
- Proposals for Graphs.jl 2.0 from the perspective of MultilayerGraphs.jl · Issue #165 · JuliaGraphs/Graphs.jl · GitHub
- Vertices should have consistent indices during their lifetime · Issue #180 · JuliaGraphs/Graphs.jl · GitHub
We also have to make sure that all algorithms implemented in the code and satellite package respect the interface specifications and do not make additional assumptions. As far as I can tell, this is not necessarily the case.
3. Long term reliability
As all open source developers know, taking care of such a vast ecosystem takes a lot of effort. I really think the community we’ve built around graphs has a great deal to offer in terms of performance and flexibility, but to succeed it would need:
- More guidance and discussions (like this one), to focus contributions on a few key packages instead of wasting time and energy
- More packages joining the organization as a way of diluting the workload
- More reactivity from maintainers (including me), to tackle issues and merge PRs
- More financial and human resources, ie. people who are actually paid to work on the ecosystem
I plan to recruit a half-time master’s student in the fall to start working on these issues. I was too late for the JSoC / GSoC calls, but as I understand there are others who got projects accepted in this general area (aurorarossi at least).
On the long term, it would be great if we could secure some funding for the JuliaGraphs organization.
I’m just a junior postdoc, but if anyone has ideas of grants we could apply to, let me know! I would love to make this open source mission a sizeable portion of my work, along with more theoretical research projects (I’m currently exploring graph neural networks, ping CarloLucibello).
Enough with the monologue. What do you think? Where should we go? How do we solve the Nebraska conundrum?
PS: Discourse limited me to 10 mentions, but feel free to add more among the usernames I couldn’t tag
@mbesancon @CarloLucibello @simonschoelly @jpfairbanks @viralbshah @aurorarossi @filchristou @etienne_dg @SuperGrobi @jlapeyre