ANN: Plans for removing packages that do not yet support 1.0 from the General registry

Community
1

In the transition to using the new Registry, we put an upper bound of Julia version 0.7 on all packages that looked abandoned (by some heuristics involving last registration time, use of Compat etc). Now, approximately one year later, we plan to execute the next step which is to remove these abandoned packages completely from the registry.

While old abandoned packages hold a big place in our heart, having developed together with Julia, we feel that the benefits of doing this clean-up are large enough that it is worth doing. To be quantitative, trimming away pre 1.0 versions and packages will reduce the total number of lines in the registry from 180k to 80k. Some benefits of not holding older versions around are:

  • Faster resolve times, we need to parse fewer versions and the data structures for the resolver are smaller.
  • Potentially faster registry clone and update times. Fewer files need to be checked out and looked at.
  • Terser error messages when the resolver fails (it doesn’t list a bunch of available versions from pre 1.0)

The user impact depends on what Julia version is being run:

  • For people using Julia version 1.0+ this change will have no impact.
  • For people running Julia 0.7, using the new package manager and are currently using packages that would be removed, we are planning to make a branch of the General registry before we remove any packages. People using 0.7 should start using that branch to still have the 0.7 packages, but note that the branch will not get any updates. Updating to Julia 1.0 is strongly recommended.
  • For people running 0.6 and earlier, this change will have no impact (since the registry is not used for these Julia versions).

The process will be as follows:

  1. The script (located at https://gist.github.com/KristofferC/e6d9bbed08f689705f73cbcb0f672866) is run against a local version of the General registry. This script finds all package versions that do not support Julia 1.0 and those packages that would end up having no versions at all left after these versions would be removed. Also, any package with a version that depends on a package that would be removed is found.
  2. We will open an issue on every package that has no versions supporting 1.0 and thus will be removed (assuming the repository still exist).
  3. Approximately, one month later, we will run the script again and now actually perform the removal. There might be a slight disruption in package registration when this is merged (due to merge conflicts) which should easily be resolved by just rerunning the registration command.

At the time of writing, the list of packages that would be removed is shown below:

AMG
ASCIIPlots
ASTInterpreter
ASTInterpreter2
AWS
AWSEC2
AWSIAM
AWSLambda
AWSSDB
AWSSES
AWSSNS
AbbrvKW
AbstractDomains
AbstractTables
Accumulo
ActiveAppearanceModels
Actors
AffineTransforms
Akagera
AlgebraicDiffEq
AndorSIF
AnsiColor
AppConf
Arbiter
ArrayViews
AtariAlgos
AudioIO
Augmentor
Augur
AutoDiffSource
AutoTypeParameters
Autoreload
Avro
BEncode
BIGUQ
BOSSArrays
BSplines
BackpropNeuralNet
BanditOpt
Bandits
BaseTestDeprecated
BaseTestNext
BayesianDataFusion
Bebop
Benchmark
BenchmarkLite
Bezier
BioArgParse
BioBridgeR
BioSeq
BiomolecularStructures
Biryani
Blocks
BlsData
Bokeh
BooSTjl
BoundingBoxes
Box0
Brim
Brownian
BuildExecutable
CBOR
CLArrays
CLBLAS
CLFFT
COBRA
COFF
COMTRADE
CQL
CRC32
CRF
CUBLAS
CUDA
CUDAnativelib
CUDArt
CUDD
CUDNN
CUFFT
CURAND
CUSOLVER
CUSPARSE
CaesarLCMTypes
Calc
Calendar
Cartesian
CasaCore
Catalan
CauseMap
Celeste
CellularAutomata
ChainRecursive
ChaosCommunications
Chatter
Checkers
ChemicalKinetics
Chipmunk
CholmodSolve2
Church
CirruParser
Clarus
ClassicalCiphers
ClinicalTrialSampleSize
ClobberingReload
Clockwork
ClusterDicts
ClusterUtils
CmplxRoots
CoinOptServices
Color
CombinatorialBandits
CommonCrawl
Commutator
CompilerOptions
CompilerTools
ComposeDiff
CompressedSensing
ConfidenceWeighted
ConformalMaps
ConicIP
ConnectSDK
ConnectionPools
ConstructiveSolidGeometry
ContMechTensors
ContinuedFractions
ControlToolbox
Convertible
ConvexSwitch
CoordinateDescent
CoreNLP
CoverageBase
Cpp
Crispulator
CrossDecomposition
Crypto
Curl
Curvelet
DBAPI
DPMeansClustering
DReal
DWARF
DataCubes
DataStreamsIntegrationTests
DataTables
DataValueArrays
DateParser
Dates
Datetime
Debug
DebuggerFramework
DeclarativePackages
DelaunayMeshes
DeterminantalPointProcesses
DeterministicPolicyGradient
Devectorize
DictFiles
DictUtils
DiffBase
DiffEqApproxFun
DiffEqPy
DiffLinearAlgebra
DiffWrappers
Digits
DimensionalityReduction
DirichletProcessMixtures
DiscretePredictors
DiscriminantAnalysis
Distance
Docile
Docker
Dopri
DoubleDouble
Drawing
DriftDiffusion
DriftDiffusionPoissonSystems
Dtree
DynMultiply
DynaWAVE
DynamicDiscreteModels
DynamoDB
Dynare
EAGOBranchBound
EAGODomainReduction
EAGOIntervalArithmetic
EAGOParametricInterval
EAGOSmoothMcCormickGrad
EEG
EGM96
ELF
EasyPhys
EasyPkg
EchoJulia
EchogramPyPlot
EcologicalNetwork
EconDatasets
EconModels
Eglob
EgyptianFractions
EllipticFEM
EmpiricalRiskMinimization
EmpiricalRisks
Ensemble
EnvelopedArrays
Equations
EscapeString
Escher
Etcd
EventHistory
Evolutionary
EvolvingGraphs
ExcelDataStreams
ExecutableSpecifications
Expect
ExperimentalAnalysis
Expr2LaTeX
ExpressionPatterns
ExpressionUtils
Extern
ExtremelyRandomizedTrees
FEHM
FEMCoupling
FEMDynamics
FEModels
FLAC
FMIndexes
FNVHash
FPTControl
FProfile
FWF
FaSTLMM
FaceDatasets
FactCheck
FactorModels
FancyDiagnostics
FastAnonymous
FastArrayOps
FastCombinations
FinancialMarkets
FiniteElementDiffEq
FiniteStateMachine
FirstOrderLogic
FirstOrderSolvers
FixedPoint
FixedSizeArrays
FixedSizeDictionaries
Fixtures
FlexibleArrays
FractionalGaussianFields
FrontierEfficiencyAnalysis
Fuji
FunctionalData
FunctionalDataUtils
FusionDirect
GARCH
GLAbstraction
GLBooks
GLPlot
GLText
GLVisualize
GLWindow
GUITestRunner
GadflyDiff
Gallium
Gasp
GeneralizedMetropolisHastings
GeneralizedSampling
GeneralizedSchurAlgorithm
GeneticAlgorithms
GeoIP
GeographicLibPy
GibbsSeaWater
Gillespie
GitLab
GnuTLS
GoogleCharts
GradientBoost
Graft
GraphCentrality
GraphGLRM
GraphLayout
GraphMatrices
GraphViz
GreatCircle
Grid
GroupSlices
GroveAlg
GtkBuilderAid
Gunrock
HDFS
HORIZONS
HPAT
HPack
HSA
HTSLIB
HTTP2
HarwellRutherfordBoeing
HeadlessChromium
Helpme
HexEdit
HiddenMarkovModels
HigherOrderClustering
HigherOrderKernels
HigherPrecision
Hinton
Homotopies
Hop
HttpParser
HttpServer
ICOADSDict
ICU
IDRsSolver
IDXParser
IJuliaPortrayals
IOIndents
IPPCore
IPPDSP
IProfile
ISPC
ImageProjectiveGeometry
ImageRegistration
ImmutableArrays
ImputeNaNs
IncGammaBeta
IndependentRandomSequences
IndexedArrays
InformedDifferentialEvolution
Instruments
IntModN
IntegerSmithNormalForm
InteractUIkit
Interfaces
IntervalLinearEquations
IntervalWavelets
InvariantCausal
IsoRank
Isotonic
IssueReporter
IterationManagers
Iterators
Ito
J4Org
JACKAudio
JFVM
JLDArchives
JPLEphemeris
JUDI
JellyFish
Jewel
JointMoments
JuMPChance
JuliaParser
JulieTest
Julz
Jumos
KCores
KDTrees
KSVD
KShiftsClustering
Kafka
KernSmooth
Klara
LARS
LASindex
LCA
LMDB
LTISystems
LaTeX
LazySequences
LempelZiv
Lexicon
LibArchive
LibBSON
LibCloud
LibHealpix
LibTrading
LifeTable
Liga
LineEdit
LinearAlgebraicRepresentation
LinearLeastSquares
LinearResponseVariationalBayes
LinguisticData
Lint
LocalMemoize
LoggedDicts
Logging
Lora
Loss
LowDimNearestNeighbors
Lumberjack
Lumira
LyapunovExponents
MAB
MATLABCluster
MCMC
MDPs
MFCC
MNIST
MP3
MPFI
MRMissing
MUMPSjInv
MachO
MachineLearning
MachineLearningMetrics
Maker
Mandrill
ManualMemory
MapLight
MarketTechnicals
MarkovTransitionMatrices
Matcha
MathToolkit
Mathematica
MatlabCompat
MatpowerCases
MatrixChainMultiply
Mayday
Meddle
MelGeneralizedCepstrums
MergeSorted
MergedMethods
Merlin
Meshes
MessageUtils
MetaTools
MetadataTools
Metamath
MichiganAutonomousVehicles
Miniball
MinimalPerfectHashes
MixtureModels
Mocha
ModelConstructors
ModelReduction
MolecularDataType
MolecularDynamics
MolecularPDB
Monads
Mongo
Morsel
Mortar2D
Mortar3D
MotoServer
MovingWeightedLeastSquares
MsgPackRpcServer
MuKanren
MultiNest
MultiPoly
MultidimensionalTables
Multirate
Murmur3
NBTesting
NEOS
NHST
NKLandscapes
NLOptControl
NMR
NOAAData
NOMAD
NURBS
NamedDimensions
Napier
NaturalSelection
NaturalSort
Neovim
NetalignMeasures
NetalignUtils
NetworkFlows
NetworkViz
NeuralynxNCS
NextGenSeqUtils
NodeNumbering
NonNegLeastSquares
NullableArrays
Nulls
NumFormat
NumberedLines
NumericExtensions
NumericFuns
NumericSuffixes
OCCA
OEIS
OSC
OSXNotifier
ObjFileBase
OpenFOAM
OpenGene
OpenSecrets
OpenSlide
OpenStreetMap
OptimPack
Options
Opus
Orchestra
PAINTER
PDFExtract
PEGParser
PGM
PLSRegressor
PLX
PSPlot
PValueAdjust
PackageStream
PageAlignedArrays
ParallelAccelerator
ParallelSparseMatMul
Pastebin
Patchwork
PatternDispatch
Pcap
PdbTool
Pedigrees
Perceptrons
Persist
PhantomJS
Phonetics
PhyloModels
PhyloTrees
Phylogenetics
Phylogenies
PhysConsts
PiecewiseAffineTransforms
PiecewiseIncreasingRanges
PkgDev
PlanOut
PlanarMaps
Playground
PlotRecipes
PolarFact
PolyaGammaDistribution
Polyglot
Polynomial
PolynomialMatrices
PortAudio
PortableGameNotation
PowerLaws
PowerSeries
PrettyNumber
PrettyPlots
PrivateModules
PrivateMultiplicativeWeights
ProgressiveAligner
ProjectTemplate
Promisables
PropertyGraph
ProximalBase
PublicSuffix
Push
PyAMG
PyLCM
PyLexYacc
PyLogging
PyProj
PySide
PySyntax
PyTest
PyX
Pyramids
Q
QDXML
QML
QPSParser
QRupdate
QWTwPlot
QuDirac
QuTiP
Quandl
QuantumBayesian
QuantumLab
QuantumTomography
Queueing
QuickHull
QuickShiftClustering
Qwt
RDF
REPL
REPLGameOfLife
ROOT
Rainflow
RandomFerns
Ranges
RationalExtensions
RationalFunctions
ReactiveBasics
ReadWriteLocks
RealInterface
RealSense
Reconstructables
Redis
RegERMs
Regression
Requests
RequestsCache
Restful
Revealables
ReverseDiffOverload
ReverseDiffSource
Revtok
Rif
Rifraf
RingArrays
RobustLeastSquares
RobustStats
RouletteWheels
RudeOil
RunTests
SALSA
SDE
SFML
SGDOptim
SIUnits
SLHA
SMC
SPTK
SQSChannels
SVM
SVMLightLoader
Samantha
Sampling
SaveREPL
SchroedingersSmoke
SearchMethods
SecureSessions
Seismic
SemidefiniteModel
SemidefiniteProgramming
SeqMaker
Shannon
ShapeModels
Shearlab
ShowItLikeYouBuildIt
Sigma
SigmoidalProgramming
SignalView
Silo
SimilarityNetworkFusion
SimpleStructs
SimpleTasks
Sims
SloanDigitalSkySurvey
Slugify
Smile
SnFFT
SnowballStemmer
SoftConfidenceWeighted
SolarSystemLib
Somoclu
SortJoin
Sparklines
SparseInterp
SparseMatrixLILs
SparseVectors
SpatialGrids
SpecialMatrices
SpeedDate
SpikingNetworks
StackTraces
StackedNets
StatefulIterators
Stats
StatsdClient
Steganography
SteinDiscrepancy
StingerGraphs
StochasticSearch
Stochy
StokesDiffEq
StrPack
StreamStats
StringCases
StringInterpolation
StringLiterals
StringParserPEG
StructsOfArrays
StructuredQueries
Subsequences
SubsetSelection
Sugar
SunlightAPIs
Surprise
SwiftObjectStores
Swifter
Switch
SwitchTimeOpt
SymmetricBitArrays
Synchrony
SynthesisFilters
SystemImageBuilder
TOML
TablesDemo
TakingBroadcastSeriously
TensorDecompositions
TermWin
TerminalUI
Terminals
TestReports
TestRunner
TestSetExtensions
Tetris
TexExtensions
TextModel
TextPlots
ThermodynamicsTable
ThingSpeak
ThreeJS
TimeData
TimeModels
TimeSeriesIO
Timestamps
TinyInt
TinySegmenter
TinySparse
Tinyurl
Tk
Tokamak
TopicModels
TraceCalls
Transit
Transpiler
TwoBasedIndexing
TypeCheck
Typeclass
UDUnits
URITemplate
UTF16
UUID
Units
UnsafeAtomics
Unums
VLFeat
VML
VSL
VStatistic
ValueDispatch
ValueSymbols
VarianceComponentTest
Vectorize
Vega
VehicleModels
VennEuler
Viridis
VirtualArrays
Voronoi
Voting
Vue
WCSLIB
Wallace
Watcher
WaveletMatrices
WebAssembly
WebFuncs
Weber
WeberCedrus
WeberDAQmx
Whippet
Whittaker
XClipboard
XDiff
XKPasswd
XMLRPC
XMLconvert
XSV
YT
Yelp
ZipCode
Zlib
fstformat
mPulseAPI

I will update this post when we have more concrete dates of the three steps to share.

40 Likes
2

Go gently into that good night, FastAnonymous!

17 Likes
3

What will happen with the names of the removed packages? Will it be possible to register a new package under these names?

3 Likes
4

Wouldn’t that raise security concerns?

5

https://github.com/JuliaLang/Pkg.jl/issues/931

1 Like
6

OK, this method is good to defend the existing packages. But the above packages will be removed. Will their names be reserved forever?

7

No, I think packages this old have lost the claim to these names. If you read that issue, I also point out that we can technically already support exact name collisions and exact name collisions are just an extreme form of typo near-collisions. If there are two packages with the same name, we can just prompt users for which one they meant, listing the most popular one first. Allowing the org/user name to disambiguate packages with the same name is also a relevant potential feature.

3 Likes
8
  • Is this removing the versions which do not support Julia v.1?
  • Should we handle the deleted repositories (or releases) as well in this purge?
  • Related, update the repository URL if these have been moved?
9
  • Yes, I thought the first post described exactly that, maybe it was unclear somehow.
  • Yeah, should probably do that.
  • This won’t cause any packages to be deleted so can easily be done at some other point.
3 Likes
10

Issues should now have been opened on the relevant repos (yay GitHub didn’t ban me).

There was a bit of confusion with packages that got renamed. The old renamed packages has a URL that points to the repo with the new name, which might very well be an active package (and would not be removed). I tried to detect this and add comments pointing out that the opened issue is about the package with the old name that is no longer used.

In approximately 1 month we will go ahead with the clean up.

Also, I feel that I should point out. Just because a package gets removed from the registry doesn’t mean it can’t just be re-registered if it gets updated some time in the future.

9 Likes
11

Hello there,

Thanks for doing this. After I was notified by an email, I tried to follow instructions, but I immediately ran into a few problems:

So for me, the transitioning from the old METADATA to the new package registration system still is not completely clear.

12

If the package was previously registered you need to use the same UUID. The script above should take care of this, otherwise look up your UUID in the General registry: GitHub - JuliaRegistries/General: The official registry of general Julia packages

1 Like
13

Link has been fixed now on Registrator.jl

14

Thanks for finding and fixing the link! I think now registration is proceeding according to protocol.

15

Any update on this?

16

Yeah, I should just run the script again and then we should do it. I want to hit a point where there aren’t so many registrations to General in-flight.

17

We just pulled the trigger on this.

We will hold off merging new versions of packages for ~24 hours for it to be a simple revert in case something is wrong.

The following packages updated to 1.0 that were slated to be removed:

BEncode
BlsData
COBRA
CUDD
ClassicalCiphers
ConformalMaps
CoordinateDescent
EgyptianFractions
Expect
FaSTLMM
FirstOrderSolvers
FractionalGaussianFields
GARCH
GroupSlices
HORIZONS
HarwellRutherfordBoeing
HigherOrderKernels
ImageProjectiveGeometry
IterationManagers
LASindex
LifeTable
MFCC
ModelConstructors
Monads
NOMAD
OptimPack
PhyloModels
PhyloTrees
PlanarMaps
PolynomialMatrices
ProximalBase
SMC
SimilarityNetworkFusion
SpecialMatrices
Steganography
TensorDecompositions
UDUnits
VSL
WebAssembly
7 Likes