Where should conversion methods for types from different packages go?

General Usage
1

I think it would be great if the Julia ecosystem could stay connective but modular, so it doesn’t end up like the fragmented ecosystem of R. One thing that would be really nice is interoperability among different types used for different applications, but I am unsure how to do this in practice.

As always, an example makes it easier. Say I use the Quandl.jl package to download some financial data, which comes in the TimeArray format defined in the TimeSeries package. I then want to do a regression analysis on the data (e.g. to look for temporal autocorrelation), so I load the GLM package, which is easiest to use if your data is in a DataFrame. I can fairly easily write a convert(DataFrame, x::TimeArray) function for my project. But it seems like this is functionality that thousands of users may want to use thousands of times, so it’d be nice to make it available.

AFAICS there are two possibilities: 1) submit the convert function to either DataFrames or TimeSeries - this would incur a dependency on the other package for that package, and would almost certainly be unpalatable; or 2) create a new package ConvertTimeArraysToDataFramesAndViceVersa.jl, that depends on both, which seems like it could easily bloat the package ecosystem.

Any well-established solutions? Perhaps the fabled optional dependencies that may come with Pkg3?

2

I would say that optional dependencies should solve this problem.

2 Likes
3

Optional dependencies will help in general, but for the specific case you describe I think TimeArray should just implement the future AbstractTable interface so that GLM will be able to build a model matrix automatically (via StatsModels). DataStreams will also allow converting to and from DataFrame as well as to and from any type which implements this interface. It appears that Query.jl is already able to do this (see this thread).

The interface will live in a small package on which all other tabular data packages will depend, and with optional dependencies it could even become optional.

2 Likes
4

Thanks, this is really helpful, as my question was also a more fundamental inquiry into the philosophy for cohesion in the julia package ecosystem. The idea of having generic interface that types can implement and then be generally compatible with functionality is a neat one - I am not sure I understand it and its implications completely, but it may become clearer as all these things click into place :slight_smile: