Error exporting dataframe from Julia to R with Feather

New to Julia
1

Hello,

since I am encountering a lot of problems exporting dataframes from julia to R (there is always something wrong with the formatting, probably a missing quote) so I am trying to use Feather to do the job.
The dataframe exported with CSV.write("dir/dataframe.tsv", df; delim='\t') has this structure:

> df <- read.table(
+   paste("dir/dataframe.tsv"),
+   header = TRUE,
+   row.names = NULL, 
+   dec = ".",
+   sep = ",",
+   fill = TRUE,
+   stringsAsFactors = FALSE,
+   comment.char = ""
+ )
Warning message:
In scan(file = file, what = what, sep = sep, quote = quote, dec = dec,  :
  EOF within quoted string
> str(hit)
'data.frame':	81365 obs. of  1 variable:
 $ Group.Sample.Start.Match_len.Read_len.Hit_len.Hit.ID: chr  "Normal\tA4\t6064657\t27\t101\t13669\tNC_038931.1 Thermoproteus tenax virus 1 (TTV1) genome\tNC_038931" "Normal\tA0\t6064658\t29\t101\t13669\tNC_038931.1 Thermoproteus tenax virus 1 (TTV1) genome\tNC_038931" "Normal\tA0\t6064659\t28\t101\t13669\tNC_038931.1 Thermoproteus tenax virus 1 (TTV1) genome\tNC_038931" "Normal\tA0\t6064660\t27\t101\t13669\tNC_038931.1 Thermoproteus tenax virus 1 (TTV1) genome\tNC_038931" ...

I have installed Feather in Julia with pkg.add("Feather") and imported it with using Feather. I created a dataframe and saved it with Feather.write("/dir/dataframe.feather", df) and it worked. I can even open it back with df = Feather.read("/dir/dataframe.feather") and get: julia> nrow(df) 128544.
The problem is with R. I am using Rstudio to test each step.
I installed the package, imported it with library(feather) and used it as:

df = read_feather("/dir/dataframe.feather")

and then R/Rstudio simply crashes.
any idea why?
Thank you

2

As I recall there were problems importing into R and we never resolved why.

Could you please provide the following information:

  1. Could you pelase ensure that you have the very latest version of Feather.jl by doing ]up Feather in Julia.
  2. When you say R “crashes”, what exactly does it do? Is there any output in the terminal?
  3. Could you try importing the same dataframe in Python using pandas py doing pandas.read_feather? (you’ll need to have pyarrow installed as well). If that loads correctly, then it is highly likely that the problem is with the R implementation of feather.
3

One workaround is saving the Feather file through R using RCall: R session aborted when reading feather file from Julia
Another is to pin FlatBuffers at v0.4.0 (see here, and linked issues).

4

The feather specification is more general than what R can represent. It is a little known fact that there are only two numeric structures in the underlying representation (symbolic expressions or SEXPRECs) for R, fixed-length vectors of 64-bit floats or of 32-bit integers. You likely had 64-bit ints or something like that in your feather file.

I saw that Ursalabs just announced a new arrow package (as in the Apache Arrow project) for R, which can read feather. It might do a better job of smoothing over the rough edges, but it is not exactly convenient to install.

1 Like