# CSV woes and SubString documentation

**URL:** https://discourse.julialang.org/t/csv-woes-and-substring-documentation/7938
**Category:** New to Julia
**Created:** [December 23, 2017, 5:57am UTC](https://discourse.julialang.org/t/csv-woes-and-substring-documentation/7938 "2017-12-23T05:57:51Z")
**Posts on this page:** 10
**Page:** 1

<div class="post-metadata">

### Author: ![anon67531922](https://avatars.discourse-cdn.com/v4/letter/a/48db29/32.png) [@anon67531922](https://discourse.julialang.org/u/anon67531922)
#### Post date: [December 23, 2017, 5:57am UTC](https://discourse.julialang.org/t/csv-woes-and-substring-documentation/7938/1 "2017-12-23T05:57:51Z")

</div>

Hi,

I have a small csv file with 8 columns and 48 rows. The first two columns are strings and the remaining columns are floats.

My goal: For each row, extract the first two strings and parse the remaining floats into a vector and create an instance of a custom type

```julia
MyType(s1::String,s2::String,v::Vector{Float64}

```

At first, it seemed like a natural candidate for CSV and Query. I know my coding ability sucks, but I can almost write it by hand faster than my code is reading that small file. It is taking 6 seconds to read and parse using CSV.read and Query. Digging around, I found a comment that DataFrames are slow if you are manipulating rows, which is what I was doing.

I then used `DataFrames.stack` to turn my rows into columns, but that was even slower (~8 seconds).

My latest attempt is to use Base.readcsv. This reads my CSV into a 2d array, which seems like something I can work with, but when I access on element of the first two columns, the type is `SubString{String}`. I can’t seem to be able to find any documentation for `SubString` and I need a String. How can I get a `String` from a `SubString{String}`?

Any ideas would be appreciated. Thanks 🙂

PS: I know the first time to run any function is slow, but I will only ever run this function once since I only need to load the data to memory once.

---

<div class="post-metadata">

### Author: ![rdeits](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/rdeits/32/286_2.png) [@rdeits](https://discourse.julialang.org/u/rdeits)
#### Post date: [December 23, 2017, 6:08am UTC](https://discourse.julialang.org/t/csv-woes-and-substring-documentation/7938/2 "2017-12-23T06:08:09Z")

</div>

It will be a lot easier to help if you can provide a working example of your data and the code you have so far. Can you post a reproducible example?

> How can I get a String from a SubString{String}?

`convert(String, x)`

---

<div class="post-metadata">

### Author: ![anon67531922](https://avatars.discourse-cdn.com/v4/letter/a/48db29/32.png) [@anon67531922](https://discourse.julialang.org/u/anon67531922)
#### Post date: [December 23, 2017, 6:14am UTC](https://discourse.julialang.org/t/csv-woes-and-substring-documentation/7938/3 "2017-12-23T06:14:29Z")

</div>

Thanks. I could swear I tried that 😅 It is that intense stress thing again. Appreciate the help 👍

---

<div class="post-metadata">

### Author: ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)
#### Post date: [December 23, 2017, 6:27am UTC](https://discourse.julialang.org/t/csv-woes-and-substring-documentation/7938/4 "2017-12-23T06:27:21Z")

</div>

This is actually a nice case where [Query.jl](https://github.com/davidanthoff/Query.jl)’s briding of the relational world with the julia type system works well. Here is how I’d do it:

```julia
using FileIO, CSVFiles, Query

data = load("rep.csv") |> @map(MyType(_.s1, _.s2, values(_)[3:end])) |> collect

```

That should give you a `Vector{MyType}`.

I have no good solution for the speed problem when you run this for the first time, other than hoping for some wonders around precompilation to happen at some point 🙂

---

<div class="post-metadata">

### Author: ![anon67531922](https://avatars.discourse-cdn.com/v4/letter/a/48db29/32.png) [@anon67531922](https://discourse.julialang.org/u/anon67531922)
#### Post date: [December 23, 2017, 7:34am UTC](https://discourse.julialang.org/t/csv-woes-and-substring-documentation/7938/5 "2017-12-23T07:34:28Z")

</div>

> [@davidanthoff](#):
>
> data = load(“rep.csv”) |\> @map(MyType(\_.s1, _.s2, values(_)[3:end])) |\> collect

Thanks @davidanthoff! This looks like a beautiful solution 😍 I wasn’t aware of FilesIO and IterableTables 👍

I have a small problem. There is a `\ufeff` at the beginning of my CSV file and I’m having trouble getting rid of it and it is corrupting `s1` so I can’t get your trick to work 🤔

Any ideas?

Edit: Nevermind. Got it! 👏

---

<div class="post-metadata">

### Author: ![anon67531922](https://avatars.discourse-cdn.com/v4/letter/a/48db29/32.png) [@anon67531922](https://discourse.julialang.org/u/anon67531922)
#### Post date: [December 23, 2017, 7:41am UTC](https://discourse.julialang.org/t/csv-woes-and-substring-documentation/7938/6 "2017-12-23T07:41:05Z")

</div>

Works beautifully 😍🙌

---

<div class="post-metadata">

### Author: ![Liso](https://avatars.discourse-cdn.com/v4/letter/l/898d66/32.png) [@Liso](https://discourse.julialang.org/u/Liso)
#### Post date: [December 23, 2017, 12:35pm UTC](https://discourse.julialang.org/t/csv-woes-and-substring-documentation/7938/7 "2017-12-23T12:35:40Z")

</div>

Could `load` recognize [BOM](https://en.wikipedia.org/wiki/Byte_order_mark) for UTF-16 with different endian?

What was your solution?

---

<div class="post-metadata">

### Author: ![anon67531922](https://avatars.discourse-cdn.com/v4/letter/a/48db29/32.png) [@anon67531922](https://discourse.julialang.org/u/anon67531922)
#### Post date: [December 23, 2017, 12:43pm UTC](https://discourse.julialang.org/t/csv-woes-and-substring-documentation/7938/8 "2017-12-23T12:43:10Z")

</div>

My solution was anything but elegant. Opened it in Excel and resaved as .csv 🙂

---

<div class="post-metadata">

### Author: ![anon67531922](https://avatars.discourse-cdn.com/v4/letter/a/48db29/32.png) [@anon67531922](https://discourse.julialang.org/u/anon67531922)
#### Post date: [December 24, 2017, 3:14am UTC](https://discourse.julialang.org/t/csv-woes-and-substring-documentation/7938/9 "2017-12-24T03:14:52Z")

</div>

@davidanthoff Just FYI. Applying your trick, I am literally removing hundreds of lines of code and gaining awesome speed boosts. Thanks again 👍

---

<div class="post-metadata">

### Author: ![davidanthoff](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/davidanthoff/32/223493_2.png) [@davidanthoff](https://discourse.julialang.org/u/davidanthoff)
#### Post date: [December 24, 2017, 6:36am UTC](https://discourse.julialang.org/t/csv-woes-and-substring-documentation/7938/10 "2017-12-24T06:36:15Z")

</div>

Yay!
