# Importing big data

**URL:** https://discourse.julialang.org/t/importing-big-data/7030
**Category:** General Usage
**Tags:** question
**Created:** [November 12, 2017, 7:58pm UTC](https://discourse.julialang.org/t/importing-big-data/7030 "2017-11-12T19:58:54Z")
**Posts on this page:** 2
**Page:** 2

<div class="post-metadata">

### Author: ![dfdx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dfdx/32/120_2.png) [@dfdx](https://discourse.julialang.org/u/dfdx)
#### Post date: [November 13, 2017, 11:07pm UTC](https://discourse.julialang.org/t/importing-big-data/7030/21 "2017-11-13T23:07:08Z")

</div>

I don’t have PostgreSQL at hand to test it, but it should look something like this (assuming you have already install Spark.jl):

1. Edit `jvm/sparkjl/pom.xml` and add the following to `dependencies` section:

```julia
<!-- https://mvnrepository.com/artifact/org.postgresql/postgresql -->
<dependency>
    <groupId>org.postgresql</groupId>
    <artifactId>postgresql</artifactId>
    <version>42.1.4</version>
</dependency>

```

In Java world, `pom.xml` is a single place you put all your dependencies. Since basic Spark installation doesn’t support PostgreSQL driver, we need to add it to the Java `CLASSPATH`. There are other ways to do it, but find this one pretty simple for basic use cases.

1. Run `Pkg.build("Spark")` for changes to take effect.

2. Create `SparkSession`:

```julia
using Spark
Spark.init()
sess = SparkSession() # uses "local" master

```

1. Read Spark’s `Dataset` using JDBC format:

```julia
options = Dict(
    "url" => "jdbc:postgresql:dbserver",
    "dbtable" => "schema.tablename",
    "user" => "username",
    "password" => "password")
df = read_df(sess, ""; format="jdbc", options=options)

```

Converting Spark Dataset / DataFrame to Julia DataFrame isn’t supported out of the box yet, but you can:

- export Spark dataset to CSV and read it from DataFrames.jl
- call `collect(spark_df)` to get a list of rows and then build a Julia `DataFrame`

Issues on GitHub are also welcome. Spark API is really huge, so instead of randomly implementing parts of it I expect users of Spark.jl to create issues so I could prioritize and plan them.

---

<div class="post-metadata">

### Author: ![ExpandingMan](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/expandingman/32/866_2.png) [@ExpandingMan](https://discourse.julialang.org/u/ExpandingMan)
#### Post date: [November 14, 2017, 2:31pm UTC](https://discourse.julialang.org/t/importing-big-data/7030/22 "2017-11-14T14:31:21Z")

</div>

At some point when the data ecosystem stabilizes some I’d be happy to make PR’s for DataFrames support (still waiting for latest to be tagged).

[Previous page](https://discourse.julialang.org/t/importing-big-data/7030.md?page=1)