# Importing big data

**URL:** <https://discourse.julialang.org/t/importing-big-data/7030>\
**Category:** General Usage\
**Tags:** question\
**Created:** [November 12, 2017, 7:58pm UTC](https://discourse.julialang.org/t/importing-big-data/7030 "2017-11-12T19:58:54Z")\
**Posts on this page:** 1\
**Showing post:** 21

<div class="post-metadata">

**Author:** ![dfdx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dfdx/32/120_2.png) [@dfdx](https://discourse.julialang.org/u/dfdx)\
**Post date:** [November 13, 2017, 11:07pm UTC](https://discourse.julialang.org/t/importing-big-data/7030/21 "2017-11-13T23:07:08Z")

</div>

I don’t have PostgreSQL at hand to test it, but it should look something like this (assuming you have already install Spark.jl):

1. Edit `jvm/sparkjl/pom.xml` and add the following to `dependencies` section:

```julia
<!-- https://mvnrepository.com/artifact/org.postgresql/postgresql -->
<dependency>
    <groupId>org.postgresql</groupId>
    <artifactId>postgresql</artifactId>
    <version>42.1.4</version>
</dependency>

```

In Java world, `pom.xml` is a single place you put all your dependencies. Since basic Spark installation doesn’t support PostgreSQL driver, we need to add it to the Java `CLASSPATH`. There are other ways to do it, but find this one pretty simple for basic use cases.

1. Run `Pkg.build("Spark")` for changes to take effect.

2. Create `SparkSession`:

```julia
using Spark
Spark.init()
sess = SparkSession() # uses "local" master

```

1. Read Spark’s `Dataset` using JDBC format:

```julia
options = Dict(
    "url" => "jdbc:postgresql:dbserver",
    "dbtable" => "schema.tablename",
    "user" => "username",
    "password" => "password")
df = read_df(sess, ""; format="jdbc", options=options)

```

Converting Spark Dataset / DataFrame to Julia DataFrame isn’t supported out of the box yet, but you can:

- export Spark dataset to CSV and read it from DataFrames.jl
- call `collect(spark_df)` to get a list of rows and then build a Julia `DataFrame`

Issues on GitHub are also welcome. Spark API is really huge, so instead of randomly implementing parts of it I expect users of Spark.jl to create issues so I could prioritize and plan them.

---

_[View the full topic](https://discourse.julialang.org/t/importing-big-data/7030)._