# Setting up Julia on Spark on AWS EMR

**URL:** <https://discourse.julialang.org/t/setting-up-julia-on-spark-on-aws-emr/63781>\
**Category:** Julia at Scale\
**Tags:** question, spark, ijulia\
**Created:** [June 29, 2021, 5:53pm UTC](https://discourse.julialang.org/t/setting-up-julia-on-spark-on-aws-emr/63781 "2021-06-29T17:53:05Z")\
**Posts on this page:** 5\
**Page:** 2

<div class="post-metadata">

**Author:** ![dfdx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dfdx/32/120_2.png) [@dfdx](https://discourse.julialang.org/u/dfdx)\
**Post date:** [July 29, 2021, 8:26pm UTC](https://discourse.julialang.org/t/setting-up-julia-on-spark-on-aws-emr/63781/21 "2021-07-29T20:26:51Z")

</div>

The log you posted points out that the following command launched via the build script fails:

```julia
mvn clean package -Dspark.version=2.4.7 -Dscala.version=2.11.12 -Dscala.binary.version=2.11

```

Can you run this command directly from the `<Spark.jl Dir>/jvm/sparkjl` directory and post the result?

---

<div class="post-metadata">

**Author:** ![dacort](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dacort/32/27532_2.png) [@dacort](https://discourse.julialang.org/u/dacort)\
**Post date:** [July 29, 2021, 10:49pm UTC](https://discourse.julialang.org/t/setting-up-julia-on-spark-on-aws-emr/63781/22 "2021-07-29T22:49:20Z")

</div>

Yea, based on this error looks like there’s some issue with outbound access to the maven repo?

```julia
[WARNING] Could not transfer metadata org.apache.maven.plugins:maven-source-plugin/maven-metadata.xml from/to central (https://repo.maven.apache.org/maven2): transfer failed for https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-source-plugin/maven-metadata.xml

```

Not sure why that would be, though, since I’m guessing the other downloads worked fine?

---

<div class="post-metadata">

**Author:** ![Sumit\_Malbari](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sumit_malbari/32/26623_2.png) [@Sumit\_Malbari](https://discourse.julialang.org/u/Sumit_Malbari)\
**Post date:** [August 2, 2021, 9:15pm UTC](https://discourse.julialang.org/t/setting-up-julia-on-spark-on-aws-emr/63781/23 "2021-08-02T21:15:25Z")

</div>

Hi @dacort , @dfdx ,

I had to create the settings.xml file in the .m2 folder in my home directory so as to allow access to maven through artifactory directory of my company. This worked fine. Now, I am facing another issue while executing the below command:

julia -e ‘using Pkg;Pkg.add(Pkg.PackageSpec(;name=“Spark”, version=“0.5.1”));using Spark;Spark.init();sc = SparkContext(master=“yarn”);sc.parallelize([1,2,3,4])’

Please find the below error:

ERROR spark.SparkContext: Failed to add /home/hadoop/.julia/packages/Spark/9bsuG/src/…/jvm/sparkjl/target/sparkjl-0.1.jar to Spark environment  
java.io.FileNotFoundException: Jar /home/hadoop/.julia/packages/Spark/9bsuG/src/…/jvm/sparkjl/target/sparkjl-0.1.jar not found  
at org.apache.spark.SparkContext.addJarFile$1(SparkContext.scala:1874)  
at org.apache.spark.SparkContext.addJar(SparkContext.scala:1902)  
at org.apache.spark.api.java.JavaSparkContext.addJar(JavaSparkContext.scala:701)  
ERROR: type SparkContext has no field parallelize

Could you please let me know what can be done about this?

Thanks and Regards,  
Sumit Malbari

---

<div class="post-metadata">

**Author:** ![dfdx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dfdx/32/120_2.png) [@dfdx](https://discourse.julialang.org/u/dfdx)\
**Post date:** [August 3, 2021, 12:26pm UTC](https://discourse.julialang.org/t/setting-up-julia-on-spark-on-aws-emr/63781/24 "2021-08-03T12:26:18Z")

</div>

> [@Sumit\_Malbari](#):
>
> /home/hadoop/.julia/packages/Spark/9bsuG/src/…/jvm/sparkjl/target/sparkjl-0.1.jar to Spark environment

It looks the JAR file hasn’t been created. Please repeat the build and verify the file is actually there.

---

<div class="post-metadata">

**Author:** ![Sumit\_Malbari](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/sumit_malbari/32/26623_2.png) [@Sumit\_Malbari](https://discourse.julialang.org/u/Sumit_Malbari)\
**Post date:** [January 4, 2022, 3:13pm UTC](https://discourse.julialang.org/t/setting-up-julia-on-spark-on-aws-emr/63781/25 "2022-01-04T15:13:50Z")

</div>

hi @dfdx , @dacort

I built the spark and its showing success, but when I am trying to run below command, I am getting below error:

julia\> text = parallelize(sc, [“hello world”, “the world is one”, “we are the world”])  
Exception in thread “main” java.lang.NoSuchMethodError: org.apache.spark.internal.Logging.init(Lorg/apache/spark/internal/Logging;)V  
at org.apache.spark.api.julia.JuliaRDD$.(JuliaRDD.scala:67)  
at org.apache.spark.api.julia.JuliaRDD$.(JuliaRDD.scala)  
at org.apache.spark.api.julia.JuliaRDD.readRDDFromFile(JuliaRDD.scala)  
ERROR: JavaCall.JavaCallError(“Error calling Java: java.lang.NoSuchMethodError: org.apache.spark.internal.Logging.$init$(Lorg/apache/spark/internal/Logging;)V”)  
Stacktrace:

Anyidea on this issue?

Thanks and Regards,  
Sumit Malbari

[Previous page](https://discourse.julialang.org/t/setting-up-julia-on-spark-on-aws-emr/63781.md?page=1)