# JavaCallError when using Spark.jl

**URL:** <https://discourse.julialang.org/t/javacallerror-when-using-spark-jl/100496>\
**Category:** General Usage\
**Created:** [June 17, 2023, 6:17pm UTC](https://discourse.julialang.org/t/javacallerror-when-using-spark-jl/100496 "2023-06-17T18:17:53Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![taotree](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/taotree/32/31982_2.png) [@taotree](https://discourse.julialang.org/u/taotree)\
**Post date:** [June 17, 2023, 6:17pm UTC](https://discourse.julialang.org/t/javacallerror-when-using-spark-jl/100496/1 "2023-06-17T18:17:53Z")

</div>

When running `using Spark`:

```julia
julia> using Spark
ERROR: InitError: JavaCall.JavaCallError("JULIA_COPY_STACKS should not be set on Windows.")
Stacktrace:
  [1] assertroottask_or_goodenv
    @ C:\Users\joel\.julia\packages\JavaCall\MlduK\src\jvm.jl:233 [inlined]
  [2] _init(opts::Vector{String})
    @ JavaCall C:\Users\joel\.julia\packages\JavaCall\MlduK\src\jvm.jl:285
  [3] init()
    @ JavaCall C:\Users\joel\.julia\packages\JavaCall\MlduK\src\jvm.jl:277
  [4] init(; log_level::String)
    @ Spark C:\Users\joel\.julia\packages\Spark\89BUd\src\init.jl:56
  [5] init
    @ C:\Users\joel\.julia\packages\Spark\89BUd\src\init.jl:16 [inlined]
...
during initialization of module Spark

```

But then if I run `using Spark` again, it goes through. When trying to use it:

```julia
julia> spark = SparkSession.builder.appName("Main").master("local").getOrCreate()
ERROR: JavaCall.JavaCallError("Class Not Found org/apache/spark/sql/SparkSession")
Stacktrace:
 [1] _metaclass
   @ C:\Users\joel\.julia\packages\JavaCall\MlduK\src\core.jl:383 [inlined]
 [2] metaclass(class::Symbol)
   @ JavaCall C:\Users\joel\.julia\packages\JavaCall\MlduK\src\core.jl:389
 [3] jcall(::Type{JavaCall.JavaObject{Symbol("org.apache.spark.sql.SparkSession")}}, ::String, ::Type, ::Tuple{})
   @ JavaCall C:\Users\joel\.julia\packages\JavaCall\MlduK\src\core.jl:225
 [4] getproperty(#unused#::Type{SparkSession}, prop::Symbol)
   @ Spark C:\Users\joel\.julia\packages\Spark\89BUd\src\session.jl:48
 [5] top-level scope
   @ REPL[4]:1

```

I have jdk 11 and mvn in path. JAVA\_HOME set to jdk 11.

Side question… Will Spark.jl allow me to read a parquet file from hdfs running in kubernetes (and with the spark master running in kubernetes if that’s needed)?

---

<div class="post-metadata">

**Author:** ![dfdx](https://sea2.discourse-cdn.com/julialang/user_avatar/discourse.julialang.org/dfdx/32/120_2.png) [@dfdx](https://discourse.julialang.org/u/dfdx)\
**Post date:** [June 18, 2023, 12:41pm UTC](https://discourse.julialang.org/t/javacallerror-when-using-spark-jl/100496/2 "2023-06-18T12:41:38Z")

</div>

Usually, it’s a JavaCall.jl issue, see Major TODOs and Caveats secion from its [README](https://juliainterop.github.io/JavaCall.jl/). Spark.jl [doesn’t set this variable](https://github.com/search?q=repo%3Adfdx%2FSpark.jl%20JULIA_COPY_STACKS&type=code), so it might be set by your environment. Try checking it in a fresh Julia session:

```julia
ENV["JULIA_COPY_STACKS"]

```

If you are running Windows, it should **not** be set.

> Will Spark.jl allow me to read a parquet file from hdfs running in kubernetes (and with the spark master running in kubernetes if that’s needed)?

Parquet files are certainly supported, HDFS should not be a problem too since we only proxy Java methods to Julia. I’m not so sure about Kubernetes. Spark.jl does not support server modes (e.g. `yarn-server`), but client modes should work fine. You can always try and report issues if any 🙂