Hi All,
I ran
JULIA_COPY_STACKS=yes
julia -e ‘using Pkg;Pkg.add(Pkg.PackageSpec(;name=“Spark”, version=“0.5.1”));Pkg.build(“Spark”);using Spark;Spark.init();sc = SparkContext(master=“yarn”);’
all this worked fine.
Can someone please help me with below issue?
julia> text = parallelize(sc, [“hello world”, “the world is one”, “we are the world”])
Exception in thread “main” java.lang.NoSuchMethodError: org.apache.spark.internal.Logging.init(Lorg/apache/spark/internal/Logging;)V
at org.apache.spark.api.julia.JuliaRDD$.(JuliaRDD.scala:67)
at org.apache.spark.api.julia.JuliaRDD$.(JuliaRDD.scala)
at org.apache.spark.api.julia.JuliaRDD.readRDDFromFile(JuliaRDD.scala)
ERROR: JavaCall.JavaCallError(“Error calling Java: java.lang.NoSuchMethodError: org.apache.spark.internal.Logging.$init$(Lorg/apache/spark/internal/Logging;)V”)
Stacktrace:
[1] geterror(allow::Bool)
@ JavaCall /usr/local/share/julia/v1.6.2/packages/JavaCall/tjlYt/src/core.jl:418
[2] geterror
@ /usr/local/share/julia/v1.6.2/packages/JavaCall/tjlYt/src/core.jl:403 [inlined]
[3] _jcall(::JavaCall.JavaMetaClass{Symbol(“org.apache.spark.api.julia.JuliaRDD”)}, ::Ptr{Nothing}, ::Ptr{Nothing}, ::Type, ::Tuple{DataType, DataType, DataType}, ::JavaCall.JavaObject{Symbol(“org.apache.spark.api.java.JavaSparkContext”)}, ::Vararg{Any, N} where N)
@ JavaCall /usr/local/share/julia/v1.6.2/packages/JavaCall/tjlYt/src/core.jl:373
[4] jcall(::Type{JavaCall.JavaObject{Symbol(“org.apache.spark.api.julia.JuliaRDD”)}}, ::String, ::Type, ::Tuple{DataType, DataType, DataType}, ::JavaCall.JavaObject{Symbol(“org.apache.spark.api.java.JavaSparkContext”)}, ::Vararg{Any, N} where N)
@ JavaCall /usr/local/share/julia/v1.6.2/packages/JavaCall/tjlYt/src/core.jl:227
[5] parallelize(sc::SparkContext, coll::Vector{String}; n_split::Int64)
@ Spark /usr/local/share/julia/v1.6.2/packages/Spark/9bsuG/src/context.jl:88
[6] parallelize(sc::SparkContext, coll::Vector{String})
@ Spark /usr/local/share/julia/v1.6.2/packages/Spark/9bsuG/src/context.jl:84
[7] top-level scope
@ REPL[5]:1
Thanks and Regards,
Sumit Malbari
malbarisumit@gmail.com