Databricks and Julia


for my job we are exploring the datalakehouse possibilities in DataBricks. Also we probably need to write new code (currently in PL/SQL) into a language that is accepted in the databricks envirroment.

I am new to the databricks field, I have seen that spark and Julia should work together but I don’t see any explicit teamups with Julia and Databricks. Does that mean I can’t use Julia in DataBricks or because spark is the underlying technology that is being used by Databricks I CAN use Julia in Databricks?if yes, what are you experiences with Databricks and Julia?


As far as I know, Databircks uses their own Spark cluster manager not available in the open source version, so Spark.jl is unlikely to work out of the box there. However, if you manage to link Spark.jl to the Databrick’s libraries instead of building the open source version, I’d expect APIs to be compatible.