I think spark udf is either Scala or python. Python is slow and Scala is a niche programming language so not many data scientists will adopt it. Julia is fast and easy to pick up so extending the system by Julia is the key. It’s also niche but could go big.
2 Likes