Tried: sess = SparkSession(master="local", enable_hive_support=true)
This actually makes a connection, launches the sparkui, but doesn’t register as a connection on the platform. The information in the sparkui doesn’t look normal.
When I try to run any query it just hangs and never finishes, and doesn’t error out.
Honestly, I have never run local when connecting to the cluster as it seems counter intuitive. Normally master = yarn, mode = cluster is the correct setting for the spark session.