JavaCoreNLP.init() also takes care of the classpath. However, it should get called automatically and this is what actually happens on my machine.
@aether could you try to run Pkg.checkout("JavaCoreNLP") (note Pkg.update() only fetches the latest released version, while Pkg.checkout() gets the latest master)? In case it fails, please try to initialize it explicitly and check if it works.