Beginner installing and trying to use JuliaPro on Windows - extremely slow experience

Thank you for these informative answers and comments. I may give a try to VSCode, but I suspect the slow plotting will be a significant problem for me. Plotting (and interactively zooming, panning) large datasets is an important part of data analysis for me. I realize they may be workarounds, super-user tricks and all, but I would have to figure this out and explain it to my students in addition to switching from one language to another one (and loosing my collection of m-files), so I donā€™t think I am ready for ā€œthe big switchā€ (and neither are my students). After all Julia is only 0.6, so I will check back later.

Slow is matter of perception and students or non-software developers wonā€™t notice that plotting in Julia is actually much faster than Excel or Google Spreadsheet.

Because unlike those commercial programs, Julia doesnā€™t have pretty loading screen and tricks to hide true loading time of application.

From a business perspective, if we want to advertise Julia as fastest numerical computation software in the world, optimization and benchmark numbers wonā€™t have much effect on public-students, corporate bosses- opinion

Maybe hiring web designer and developer to make JuliaPro prettier would help Julia to appeal to a public more :slight_smile:

(This is just some random idea, I always feel grateful for all the hard works done by Julia developers. I wish I can express those feeling in more sincere manner)

I think aaowens gave a very good response.

I think it is important to recall that Julia is still in development (we are at v0.6.2); this is probably one of the reasons why it cannot compete (in some aspects such as documentation/stability of code,ā€¦) with established programming languages.

I am a Windows user too. In my view:

  1. the package manager is slow (maybe Pkg3 will improve this). Pkg.update() can take AGES!
  2. using takes some time
  3. precompilation and compilation also takes time (ā€˜even moreā€™ (subjectively) if you are not used to it)

I want to add
a) It is quite easy to create a fast programme for non-programmers (I am a mathematician) in Julia! This is possibly the foremost reason for me to use Julia
b) I would not compare all the precompilation and start-up times, but rather judge Julia by the ā€œsubsequent plot callsā€

I have no argument for b) (I simply like Julia; I am biased)

As a side note: Interestingly the plot command actually crashes Julia for me, see below

               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: https://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _  |  |
  | | |_| | | | (_| |  |  Version 0.6.2 (2017-12-13 18:08 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-w64-mingw32

julia>

julia> versioninfo()
Julia Version 0.6.2
Commit d386e40c17* (2017-12-13 18:08 UTC)
Platform Info:
  OS: Windows (x86_64-w64-mingw32)
  CPU: Intel(R) Core(TM) i7-6600U CPU @ 2.60GHz
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY Haswell)
  LAPACK: libopenblas64_
  LIBM: libopenlibm
  LLVM: libLLVM-3.9.1 (ORCJIT, skylake)

julia> using Plots

julia> x=linspace(0,25);

julia> y=sin.(x);

julia> plot(x,y) #R6034 Runtime Error (MS Visual C++ Runtime Library)
This application failed to start because it could not find or load the Qt platform plugin "windows"
in "".

Available platform plugins are: minimal, offscreen, windows.

Reinstalling the application may fix this problem.

error

This might be because I messed up my C++ installation somehow. If anyone else experiences this, I am happy to file an issue.

The plots problem I believe is caused by Qt5 on Windows, and Python 2.7. I just had to mess with this yesterday.

You can first try setting the backend to Qt4:

ENV["MPLBACKEND"]="Qt4Agg"
using Plots

The alternative is to use Python 3. I ended up following the last post in this discussion https://github.com/JuliaPy/PyPlot.jl/issues/278

This is because Pkg2 uses a lot of small files and folders, and Windows (other filesystems as well) donā€™t like this at all. It just slows them to a crawl. Pkg3 uses larger manifest files. Done.

This is part of a larger problem that not enough precompiles. Some others have been playing with tools like SnoopCompile.jl to fix this though, and I think a much better solution can come in a 1.x

1 Like

I do worry that we will have a very hard time with inexperienced users coming from R and Python who will perceive Julia as ā€œslowā€ because it compiles. Since my background is largely in C++ and I have an easy time accepting that code must compile, I find it really frustrating that people would prefer an interpreted language simply to avoid a 500ms compile time.

The slowness of plotting however is a legitimate problem that I donā€™t really understand and needs to be fixed (i.e. this is certainly not just a 500ms issue). On the bright side I can say that I havenā€™t encountered any compile or load times outside of plotting that I considered to be truly problematic.

1 Like

Itā€™s because the backends are setup conditionally and lazily, making the code it touches not precompilable, along with the fact that precompilation isnā€™t comprehensive. Starting up a backend in Plots is literally the Julia worst case scenario right now.

2 Likes

Does this have something to do with them having non-Julia dependencies? I would think even if you load Julia modules conditionally, as long as you have the compiled binaries it wouldnā€™t be too bad.

They could be Julia dependencies and it would still happen. The code literally uses @eval to using modules in the backend setup calls (gr()), and then defines backend-specific functions as well. Thus a large portion of the global state doesnā€™t even exist at precompile time to be precompiled. The idea is to change this to modules with a dispatch structure on a small set of interface functions so itā€™ll work in a way thatā€™s compatible with precompilation.

So in other words just a normal wrapper? Is that actively being worked on?

No, linker modules like PlotsGR that instantiates some small set of dispatches and itself can precompile. Is it being worked on? Hereā€™s the issue:

while Simon thinks he has a solution by simply starting from scratch with Makie.jl

https://github.com/SimonDanisch/Makie.jl/issues/4

I think the possibility that Makie could just redo the internals entirely has slowed progress as we wait and see what happens there.

2 Likes

Iā€™d like to expand a bit on what @ChrisRackauckas said. The thing is that Plots.jl managed to become rich in terms of features and pleasant to use (syntax-wise) thanks to the effort of many contributors but still has two structural issues:

  1. Slow first plot time, due to lazy loading of the back-end plotting library (I started using Julia at a time where everything had a crazy long starting time, things have improved massively since then, so now Plots.jl is one of the few libraries that still takes long)
  2. Plots.jl interfaces with the backends in a high level and not fully documented way, which makes it difficult to maintain a backend

There have been plans to solve 1) for a while with the creation of linkers modules (such as PlotsGR). However, recently the possibility of solving 1) and 2) at the same time emerged (with Makie ). It is a riskier approach as itā€™s hard to tell whether/when Makie can reach feature parity with Plots, but if it worked it would bring many advantages to the table (it can plots many points very quickly, can -experimentally- be statically compiled thus reducing start up time drastically, is extremely interactive). On the other hand, it slowed down progress on Plots side as it makes little sense to start refactoring Plots when another more radical refactoring is taking place.

2 Likes

Iā€™m neither an expert on packages nor a user of JuliaPro, but it seems that there is a relatively simple workaround to the ā€œbeginning userā€ problem that doesnā€™t require any revision to the Julia core or packages. When a user downloads and installs JuliaPro, a dialog box pops up that says:

Precompile all packages now? (Warning: this may take > 60 minutes but will speed up package loading during subsequent JuliaPro usage. Refer to documentation.)

  • Yes
  • No

Then the user has a choice and is at least warned that if he/she chooses ā€œnoā€, there will be a delay later each time a using command is issued.

3 Likes

HI guys,

Iā€™m an old timer, but still consider myself a newbie. Itā€™s been a while since I did any serious development in Julia and just started up again.

I have a lot of sympathy for the concerns in the original post. I donā€™t remember Julia being so excruciating slow or I wouldnā€™t have been an early believer / evangelist.

My own private package takes 11 seconds to load. That is not acceptable when you are developing a package. I know you guys are pushing for 1.0 and I know the discussions are happening elsewhere (and sorry I am not very interested in reading them), but I would say you are nowhere near 1.0 if this is the experience newcomers will have. 1.0 means, to me anyway and most people I think, ā€œThis package is ready for production developmentā€. Production with this kind of miserable experience? I donā€™t think so.

Revise.jl looks awesome and can probably help a lot with the pain, but without a good user experience, I would delay the 1.0 moniker as it will bring a lot of disappointment to a lot of people.

I say this out of love for Julia. Maybe I am doing something silly, but 11 second delays between every minor code change is painful.

3 Likes

:slightly_frowning_face: Thatā€™s not a terribly helpful way to participate.

11 second delays between every minor code change is painful.

Yeah, that would be awful. Fortunately, Revise reduces that to ~0 seconds in most cases. Maybe try it?

Thanks @rdeits. You are hearing intense stress coming through. Sorry about that #startuplife :sweat_smile:

I am trying to try Revise.jl and expect it to be awesome but am having other problems at the moment :sweat_smile:

PS: If Revise is as awesome as Iā€™m hoping, itā€™d be great to have it rolled into Base as standard.

Edit: I just got Revise.jl working and it is beautiful :heart_eyes: A game changer as far as Iā€™m concerned.

2 Likes

Yeah, I think it would make a great addition to the new stdlib (although that could easily happen post-1.0 since it wouldnā€™t be a breaking change).

I like this idea.
One could add Pkg.update() to this and possibly test all installed packages during such a precompilation run

Could you at least add notifications that things are being recompiled/installed? Iā€™m a first time user of Julia. (got v0.6.2.2 Pro with Juno etc). Even after the initial 1+ minute waits for anything to happen. ā€œusing Gadflyā€ takes 9 seconds. Every time the REPL is restarted. Is it recompiled from scratch every time?

You may give a try to Gnuplot.jl, which I developed exactly for quick data exploration. It uses gnuplot as backend (which must be installed separately) and allows to exploit all its 2D/3D functionalities.

E.g., to plot an animated spiral:

@time using Gnuplot
t = 0.:0.1:100
for a in 30:360
    @time @gsp verb=0 "set view $a, $a" sqrt.(t).*cos.(t) sqrt.(t).*sin.(t) -t "w l"
end

Loading the package takes 1.2 seconds on my laptop, the first plot takes 3.6 seconds, and the remaining plots ~0.02 seconds each (1000 points are being drawn each time). For my purposes (Iā€™m an astronomer) is more than enough and I can live with it until a complete and quickly reactive plotting package for Julia will be available.