Sorry for the potentially opinionated question here, but I figured that the folks on Julia discourse have the best insights into this question.
So I want to start using more of the Julia multi-threading and GPU packages like CUDA.jl to improve the performance of some of my packages. I have worked through some CUDA.jl tutorials/demos and even the task-based parallelism for multi-threading before; however I have not had a chance to really apply these tools to actual projects. But things are changing a bit now, and parallelism is becoming important for me.
I was really wondering if I need to learn more C++ programming in order to better understand and work with Julia CUDA.jl, KernelAbstractions.jl, and multi-threading packages? I have written small snippets of C++ and Fortran code before, but nothing much beyond basic homework assignment level.
I come from the Stats and ML world, but nowadays I have to do more optimization and numerical computing. So the parallelism I need is for solving say constrained non-convex optimization problems or optimal control problems. My code is mostly writing mathematical solvers, so I would need to write CUDA.jl stuff as well as KernelAbstractions.jl stuff for Nvidia GPUs and AMD CPUs/GPUs.
So I was just wondering if I should invest the time to learn more C++ if I am working on these parallelism tools in Julia? Like how deep do I need to get into C++, meaning how much is enough. I don’t think I have to go very deep into C++ given my tasks, but I don’t have enough experience to know the answer.
I don’t think that’s necessary. For CUDA.jl at least, it’s perfectly doable to take some introductory CUDA material (like the “CUDA By Example” book, or something more recent), and implement it using Julia and CUDA.jl. The kernel programming language in CUDA.jl is very similar to that in CUDA C, so there’s no need to use C or C++ for that. Sometimes you may have to check the CUDA.jl API reference (or grep the repository) to find the exact counterpart of a CUDA C function, but that should be relatively rare. Most of the learning is about the parallel nature of the GPU, and how to use the hardware effectively, which is identical in both languages.
I think you’re seriously underestimating how unhelpful reading the CUDA C docs is for people who don’t know C / C++.
Reading documentation is already confusing enough when you don’t understand the subject matter of the documentation well, but adding an additional element of confusion from not understanding the syntax causes compounding problems. Whenever you don’t understand a statement or example, it’s never clear if you don’t understand the content, or some syntatic quirk of C / C++ just threw you off.
The CUDA.jl docs have improved a lot in the past few years, but they still feel very much like the CUDA C documentation is a required supplementary material.
The insights here are very helpful. So I like @maleadt idea of using a book like “CUDA By Example” and working backward from there. I know that book is from like 2011 or something, but I can use that book as well as some others to work backwards and see what elements of the C or C++ languages I need.
@Mason I hear what you are saying as well. I am not averse to revisiting C and C++ programming again. The basic language is simple enough. The hard parts are getting used to the error messages, etc. My concern is really getting hung up on parts of C or C++ that take time to learn and that I won’t really need to use in practice. For example, I am pretty sure I will never do string manipulation in C or C++. Those libraries and containers are pretty cumbersome to learn, and in generally I will do tasks like that in Julia. But other things I am not sure about. So do I really just need C and not C++? Is that enough. Do I need to worry about things like Metaprogramming and Templates, if I am really only using C/C++ for GPU stuff? I have used C++ objects before, so it is not a big deal–I just have to remember the boilerplate stuff similar to python. I don’t really know which parts of C and/or C++ are important to learn, hence I am posting the question.
Then there is the added complexity of the different C++ versions like C++11, C++14, … C++20.
I am trying to just figure out a minimal syllabus that I would need to cover.
I am thinking that I can watch some videos to refresh my memory on the basics of C and C++, as well as some of the intro CUDA videos. Then I can try to implement some demos in both C/C++ as well as CUDA.jl. I will start with implementing some simple optimization solvers in C/C++ and then in CUDA.jl, just for good practice. Line search, Newtons method, etc. Of course there are good package for optimization like IPOPT, SNOPT, etc., as well as Optimization.jl but depending on the problem it helps to be able to write and benchmark some of those solvers myself.
If both of you have any further suggestions please pass them along.
Yeah, perhaps. I guess I’m expecting a minimal understanding of C/C++, which should be sufficient as the examples in the CUDA documentation generally consist of straightforward, syntactically-simple code (i.e., no advanced C++ shenanigans).
I do agree that it would be much better to newcomers if we had better documentation and tutorials written in Julia, but for now it is indeed mostly expected that you get your CUDA documentation somewhere else.
No worries @maleadt , I don’t have a problem reviewing all of the C and C++ stuff I have done in the past. I was looking over that material today and it is not that bad. It takes a little to remember pointers and templates and such, but the rest is not too bad. I am actually having some trouble finding like nice or complete implementations of some of the optimization algorithms, backtracking line search, Newton with trust regions, etc. Kochenderfer’s book is good but the algorithms are not complete. I was looking at the Numerical Recipes book too by Press and Teukolsky, etc. That code is pretty messy, and uses a lot of advanced C++ features, but more complete. It is not hard to figure out myself, but I would have liked some complete implementations to benchmark or compare against.
Point is that yeah, I think some familarity with C/C++ is not too much to ask. But we definitely need to write more posts or such about how to convert algorithms from C/C++ to CUDA.jl or KernelAbstractions.jl. It does not seem hard, but I think it probably intimidates people because there are so many pieces to keep track of.