I am not quite sure this is the best place for this discussion so fee free to let me know and I’ll move that somewhere more appropriate if needed.
Put simply, I want to know if there exists already or if some people have been thinking of developing some Julia tools to create VR content from Julia data and visualizations. Note that I haven’t looked at or thought through the feasibility or scope of what this would entail but I think it would be quite awesome, in the literal sense.
For instance, in a utopia world, I would love to be able take my GLVisualize rendering (e.g. an animation of the rotating earth with a constellation of satellites orbiting it) and magically port it to a VR environment (and I don’t mean a “Julia” VR environment – although that would be sick – but porting it to already commercially available devices)
Are you envisioning this as an offline process where you export your rendering and import it into VR, or an online process where you can have your HMD as a target device for your rendering?
The offline case, the easiest solution would probably be to export your model and import it into a game engine like Unreal and display it in VR that way. It’d mostly be just finding an export format that your game engine of choice supports.
For online it’d require a bit more work but you could link in OpenVR and expose that for GLVisualize. At the simplest level OpenVR will give you two camera matrices (one per eye) per frame and you just pass back the rendered frame for those matrices. Some sort of OpenVR.jl could be very interesting where in Pseudocode you’d be able to do something like this:
The Async thread will just on every frame query your callback and render it to the HMD. Doing it along these lines your could wrap any drawing code and as long as it returns some image for the eye it’d work fine.
Good question – I think the offline approach could be a first step but the online case sounds more appealing, of course
I like where you’re going with that pseudocode, I don’t have the time (or knowledge for that matter) to take that on but that’s definitely something to look at and keep in mind.
Wrapping OpenVR would be awesome! That indeed would be mostly it to bring GLVisualize renderings to a VR headsets - at least to get basic scene display working. Complete immersion with all the fancy tricks will be a bit more involved, but maybe OpenVR offers some help there.
We will need to pay attention to latency, but I think I’m on a pretty good way in Visualize.jl to cut down latency.
I can move the rendering almost completely to the GPU - Which is great, since it means the CPU isn’t involved as much anymore and the GC can’t actually stop what’s executing on the GPU.
We just need to figure out if we can synchronize the GC, so that it get’s scheduled while the GPU is busy rendering.
Any update on this? I’m just starting to discover Julia but I’ve been working in VR (and AR) for a couple of years now. I think the easiest way would be to have the dataset generated from your processing done in Julia to a standard exchange format e.g. JSON then visualize it in VR. There are quite a few visualization tools out there for that e.g. https://github.com/mustafasaifee42/vr-viz
What could be more interesting but also more challenging is indeed as some are suggesting in this thread to directly make Julia work in VR. Here again I can see as a bridge https://github.com/rdeits/MeshCat.jl which I think would be especially interesting for pair exploration. One person is in VR visually exploring a model or dataset while a partner would be on the computer parsing and visualization the data with Julia code.
I have written [my very first lines of Julia toda (JuliaBox first tutorials) so I won’t pretend I can help but I’m happy to explore!
I’m curious if anything has happened in this space. I’d love to be able to make some of my 3D Makie.jl visuals available on a VR headset. It’s really nice being able to view them in an electron window, zoom, pan, etc., but it would be amazing if the next time my boss asks me to do an analysis I could just show up with a VR headset and say “Here it is”
I’m waiting on my Valve Index kit to arrive (probably will be a few weeks due to their production backlog), and when it does I plan to setup a Julia OpenVR wrapper (if one doesn’t already exist), as well as some examples for using the headset.
I ordered mine on May 12th, so I guess I’m almost 2 months away from expecting to receive it. I’m really looking forward to using it, though! What platform are you using it with?
Performance is a little quirky, mostly it can jump around a bit until a game is entirely loaded into RAM. But I’m also running a lot of things through Steam Play or Proton. OTOH Half-Life Alyx is really well-done and a lot of fun (but much scarier than I expected. And Beat Saber is ridiculously smooth, so that’s a lot of fun. Oh, and Elite Dangerous is much easier in VR, since you can look all around you.
