From Julia to VR



Hi all,

I am not quite sure this is the best place for this discussion so fee free to let me know and I’ll move that somewhere more appropriate if needed.
Put simply, I want to know if there exists already or if some people have been thinking of developing some Julia tools to create VR content from Julia data and visualizations. Note that I haven’t looked at or thought through the feasibility or scope of what this would entail but I think it would be quite awesome, in the literal sense.

For instance, in a utopia world, I would love to be able take my GLVisualize rendering (e.g. an animation of the rotating earth with a constellation of satellites orbiting it) and magically port it to a VR environment (and I don’t mean a “Julia” VR environment – although that would be sick – but porting it to already commercially available devices)



Are you envisioning this as an offline process where you export your rendering and import it into VR, or an online process where you can have your HMD as a target device for your rendering?

The offline case, the easiest solution would probably be to export your model and import it into a game engine like Unreal and display it in VR that way. It’d mostly be just finding an export format that your game engine of choice supports.

For online it’d require a bit more work but you could link in OpenVR and expose that for GLVisualize. At the simplest level OpenVR will give you two camera matrices (one per eye) per frame and you just pass back the rendered frame for those matrices. Some sort of OpenVR.jl could be very interesting where in Pseudocode you’d be able to do something like this:

using OpenVR

RegisterEyeCallback( MyRenderFunction )
StartOpenVRAsync(<some HMD parameters>)

<Your logic runs here>

The Async thread will just on every frame query your callback and render it to the HMD. Doing it along these lines your could wrap any drawing code and as long as it returns some image for the eye it’d work fine.


Loic, there is a version of Paraview which works with VR setups:

I guess you then ask how that can relate to Julia! A Google search turns up:


Good question – I think the offline approach could be a first step but the online case sounds more appealing, of course :slight_smile:

I like where you’re going with that pseudocode, I don’t have the time (or knowledge for that matter) to take that on but that’s definitely something to look at and keep in mind.



@John_Hearns Oh the ParaView road looks interesting as well. I’ve played a tiny bit with VTK stuff in Julia so I could see that being of interest.



Wrapping OpenVR would be awesome! That indeed would be mostly it to bring GLVisualize renderings to a VR headsets - at least to get basic scene display working. Complete immersion with all the fancy tricks will be a bit more involved, but maybe OpenVR offers some help there.

We will need to pay attention to latency, but I think I’m on a pretty good way in Visualize.jl to cut down latency.
I can move the rendering almost completely to the GPU - Which is great, since it means the CPU isn’t involved as much anymore and the GC can’t actually stop what’s executing on the GPU.
We just need to figure out if we can synchronize the GC, so that it get’s scheduled while the GPU is busy rendering.