VR in Julia!

General Usage
1

I’m a game developer who generally programs in c++. I’m new to the language and came up with a plan to make it really simple to call my codebase from julia. I succeeded enough on this to make it really trivial to try embedding some of my code in Julia.
I tried it out on some trivial functions and then made a little dll to export some of our rendering engine’s functionality:

class CTestApp : public I3DBaseClient
{
	typedef I3DBaseClient BaseClass;
public:
	using I3DBaseClient::I3DBaseClient;

	CLockableVector<v4f> m_vSpheres;

	void RunMainLoop() override;
	
	void SetupFrame( CRenderFrame *pFrame ) override;

	void SetupUI()
	{
		BaseClass::SetupUI();
	}

	void AddSphere( float flX, float flY, float flZ, float flRad )
	{
		with_lock( m_vSpheres )
		{
			m_vSpheres.push_back( v4f( flX, flY, flZ, flRad ) );
		}
	}
};

void CTestApp::SetupFrame( CRenderFrame *pFrame )
{
	BaseClass::SetupFrame( pFrame );
	with_lock( m_vSpheres )
	{
		for( v4f const &sph : m_vSpheres )
		{
			pFrame->Draw3DMesh( &Renderer()->m_sphereMesh, sph.AsVector3D(), { 0, 0, sph.w },  { 0, sph.w, 0 }, { 1, 1, 1 } );
		}
	}
}

void CTestApp::RunMainLoop()
{
	m_flFarClip = 1000.f;
	m_flNearClip = .1f;

	BaseClass::RunMainLoop();
}

CTestApp *g_pApp;

void ClearSpheres()
{
	with_lock( g_pApp->m_vSpheres )
	{
		g_pApp->m_vSpheres.clear();
	}
}

void AddSphere( float flX, float flY, float flZ, float flRad )
{
	g_pApp->AddSphere( flX, flY, flZ, flRad );
}

void StartRendering()
{
	Log( "StartRendering, app=", g_pApp );
	new std::thread( []{ g_pApp->RunMainLoop(); } );
}
void StopRendering()
{
	g_pApp->RequestQuit();
}


void StartDLL()
{
	// Call this before setting the application object so that we can look at the command line args
	ApplicationInit( 0, nullptr );
	CTestApp *pApp = new CTestApp( 0, nullptr );
	g_pApp = pApp;
	pApp->Initialize();
	EXPOSEFN( StartRendering );
	EXPOSEFN( StopRendering );
	EXPOSEFN( AddSphere );
	EXPOSEFN( ClearSpheres );
}

This basically provides a real time 3D Display via vulkan. I export a function to add spheres to the sphere list that Julia can call.

Then i can exercise it:

using Mmap
using Meshing
using StaticArrays


# startup the 3d renderer. It runs on it's own threads and won't block the REPL
include( "../../julia_notebooks/importcpp.jl");
ImportCPP( "expose_test2" );
StartRendering()


# simplest volumetric data format to read on earth
function LoadVolumeDataSet( fname )
    local myFile = open( fname, "r" )
    local width = convert( Int64, read( myFile, Int16 ) )
    local height = convert( Int64, read( myFile, Int16 ) )
    local depth = convert( Int64, read( myFile, Int16 ) )
    return Mmap.mmap( myFile, Array{Int16, 3}, ( height, width, depth ) )
    close( myFile )
end

beetle = LoadVolumeDataSet( "stagbeetle832x832x494.dat" );

points, faces = isosurface(beetle.-1, origin=SVector(-1,-1,-1.), widths = SVector(2,2,2.) )

for i in 1:100:size( points, 1 )
    AddSphere(points[i][2], points[i][1], points[i][3], .01 )
end

And:
One very lumpy looking beetle running in VR on an HTC Vive:

image
image664×1084 153 KB

26 Likes
2

Fantastic!

1 Like
3

It’s a bit rough around the edges, but do you know about Vulkan.jl? I’ve played around with it a little bit getting the official Vulkan C++ tutorial working, but maybe you can find some use. It’ll take work to incorporate it into an engine or write a new one (not feasible, I’m aware :slight_smile: )

An exciting showcase though!

2 Likes
4

Did you get the Vulkan tutorial working in it?

5

Yes! I got up to transforming the first triangle, at which point other work became more important :slight_smile: I sadly don’t have access to the source files I used then, but it’s basically just following the C++ tutorial and substituting julia calls. Due to the way Vulkan.jl is designed, this works basically just 1:1, except you don’t have to free the resources yourself (you can if you want to), as it’s all done through finalizers.

1 Like
6

I’d like a pure julia-based renderer but there was no way I was going to port my whole renderer…

One nice thing about the setup I have right now is that the threads running the renderer are completely independent of Julia - your Julia code could be doing heavy garbage collection, or compiling, or just spending a few minutes on math, and you still get the >90fps needed for vr in the render window.

4 Likes
7

Yeah, having control of the render thread is definitely an upside of the dual setup.

1 Like
8

I’ve finally got my Valve Index HMD working with Monado (for OpenXR support), and now I’m working on an OpenXR.jl package to wrap the OpenXR API and expose some helpers for configuring and using OpenXR (with OpenGL to start with). Once I’ve gotten the basics working, I’ll publish it along with a demo.

3 Likes
9

Any progress?

10

Not really, I’ve mostly paused on the OpenXR.jl work because of time constraints. A promising alternative approach for Linux might be using GitHub - serenity4/Wayland.jl and ~bl4ckb0ne/wxrc - sourcehut git together to access VR devices, as wxrc already provides a full compositor for VR (unlike OpenXR, which needs to tie into an existing compositor to work).

When I find time later this year, I’ll probably come back to this.

1 Like