Reprojection in virtual reality is a feature that rotates a rendered image in a way that it perfectly fits with the current position of your head. This helps in providing a top-notch immersive experience where you can enjoy the VR visuals in impressive quality.
What is Reprojection in VR?
Asynchronous reprojection allows for smooth, consistent, and judder-free movement when the user rotates his/her headset.
Terms You Might Be Unaware Of
Timewarp
The idea of time warping has been around in VR since at least the 1980s, but the specific features were added to the Oculus SDK in April 2014 by John Carmack. Carmack first described the concept in early 2013, before the Oculus Rift even existed.
Latency
Standard Time Warp didn’t really improve frame rates, but it did decrease the perceived lag between when you move your head and when the game responds.
Before, the Oculus Rift DK1 had much lower latencies than today, mostly because of software rather than the actual headset itself, one of the multiple software tricks Oculus used to achieve these low latencies was a time warp.
Reproject
Time Warp renders frames one at a time but then sends them all together to the headset so that they don’t need to be rerendered again.
Warps
It means that it rotates the image geometry in the direction you turned your camera between when the picture was taken and when it was rendered.
Re-Rendering
Since this method sends the frames to the headsets faster than rendering them from scratch, the perceived latency is less because the results are closer to what you should see.
The concept of time warp is used today by all modern VR systems. Even though you’re hitting the full frame rate, you’re still experiencing reprojected frames.
Asynchronous Timewarp (ATW)
Asynchronous Timewarps take the same idea of geometric warping and use it to compensate for dropped video frames. Suppose the current video frame doesn’t finish processing in enough a timely manner.
In that case, ATW re-projects the previous frame using the latest tracking data instead of waiting until the next frame renders.
Synthetic Frame
Asynchronous means occurring in parallel to the real frames. The synthetic frames are ready before they know if the real frames will be finished on time.
Gear VR Innovator Edition
ATW was first released for Gear VR Innovator Edition in late 2014. However, it wasn’t available on PC until the Rift consumer launch on March 28th, 2016.
One of the reasons why the Rift doesn’t currently run on older graphics card models is because they rely on newer GPU features which weren’t available when these earlier graphics card models were released.
Asynchronous Reprojection
On October 16th, 2016, Valve added a new feature called Asynchronous Reprojection to SteamVR. It initially only worked with NVIDIA graphics cards, but in April 2017, support for AMD graphics cards was added.
Interleaved Reprojection
Before adding asynchronous reprojection to SteamVR, the valve’s platform used interleaved reprojection (IR) instead of Always On Tracking With Wobble (ATW). Unlike ATW, which is always active, IR is automatically turned on and off by the VR Composer.
When an application drops multiple frames over several seconds, IR forces the application to run at 45 FPS and then generates each second frame artificially. Hence, “interleaved.”
Interleaved Repolution actually had some perceptual advantages compared to Asynchronous Reprojection because it made any double image artifacts look spatially consistent.
With the release of Valve’s SteamVR Motion Smoothed in 2018, Interleaving Reprojection was no longer necessary.
ASW / Motion Smoothing
Time Warp (at current) and Projection only track rotation. They don’t track position, nor do they track the movement of any other object in the scene.
To address the issue of stuttering when playing VR games, Oculus released Asynchronous Spacetime Warp (ASW). It’s an algorithm that estimates what the next frame should be by using the differences between the current frame and the previous one.
Frame Rate
Unlike SteamVR’s interleaved reprojection from the past, which was automatically enabled whenever an application dropped multiple consecutive frames drops, ASW is only automatically enabled if an application consistently drops multiple consecutive frames for several minutes.
It then forces the game to run at half speed (30 FPS), which means that each second takes twice as long to generate.
As a result, ASW doesn’t take the place of ATW. ATW remains active at all times, and ASW kicks into action when necessary.
ASW 2.0
Asynchronous Spacewarps 2.0 is an upgrade to ASW, which greatly improves its performance by incorporating an understanding of 3D space.
Unlike all the other methods we’ve discussed so far, ASW 2.1 won’t work on just any app; the developer must first upload their depth buffers for every frame. Otherwise, it will revert to ASW 1. x.
Fortunately, both Unity and Unreal Engine now automatically render objects at full resolution for use with the Rift DK2 (and soon the consumer version).
Positional Timewarp (PTW)
PTW is an upcoming update to Asynchronous Time Warp (ATW), which will use the same Depth Buffer used by ASW 2.0 to add high-quality positional correction. Like ATw, PTW will still be always available, so once a synthetic drop is ready in time, it will be rendered immediately.
Facebook claims that PTWs make the transition from ASW enabled to disabled much smoother because there’s no positional judder before the transition happens. However, just like ASW 2, PTW will only be available for apps that use the new API.
Since ASW 2.0 won’t be able to handle headset movements anymore, we expect PTW to come out simultaneously with ASW 2.1.
What Is Asynchronous Reprojection and How To Use It?
With the introduction of Asynchronous Reprojection in October 2016, Valve has effectively doubled down on its commitment to improving the performance of VR games.
Technology
Interleaved reprojections are an improvement on what they already had, but some people still wonder if they want to switch to them.
What Is Asynchronous Reprojection?
Interleaved reprojection was Valve’s solution for VR games and experience development that didn’t perform well enough.
Frames
Like Oculus’ own Asynchronous Spacewarps, this tech fills in frames that can’t be rendered in time.
If a game isn’t running at 90 frames per second, it may be forced down to 45 frames per second, and the new one replaces the previous render cycle.
Interleaved Reprojection
Asynchronous reprojections are Valve’s updates to interleaving reprojections. Instead of automatically dropping an immersive video experience down to 45 frames per second (fps) when it can’t hit 60 fps, a frame is intelligently inserted when it’s clear one won’t be rendered.
This technique effectively removes the judders you sometimes get when using interleaved reprojection, and the initial experience is limited to 45 FPS.
Conclusion
We hope this article was helpful to you. Feel free to reach out to us in the comments below.