S is for Santa

Of course! Ho ho ho ho... Have you been good?

I'm pretty sure that you have, so, as is his custom, Santa is going to bring you something that you had been asking for, but also something unexpected.

Some of you have guessed it already. S is for Santa, but it's also short for Stereoscopic. And for Space, which is where we'll be going with our next adventure []-D

Actually we've just been to space, with Star Wars Outlaws. But this time we're moving to the best-looking planet in the whole universe: Pandora. And what do we need to get rid of, in order to fly there unencumbered and be able to enjoy each tiny detail of a wonderfully reconstructed virtual world?

S is for Shimmering. S is for Stop. S is for silly, which is what I'm being now instead of coming to the point.

Once again, I used to think that this would not be possible. And once again, thanks to your enthusiasm and support, I was proven wrong.

Despite many requests for it, I had been postponing indefinitely the horrible task of fixing TAA or DLSS for our VR supported titles, because it was a huge time sink that would need to be replicated from scratch for each new game. When I did it for the very first mod that was available on this Patreon, I had to spend weeks reverse-engineering the way the game computed and stored the motion vectors, literally down to each individual hair strand in the horses' manes.

It was a bloodbath, and back then I told myself I would never go through such a painful experience again, just to get rid of a few shimmering pixels around nearby objects and NPCs. Unfortunately, without precise motion vectors neither TAA nor DLSS can work properly, so it always seemed to be an all-or-nothing affair where deciding to fix a single game's antialiasing implementation to make it work in stereoscopic 3D would take me more time than modding the game itself, and crush my soul in the process :-(

Then suddenly, a few months ago, I had an idea for an algorithm that could perhaps reconstruct the correct stereoscopic motion vectors (or at least a decent approximation of them) for any game, with no need to reverse-engineer the byzantine ways in which the vectors are computed by the engine, but instead using only the information that the engine already provides to NVIDIA's DLSS libraries.

I didn't announce anything, because I didn't want to give people false hope in case the idea turned out to be a dead end. Also, as they say, ideas are a dime a dozen, and execution is everything. So I patiently set down to work on it, and as usual it took me WAY longer than I had anticipated. Many failed attempts, more sleepless nights than I care to admit, a number of apparently promising intuitions that turned out to be completely wrong. And a plethora of details to take care of, dozens of unexpected corner cases to smooth out.

But in the end, as you perhaps already saw from Brandon's video, the result was actually better than I had dared hope for. I thought there would be some incremental progress in image quality, that maybe I could make the defects a little less annoying; as a matter of fact, sometimes the upscaling is almost as good as native, and I was genuinely surprised at how much missing information it's actually possible to reconstruct without coaxing the game engine into doing any extra work.

Now, before you get carried away, I want to go into some details of what this new DLSS fix can and cannot do, and also give you some hints on how to make it function at the best of its capabilities.

Want the good news or the bad news first?

Yeah, let's start with the really bad stuff. Just like AER v2, this is NVIDIA only. If you have an AMD card, you won't be able to take advantage of the new technique with my mods. I apologize to Team Red, but I'm only one person. I know there are some third-party mods that make it possible to emulate DLSS on AMD cards using FSR, and it's possible they will work with this new tech of mine, but it is completely uncharted territory and you will have to experiment on your own. Also, though I have no affiliation whatsoever with NVIDIA and receive no money from them, I think that their cards are simply the best choice for VR. And basically everyone on the web seems to agree that DLSS is far superior to FSR in terms of image quality. AMD cards just don't have the hardware needed to compute dense optical flow in real time, and there's only so much you can do in software.

Next, the still-bad-but-not-quite-as-much stuff: this only works with DLSS. Games that only support TAA but not DLSS will not see any improvements. That is because, unlike DLSS, ordinary in-engine TAA doesn't have any standardized way of conveying the motion information, so it's not possible for me to make a fix that will work across different engines.

Moving on to the so-so news. The technology is based on a series of approximations, heuristics and a real-time reconstruction of missing data, so it is nowhere perfect. In many cases it works amazing well. But there are other situations where it fails. In particular, distant moving objects will tend to show longer trails than they do in 2D. There will be some shimmering left, especially on sharp contours. Some games may also give less satisfactory results, like for instance titles based on the UE4 engine.

Also, having DLSS work (almost) correctly doesn't mean that we no longer need to use AER. The requirement for the game to render two stereoscopic images instead of just one remains. So the motion artifacts of AER v2 (or the judder of legacy AER) will still be there, completely unchanged.

But there's good news too! First of all, the new tech is essentially free, in the sense that it has no measurable performance hit compared to normal (non stereoscopic) DLSS. And even though there might be moments in the game where it doesn't work perfectly, it is always a big improvement in quality versus the old unfixed TAA or DLSS, especially on large nearby objects like NPC faces and hair.

Moreover, the fix is more or less "universal", in the sense that it is possible for me to implement it on new games with just a reasonable amount of customization. Talking days instead of weeks. Already in this release, the tech will be immediately available for all the titles in our list that offer support for DLSS (including Elden Ring, which has no native support but can do DLSS via a third-party mod). Of course you will also get a checkbox to disable it in case you find any portions of the game on which it doesn't work satisfactorily.

And finally, the super-good news: the quality improvement that you can squeeze out with DL(S)SS is so vast that you can now actually use DLSS not just on the Quality setting, but also on Balanced and all the way down to Performance, despite the huge magnification that VR applies to everything. Together with the FOV optimizations, this means that for the very first time it's becoming possible to play a game in VR with essentially the same image quality that you would get in 2D.

I think the importance of this cannot be overstated. Now maybe the studios and publishers will finally accept that they can offer official VR ports of their games without huge investments, while at the same time keeping the graphics quality at AAA levels.

See you tomorrow for the release!