The other day I was asked to explain the difference between augmented reality (AR) and mixed reality (MR). I had just given a presentation on the topic and clearly botched communicating the difference between the two, before spiralling into the inevitable log jam of virtual and extended reality. Big mistake.
What I've found is, you can’t play hide and seek with content in AR. That’s it, there’s no way around it. A familiar character can help demonstrate this:
What’s happening here? In the image on the left, the AR camera is successfully tracking the scene and properly inserting the object in terms of perspective and lighting. But the camera can’t “see” the tree in terms of depth. All it can really do is detect a collection of points of contrast in the image raster, and when it finds enough to build a planar grid, it can build a virtual ground plane and lock to the target. All in real-time, at 60 frames per second.
On the right, the Mixed Reality camera is assisted by a depth scanner, sending light rays or lasers to read the depth of your environment, and literally reconstructing the scene in 3D so it can composite the virtual object back into the real-world, behind the tree. Still in real-time, at 60 frames per second. That’s a lot. Trust me. As a 3D artist, the potential for mixed reality is mind-blowing. Holograms! In-camera previsualization! Real-time film-making!
Currently, there are only a couple of consumer devices that can do mixed reality: the Microsoft Hololens and the yet-to-be-released Magic Leap One. Both are packing a lot of tech, and a little pricey. Which will probably keep MR at bay for the near future. Thankfully we can expect the technology to improve in quality and costs to lower.
So, what about AR? Here I’m drawing a line in the sand in terms of fundamental definition. Will this limitation be the undoing of the technology? Nope - it’s actually helping move things forward. Apple’s release of ARKit and ARCore by Google have opened up the playing field any hobbyist with a modern device. These individuals, along with the seasoned developers, are pushing boundaries both in terms challenges for the medium and solutions to them. AR on mobile devices also has an advantage - the visual rendering quality on mobile devices is superior in terms of resolution.
There are workarounds - clever animation techniques to allow for characters to reveal themselves. For instance, in Pokemon Go, a character will emerge from virtual shrubs. Or pre-configured spaces (such as a gallery) can be loaded into the virtual scene, recreating the real-world environment. In theory, using geolocation to augment a national monument or city skyline, along with some image recognition, should be possible.
Of course in certain situations, such as wide open spaces, the lack of a depth camera is a non-issue. There might also be a way around the hardware required for mixed reality as well, or at least a decent fake. There are non real-time software techniques such as photogrammetry where the camera can reconstruct a scene when provided with an array of angles. Will this exist on your phone someday? Probably. I can guarantee it won't be as effective as a hardware based, true mixed reality experience. But adding this ability will open a lot of doors.
For me, I’m eagerly anticipating a game of hide and seek on a mobile device. That will be a killer app that will open a trove of possibilities: treasure hunts, laser tag, and first person shooters. The genre of Horror. The element of surprise integrated into the real world will make mixed reality a must-have experience.