(CNN) — Mark Zuckerberg watches an animated version of himself in a virtual world and tries to choose an outfit for his avatar. Zuckerberg waves his hand to change the avatar’s outfit, changing from a black shirt, pants and white sneakers to a skeleton costume and then an astronaut suit.
“Okay. Perfect,” he says finally, settling on the shirt, pants, and shoe combination for his avatar, which turns out to be the exact same outfit he’s wearing. Zuckerberg’s avatar then teleports into a virtual spaceship to meet friends, one of whom is floating, and another who appears as a large red robot.
The video, which Facebook introduced in October to help explain its name change to Meta and its new direction as the architect of the “metaverse” that isn’t real yet, is captivating. It envisions the ability to create true digital imitations of ourselves in a virtual world.
But there’s just one problem: It’s still way beyond the capabilities of Meta’s current VR. Today, if you wander around the Meta VR app, Horizon Worlds, you do it without legs or feet. While you can customize your avatar to somewhat resemble yourself, it will still have the ghostly appearance of a floating torso with just a head, arms, and hands.
The difference between the responsive, full-body avatar that Zuckerberg envisions and the typical options currently available in VR apps is more than aesthetic. Using virtual reality (VR) without meaningfully mimicking your every movement in real life may limit your ability to feel fully immersed in virtual reality services. And it makes the future of Meta, with its lofty promise of a metaverse — a far-reaching virtual realm that people can explore through digital avatars — feel that much further away. But going beyond the kind of legless avatars seen in the company’s Super Bowl ad on Sunday will be a challenge.
Meta CEO Mark Zuckerberg interacting with a full-body cartoon avatar of himself during a demo video showing the possibilities of virtual reality. Today, Meta’s Horizon Worlds VR app displays user avatars from the waist up.
Meta, who declined to comment for this article, has been considering for years how to make avatars more realistic. In an Instagram AMA (Ask Me Anything) session earlier this week, Andrew Bosworth, VP of Reality Labs and CTO of Meta, acknowledged the difficulty of the task and said the company is looking at how to solve it.
“Tracking your own legs accurately is super difficult and basically just doesn’t work from a physics standpoint with existing headsets,” Bosworth said.
Companies can track a person’s upper body reasonably well with headsets and controllers, but actual leg tracking is pretty much non-existent in VR right now, at least when it comes to the kind of VR you’d probably use in your living room. Some apps, like VRChat, allow people to have full-body avatars, but tend to use software to approximate lower-body movements; it can seem silly at best and puzzling (or even disgusting) at worst.
Despite all the progress made in recent years to perfect the technology behind VR headsets, it’s still difficult to perfectly track your legs in real life and recreate the same movements in VR without placing an array of sensors in or around them. of your body. Still, several VR experts told CNN Business they think it’s important to get your legs into virtual spaces.
“The reason you want legs is because it connects us,” said Avi Bar-Zeev, an Oakland, California-based virtual reality and augmented reality consultant and a former Apple and Amazon employee who also worked on Microsoft’s HoloLens. “It binds us to the world.”
The evolution
It’s not easy to create a sense of presence in virtual reality, but the options have come a long way.
If you bought an Oculus Rift headset in 2016, for example, you had to connect it to a powerful PC and a camera sensor on a mount that tracked the headset. At first, the headset didn’t even come with controls that tracked hand movement; initially shipped to customers with an Xbox controller and a small remote for your hands.
In recent years, VR headsets and external sensors have mostly been replaced by what’s considered a more user-friendly option: an all-in-one wireless VR headset with built-in sensors, like Meta’s Quest 2, which is the most popular by far. (Market researcher IDC estimates that VR headsets accounted for three-quarters of all recent headset shipments.)
These scopes are much more portable than their predecessors and easier to set up and use (there’s no risk of literally getting tangled up while wearing one). But tracking is largely limited to the body parts that are the most essential and can be tracked well with the headset’s sensors and a couple of controls: the head and hands.
If a company wants to realistically represent a person’s legs in VR, they need to find a way to control what those legs are actually doing in real life. Adding more sensors to the headset, like cameras on the bottom that point toward the ground, might seem like a possible solution, but, as Bar-Zeev said, it’s not that easy.
As this Meta demo video shows, the company envisions a future where you can be represented by a full body in VR (whether as a human, a robot, or anything else).
Bodies come in many shapes and sizes, and they change over time. For many of us, that means those cameras wouldn’t have a good view of our legs and feet, making it difficult to capture enough cues about your movement to infer what your legs should be doing in VR. Among other obstacles, if you were to tilt or turn your head, Bar-Zeev noted, a ground-facing camera on a headset would lose sight of the limbs it’s trying to track.
“That’s not a very reliable way to get people’s legs, and it’s a pretty bad angle to capture legs,” he said.
There’s another option for bringing legs into virtual reality that’s already on the market: physical sensors that stick to your body. HTC, which sells various virtual reality headsets like the Vive VR, mostly aimed at business users, also offers $129 trackers that you can strap to your limbs or objects (like a tennis racket) to track them in VR. For now, though, the trackers only work with headsets that connect to a PC and also require a base station.
HTC is moving in the direction of tracking more powerful and portable. It soon plans to start selling a $129 wrist tracker that can track an arm from fingertips to elbow and is designed to work with the Vive Focus 3 (a $1,300 standalone VR headset currently marketed to businesses). instead of individual users). It’s a step toward giving people more realistic hands and arms in VR than the hand controls they currently provide, and it may help achieve greater limb tracking.
Daniel O’Brien, general manager of HTC America, said that over time he hopes to be able to track more points on the body, such as feet and hips. “I think full-body immersion and tracking with an all-in-one headset is what everyone wants and everyone is working on,” said O’Brien, though like Meta’s Bosworth, he cautioned that it’s just not possible at the moment. .
Avatar bodies in Meta’s Horizon Worlds VR app are limited to floating torsos with heads, hands, and arms.
“Virtual reality is hard enough as it is”
More realistic body tracking and renderings could help drive more interest in VR, but the technological improvements needed to make that happen risk making the user experience more cumbersome, expensive and less engaging for customers. potential, at least in the short term.
“Virtual reality is hard enough already; no one is going to assume that consumers will get [sensores]said Timoni West, vice president of virtual and augmented reality at game development platform Unity.
A Super Bowl ad for Meta’s Quest 2 VR headset and Horizon Worlds app shows avatars from the waist up.
West suggests that leg movements may eventually be animated with the help of artificial intelligence (AI): movement could be predicted based on viewer data about how a person’s head moves. However, getting this right would require a lot of data on the way people walk, for example, and since it wouldn’t include leg-specific tracking, it wouldn’t be true to how each person moves. (Meta already uses AI in conjunction with sensors to track headsets, controllers, and hands.)
Depending on how realistic the legs appear, this approach can also bring up the subject of Uncanny Valleyor the creepy feeling some people get in response to human-like depictions that aren’t quite human.
Not everyone wants a virtual reality body that mimics the one they have in normal life. West prefers non-human avatars, such as an anthropomorphized stick of butter that is one of the many options in VRChat. They also predict that instead of seeing increased legs in VR, there will be fewer people asking about VR legs over time.
“The problem is that the closer it gets to reality, the more it starts to bother people who aren’t real,” West said.