Facebook Can Make VR Avatars Look—and Move—Exactly Like You


Research assistant Autumn Trimble sits inside “Mugsy,” one of the capture facilities Pittsburgh’s Facebook Reality Lab uses to create “codec avatars.”

“There’s this big, ugly sucker at the door,” the young woman says, her eyes twinkling, “and he said, ‘Who do you think you are, Lena Horne?’ I said no, but that I knew Miss Horne like a sister.”

It’s the beginning of a short soliloquy from Walton Jones’ play The 1940’s Radio Hour, and as she continues with the monologue, it’s easy to see that the young woman knows what she’s doing. Her smile grows while she goes on to recount the doorman’s change of tune—like she’s letting you in on the joke. Her lips curl as she seizes on just the right words, playing with their cadence. Her expressions are so finely calibrated, her reading so assured, that with the dark background behind her, you’d think you were watching a black-box revival of the late-’70s Broadway play.

There’s only one problem: her body disappears below the neck.

Yaser Sheikh reaches out and stops the video. The woman is a stunningly lifelike virtual-reality avatar, her performance generated by data gathered beforehand. But Sheikh, who heads up Facebook Reality Labs’ Pittsburgh location, has another video he considers more impressive. In it, the same woman appears wearing a VR headset, as does a young man. Their headsetted real-life selves chat on the left-hand side of the screen; on the right side, simultaneously, their avatars carry on in perfect concert. As mundane as the conversation is—they talk about hot yoga—it’s also an unprecedented glimpse at the future.

For years now, people have been interacting in virtual reality via avatars, computer-generated characters who represent us. Because VR headsets and hand controllers are trackable, our real-life head and hand movements carry into those virtual conversations, the unconscious mannerisms adding crucial texture. Yet even as our virtual interactions have become more naturalistic, technical constraints have forced them to remain visually simple. Social VR apps like Rec Room and Altspace abstract us into caricatures, with expressions that rarely (if ever) map to what we’re really doing with our faces. Facebook’s Spaces is able to generate a reasonable cartoon approximation of you from your social media photos but depends on buttons and thumbsticks to trigger certain expressions. Even a more technically demanding platform like High Fidelity, which allows you to import a scanned 3D model of yourself, is a long way from being able to make an avatar feel like you.

That’s why I’m here in Pittsburgh on a ridiculously cold early March morning, inside a building very few outsiders have ever stepped foot in. Yaser Sheik and his team are finally ready to let me in on what they’ve been working on since they first rented a tiny office in the city’s East Liberty neighborhood. (They’ve since moved to a larger space on the Carnegie Mellon campus, with plans to expand again in the next year or two.) Codec Avatars, as FRL calls them, are the result of a process that uses machine learning to collect, learn, and re-create human social expression. They’re also nowhere near being ready for the public. At best, they’re years away—if they end up being something that Facebook deploys at all. But Sheik and his colleagues are ready to get this conversation started. “It’ll be big if we can get this finished,” Sheik says with the not-at-all contained smile of a man who has no doubts they’ll get it finished. “We want to get it out. We want to talk about it.”

In the 1949 essay “The Unconscious Patterning of Behavior in Society,” anthropologist Edward Sapir wrote that humans respond to gestures “in accordance with an elaborate and secret code that is written nowhere, known by none, and understood by all.” Sixty years later, replicating that elaborate code has become Sheik’s abiding mission.

Before he came to Facebook, Yaser Sheik was a Carnegie Mellon professor investigating the intersection of computer vision and social perception. When Oculus chief scientist Michael Abrash reached out to him in 2015 to discuss where AR and VR might be going, Sheikh didn’t hesitate to share his own vision. “The real promise of VR,” he says now, both hands around an ever-present bowl of coffee, “is that instead of flying to meet me in person, you could put on a headset and have this exact conversation that we’re having right now—not a cartoon version of you or an ogre version of me, but looking the way you do, moving the way you do, sounding the way you do.”

(In his founding document for the facility, Sheik described it as a “social presence laboratory,” a reference to the phenomenon wherein your brain responds to your virtual surroundings and interactions as though they’re real. Then again, he also wrote that he thought they could accomplish photorealistic avatars within five years, using seven or eight people. While the mission remained, expectations necessarily changed. So did the name: Oculus Research became known as Facebook Reality Labs last year.)

The theory underlying Codec Avatars is simple and twofold, what Sheikh…

Follow Me

Peter Bordes

Exec Chairman & Founder at oneQube
Exec Chairman & Founder of oneQube the leading audience development automation platfrom. Entrepreneur, top 100 most influential angel investors in social media who loves digital innovation, social media marketing. Adventure travel and fishing junkie.
Follow Me

More from Around the Web

Subscribe To Our Newsletter

Join our mailing list to receive the latest news from our network of site partners.

You have Successfully Subscribed!

Pin It on Pinterest