Skip to content
  • Stories
    • Create
    • Connect
    • Capitalize
    • Intelligent Content
  • Events
  • Sign Up
  • Sign In
October 11, 2022
Posted December 29, 2021

“Alter Ego:” The Promise (and Production) of Live Virtual Performance

author
Julian Mitchell
editor
media maker publishing
Alter Ego FOX avatar Unreal Engine
Featuring real singers disguised with digital avatars, FOX’s new talent show “Alter Ego” is powered by Unreal Engine. Cr: FOX Media

The unstoppable talent show genre has gone from the humiliation of wannabes from American Idol and the more kind offering of the turning chair of The Voice to the latest twist of how to inject gentler competition back into the genre — a kind of reincarnation for frustrated and inhibited singers with Fox’s Alter Ego.

Alter Ego features real singers disguised with digital avatars, accomplished with a near to real-time performance capture process. Singers in motion capture suits, witness cameras and microphones are composited together in a live audience studio, which is in the same building as the performance capture volume.

  • Kaylee Franzen as Alter Ego Aster in season 1 episode 2 of “Alter Ego.” Cr: FOX Media
    Kaylee Franzen as Alter Ego Aster in season 1 episode 2 of “Alter Ego.” Cr: FOX Media
  • Kyara Tetreault in season 1 episode 1 of “Alter Ego.” Cr: FOX Media
    Kyara Tetreault in season 1 episode 1 of “Alter Ego.” Cr: FOX Media
  • Dasharra Bridges as Alter Ego Queen Dynamite in season 1 episode 1 of “Alter Ego.” Cr: FOX Media
    Dasharra Bridges as Alter Ego Queen Dynamite in season 1 episode 1 of “Alter Ego.” Cr: FOX Media
  • Erny Nunez as Alter Ego Bernie Burns in season 1 episode 1 of “Alter Ego.” Cr: FOX Media
    Erny Nunez as Alter Ego Bernie Burns in season 1 episode 1 of “Alter Ego.” Cr: FOX Media
  • Jacob Thomsen in season 1 episode 1 of “Alter Ego.” Cr: FOX Media
    Jacob Thomsen in season 1 episode 1 of “Alter Ego.” Cr: FOX Media
  • Kobe Vang as Alter Ego Kai in season 1 episode 2 of “Alter Ego.” Cr: FOX Media
    Kobe Vang as Alter Ego Kai in season 1 episode 2 of “Alter Ego.” Cr: FOX Media

The render is simultaneously broadcast to the TV audience as a kind of hologram, although the judges and live audience are watching the show on various monitors placed on-stage.

Unreal Engine is the rendering powerhouse behind the show, and the company describes Alter Ego as a striking experience. “There are the singers, who are competing to be recognized as professional stars; the famous judges including Grimes, will.i.am, Nick Lachey, and Alanis Morissette; and even the live audience of 200 fans. There’s just one very important difference: every single competitor performing live on Alter Ego’s stage is an avatar.”

Director Sam French admits to having anxiety about relying on such complicated technology for a “shot as live” entertainment show. “I was excited but also nervous,” he said. “I had concerns over feet being on the ground and tracking and mocap. All of those things as you really are bringing two worlds together.”

The show had just 10 weeks to create the 20 avatar characters, each of whom had to be outfitted with four different wardrobe options. Each avatar had their own individual effects, as well as several visual effects tied to all the characters.

  • Jay Miah as Alter Ego Nevaeh King in season 1 episode 3 of “Alter Ego.” Cr: FOX Media
    Jay Miah as Alter Ego Nevaeh King in season 1 episode 3 of “Alter Ego.” Cr: FOX Media
  • Sarah Isen as Alter Ego St. Luna in season 1 episode 3 of “Alter Ego.” Cr: FOX Media
    Sarah Isen as Alter Ego St. Luna in season 1 episode 3 of “Alter Ego.” Cr: FOX Media
  • Sarah Isen in season 1 episode 3 of “Alter Ego.” Cr: FOX Media
    Sarah Isen in season 1 episode 3 of “Alter Ego.” Cr: FOX Media
  • Milton Patton as Alter Ego Wylie in season 1 episode 4 of “Alter Ego.” Cr: FOX Media
    Milton Patton as Alter Ego Wylie in season 1 episode 4 of “Alter Ego.” Cr: FOX Media
  • Milton Patton in season 1 episode 4 of “Alter Ego.” Cr: FOX Media
    Milton Patton in season 1 episode 4 of “Alter Ego.” Cr: FOX Media
  • Mia Cherise Hall as Alter Ego Fern in season 1 episode 4 of “Alter Ego.” Cr: FOX Media
    Mia Cherise Hall as Alter Ego Fern in season 1 episode 4 of “Alter Ego.” Cr: FOX Media

Multiple cameras are tracked in the performers’ volume space, which is the same size as the actual stage. Michael Zinman, Lulu AR executive producer and co-executive producer of Alter Ego, detailed the workflow. “All of that data from the motion capture, both the body and the face go to the engines, including the camera tracking and the lighting. From there it gets composited on the stage,” he said.

  • Jay Miah and Mariah Rosario in season 1 episode 5 of “Alter Ego.” Cr: FOX Media
    Jay Miah and Mariah Rosario in season 1 episode 5 of “Alter Ego.” Cr: FOX Media
  • Danielle Cetani and Alter Ego Phoenix Embers in season 1 episode 5 of “Alter Ego.” Cr: FOX Media
    Danielle Cetani and Alter Ego Phoenix Embers in season 1 episode 5 of “Alter Ego.” Cr: FOX Media
  • Jay Miah and Alter Ego Nevaeh King in season 1 episode 5 of “Alter Ego.” Cr: FOX Media
    Jay Miah and Alter Ego Nevaeh King in season 1 episode 5 of “Alter Ego.” Cr: FOX Media
  • Kyara Tetreault as Alter Ego Seven in season 1 episode 6 of “Alter Ego.” Cr: FOX Media
    Kyara Tetreault as Alter Ego Seven in season 1 episode 6 of “Alter Ego.” Cr: FOX Media
  • Dasharra Bridges as Alter Ego Queen Dynamite in season 1 episode 6 of “Alter Ego.” Cr: FOX Media
    Dasharra Bridges as Alter Ego Queen Dynamite in season 1 episode 6 of “Alter Ego.” Cr: FOX Media
  • Mama Yaya and Alter Ego Siren in season 1 episode 6 of “Alter Ego.” Cr: FOX Media
    Mama Yaya and Alter Ego Siren in season 1 episode 6 of “Alter Ego.” Cr: FOX Media

“The two important features for us from Unreal would be the DMX that gives us the success to actually program a show quickly and efficiently. And then certainly the composting so that you buy into the actual avatar on the stage.

“I didn’t think it was going to be possible to do. Not because you couldn’t have an avatar on stage singing in a motion body capture suit. But how do you do 10 of these performances in a day, keep it on budget, and keep it transparent, as elementary as you can through the network, and through the producers on stage — no different to a show with live-action people.”

Unreal Engine further explained the technical process. “Every performance on the show would be captured live using 14 cameras, eight of which were fitted with stYpe tracking technology. Avatar data including eye color, height, and special effects, as well as motion-capture data, lighting data, and camera data were then sent to a hub of Unreal Engines behind the main stage.

“The result was a virtual avatar that would always accurately reflect the motion-captured performance behind the scenes. When a contestant cried, the avatar would cry, too. And blush. And run their fingers through their hair. All final performances could then be seen directly through on-set monitors, making it easy for the live audience to engage with the final characters and be immersed in their journey, rather than feeling removed from the story due to the computer-generated imagery.”

  • Mia Cherise Hall as Alter Ego Fern in season 1 episode 7 of “Alter Ego.” Cr: FOX Media
    Mia Cherise Hall as Alter Ego Fern in season 1 episode 7 of “Alter Ego.” Cr: FOX Media
  • Matthew Lord as Alter Ego Wolfgang Champagne in season 1 episode 7 of “Alter Ego.” Cr: FOX Media
    Matthew Lord as Alter Ego Wolfgang Champagne in season 1 episode 7 of “Alter Ego.” Cr: FOX Media
  • Samaera Hirsch as Alter Ego Misty Rose in season 1 episode 7 of “Alter Ego.” Cr: FOX Media
    Samaera Hirsch as Alter Ego Misty Rose in season 1 episode 7 of “Alter Ego.” Cr: FOX Media
  • Chase Padgett as Alter Ego Orlando Deville in season 1 episode 8 of “Alter Ego.” Cr: FOX Media
    Chase Padgett as Alter Ego Orlando Deville in season 1 episode 8 of “Alter Ego.” Cr: FOX Media
  • Israa Darwich as Alter Ego Night Journey in season 1 episode 8 of “Alter Ego.” Cr: FOX Media
    Israa Darwich as Alter Ego Night Journey in season 1 episode 8 of “Alter Ego.” Cr: FOX Media
  • Sarah Isen as Alter Ego St. Luna in season 1 episode 8 of “Alter Ego.” Cr: FOX Media
    Sarah Isen as Alter Ego St. Luna in season 1 episode 8 of “Alter Ego.” Cr: FOX Media

Dan Pack, managing director of Silver Spoon Animation, underlined the importance of a rigid control protocol. “The great thing about Unreal Engine versus traditional rendering pipelines is that we can preview incredible changes that would normally take a lot of time in post to render… so we are able to change character hair color, eye color, texture, and control the effects, all through this DMX control panel. We pushed DMX on this show further than it has ever gone before in a real-time setting.”

“In order to pull off a show like Alter Ego, you need Unreal Engine’s DMX capabilities; this is the vocabulary and protocol we all speak and the only way to bring in a network competition show on schedule,” said Zinman. “When lighting directors can program virtual and real studio lights on one console, and the characters can be programmed on another, you run through rehearsals much faster. This is how we were able to do 10 unique performances each day.”

In tandem, Silver Spoon was tapping Unreal Engine’s Live Link capabilities to stream data from more than ten mocap instances into the avatars. With virtually no latency, Unreal Engine is able to render high-quality animation captured from the actors onto the avatars in real time — as it happens. Live Link is also used for facial animation via the dedicated Live Link Face app, which enables facial performances to be captured via an iPhone or iPad and rendered in real time in the engine. Using Live Link removed the need for Alter Ego to write a dedicated plugin for the Vicon data packets, giving the team a free-flowing way to animate their skeletal meshes.

  • Dasharra Bridges as Alter Ego Queen Dynamite, Israa Darwich as Alter Ego Night Journey, Samaera Hirsch as Alter Ego Misty Rose, and Matthew Lord as Alter Ego Wolfgang Champagne in season 1 episode 9 of “Alter Ego.” Cr: FOX Media
    Dasharra Bridges as Alter Ego Queen Dynamite, Israa Darwich as Alter Ego Night Journey, Samaera Hirsch as Alter Ego Misty Rose, and Matthew Lord as Alter Ego Wolfgang Champagne in season 1 episode 9 of “Alter Ego.” Cr: FOX Media
  • Israa Darwich as Alter Ego Night Journey in season 1 episode 9 of “Alter Ego.” Cr: FOX Media
    Israa Darwich as Alter Ego Night Journey in season 1 episode 9 of “Alter Ego.” Cr: FOX Media
  • Samaera Hirsch as Alter Ego Misty Rose in season 1 episode 9 of “Alter Ego.” Cr: FOX Media
    Samaera Hirsch as Alter Ego Misty Rose in season 1 episode 9 of “Alter Ego.” Cr: FOX Media
  • Kyara Tetreault, Mariah Rosario, Jacob Thomsen, and Chase Padgett in season 1 episode FOX  Media
    Kyara Tetreault, Mariah Rosario, Jacob Thomsen, and Chase Padgett in season 1 episode FOX Media
  • Jacob Thomsen as Alter Ego Dipper Scott in season 1 episode 10 of “Alter Ego.” Cr: FOX  Media
    Jacob Thomsen as Alter Ego Dipper Scott in season 1 episode 10 of “Alter Ego.” Cr: FOX Media
  • Kyara Tetreault as Alter Ego Seven in season 1 episode 10 of “Alter Ego.” Cr: FOX Media
    Kyara Tetreault as Alter Ego Seven in season 1 episode 10 of “Alter Ego.” Cr: FOX Media

Director Sam French sums up the potential of the tech behind Alter Ego, “Alter Ego is probably the best example of the scale of what you can do. Watching them interact on a very normal TV production schedule was incredible. And I think it just the start of what can be done.”

READ MORE: The rise of the virtual singer: the making of Fox’s Alter Ego (Unreal Engine)

  • Content Creation
  • Live Event Production
  • Intelligent Content
  • Acquisition and Production
  • Management and Systems
  • Post Production
  • Metadata Capture
  • Motion Capture / Virtual Production
  • Television / Video Production
  • Workflow Software and Solutions
  • Al / Machine Learning
  • Workflow Software and Solutions / Systems Integration
  • Animation and VFX / 3D Graphics
  • Motion Graphics
  • Workflow Solutions

Subscribe

for more content like this sent directly to your inbox:

Sign Up
  • #NABAmplify
  • More NAB:
  • NAB Amplify
  • NAB Show
  • NAB Show New York
  • Policy
  • Privacy Policy
  • Terms of Use
  • Code of Conduct
  • Cookie Policy
  • Quick Links
  • Advertising & Thought Leadership
  • NAB Amplify Press
  • FAQs
  • Technical Difficulties
  • Contact
  • Cookie Preferences
  • RSS Feed
The Angle Newsletter

Weekly editorial newsletter covering the latest content, events and more taking place on NAB Amplify.

Subscribe

© 2023 National Association of Broadcasters. All Rights Reserved.