Unreal Graphics Go On Air: Real-Time Analysis for FOX Sports

Epic Games has staged a virtual event series, “Unreal Build: Broadcast & Live Events,” bringing together some of the industry’s most innovative thinkers to discuss the role of real time graphics and animation in live production.

Epic Games’ Unreal Engine is an open real time 3D tool. Creators across games, film, television, architecture, automotive and transportation, advertising, live events, and training and simulation work with Unreal to deliver cutting-edge content, interactive experiences and immersive virtual worlds.

FOX Sports CEO Eric Shanks has been instrumental in elevating the viewing experience for sports fans for many years. For the Epic Games virtual conversation he was joined by FOX Sports SVP of Graphics and Innovation Zac Fields. The session was moderated by Epic Games’ Head of LA Lab Connie Kennedy, covering how the network has used and adapted technology through the years to tell a better story — including adoption of virtual sets, AI and AR.

Eric Shanks: “Of all of our peers we are the youngest. We were created in 1994 and were literally born to do football. So football is what we live and breathe every day at FOX Sports and we’re the leader, whether it’s NFL or college for America’s favorite sport.

“Most of our time is spent around storytelling on football. Clearly we have other events like NASCAR and the Daytona 500, the World Series, the FIFA World Cup — both men’s and women’s — but it’s probably football that drives most of the innovation. And a lot of that is about showing our viewers things they can’t see on television.

“Our mission and goal is to bring people closer to the game they love, and a lot of that is through technologies and storytelling that they can’t see. And that’s where things like the yellow First Down line came from, and the glowing puck, and the driver pointers on NASCAR: things that they can’t see. And the technologies that help us show those things are what we really gravitate towards. They’ve been the foundation of almost everything we’ve done at FOX Sports for the last 28 years.”

What are some of the big trends you are seeing in the industry, where new technologies are enabling and supporting live content creation?

Zac Fields: “In terms of graphics, Eric mentioned the yellow line: I mean we wouldn’t be talking here today without real time graphics. Using those tools, whether in AR, a virtual set or other new areas of design and animation, those are examples we’re really seeing expand throughout the industry.

“NASCAR is unique. It offers a lot of opportunity, just based on how much data is there. In order to actually put all the pointers on you need to be able to track all the cars. Over the last 20 years NASCAR has really grown in terms of supplying us with all this data — and so it kind of lends itself to technology.

“A few years back we saw some demos of virtual set technology and it was at a point that we thought, “this is ready”. It was not a green screen or someone in front of a drape like it was in the past. Now you could interact. You could have a really large-scale set.

“We wanted it to feel grand, not like you were in a small space. We wanted the talent to be able to interact with it. We wanted to be able to give them new tools for analysis. And that’s when we decided to dive in and come up with a new NASCAR set — and really start using this every single day to tell stories.”

What has been the audience response to immersive storytelling, driven by real time technologies like the Unreal Engine from Epic Games?

Eric Shanks: “You can have the best camera angle, the best graphics and best audio in the world. But if the viewer doesn’t like your talent, and if your talent doesn’t buy in to telling those stories, then none of it matters.

“We have a culture that is very talent focused, very analyst driven. And our team spends a ton of time with the talent enabling them to tell those stories, crafting the graphics and technology to match their voice. If you don’t do that it’s going to fall flat and people at home aren’t going to be engaged.”

Zac Fields: “Yes it only works if the talent buys in. The NASCAR set in Charlotte was an example. From the beginning, our concern was how they were going to feel in a giant green space that is unfamiliar. They saw the potential and dove right in and were all for it, with tons of feedback and incredible ideas. And that’s one of the reasons it has been successful.

“What else can we do with other designs? Real time offers a lot of benefits for storytelling. When you combine that with a heightened sense of realism, those stories become a lot more engaging and interesting to the viewers.”

Eric Shanks: “Real time [engines] are now in every facility we have. The producers are learning in their pre-production and pre-season, figuring out the signature moments they can make. And then the other way we are scaling it is on the road. Obviously it’s more difficult on the road, with calibration and making sure it’s ready to go on the air — and that for each individual environment, the graphic design matches the event and matches the venue.

“What has really been amazing to me, and I think the biggest reason it can scale, is the calibration to cameras. I still don’t understand how a computer and a pan-and-tilt head can learn an area and stick a graphic in there in frames and make it smooth. I still don’t understand it — and that’s probably a good thing!

“That, in and of itself, is the biggest reason we can scale it. Because our biggest events are not in the studio: our biggest events are on the road. And if you’re going to get that signature element in your broadcast, you have to be able to do it on the road.”

How do you see Unreal Engine changing the way you think about content, and what are some of the benefits of introducing it to your workflows and pipelines?

Zac Fields: “I think the next level is when you start getting into customization with viewers. In terms of layering content on top of that produced broadcast, that’s going to be incredibly interesting: taking the camera data that comes in, and enhancing the broadcast further. I don’t think it’s as far away as some might think. That’s going to be really interesting and will create some unique content.

“We create a lot of pre-rendered animations, really good 3D from very polished designers. How do we get them up to speed so that some of these rendered animations could be done in real time — and what are the benefits there? So it’s not just AR, it’s some of the more traditional design elements — helping not only in render time but in customization.

“If you’ve created a really unique storytelling graphic, and someone wants to alter it a little bit for the next game, with different teams and players, to be able to change those quickly is advantageous — especially with multiple crews. If you’ve got seven or eight crews out on an NFL weekend, you can provide them with higher quality content.”

Eric Shanks: “The stories are told by the talent, but it’s how we support the stories that has changed. And the speed with which an idea can come out of a production meeting and get on the air. You used to have multiple days; then it was one day; and now, it is basically real time. And they just look better; we can see the evolution of this supporting mechanism.”

How is Epic Games helping you create more immersive audience experiences? What’s the impact for sports presenters and anchors, and the conversion to real time graphics?

Eric Shanks: “Talent interacting with technology probably started with the Telestrator, back in the day. If you remember, when the Telestrator first launched it was before FOX Sports even existed. The talent were not allowed, by union rules, to use the Telestrator. So John Madden would be standing in his booth, and to be able to draw a circle in real time he would have to ask somebody else to do it — which wasn’t really real time!

“So there was a point where John actually convinced the union that he should have an exemption to be able to do that — and so John Madden was the only talent in the entire country that had the exemption with that Telestrator pen!

“That marked the beginning. And today we have talent who literally stand in a green studio all day and you would never know it — and I think the viewers at home have gotten accustomed to knowing that it’s not real but still buying in. And I think you’ve seen, over the course of the period that FOX Sports has been around, talent really taking ownership over the tools and driving those tools to an extent — whether it’s the tools they control or the tools down in the truck. The pace of advancement in that area has picked up a lot and I think it still has a lot of room to grow, as we get further into virtual with talent and further AR with talent.”

How will game engine technology help to shape this area, looking to five years ahead?

Eric Shanks: “It’s going to rely on the ability to put graphics anywhere you want in real time perspective in telling the story — but layering AI on top of that. The fact that the machine is going to know what you want before you want it: what that’s going to do is allow you create all of those multiple variations for fans at home. And the AI for you may be different to the AI for me: you may be watching on a flat screen and I may be watching through virtual glasses. I think this is much closer than you might think.

“But with all of those extra feeds, or potential extra feeds, the one thing that needs to be constant is the best audio. Sometimes you go to these other feeds and the audio quality of the announcers is not the best. If you can get Joe and Troy’s audio with our best audio, along with those other feeds that might just be slightly customized for you, I think that’s going to be a big portion of what we’re going to see that’s different in the next five years.”

Zac Fields: “Epic is a leader in this space. As soon as we think we might have a handle on what something could be, there’s another iteration or potential to do that much more. And so from a creative standpoint I’m really excited to see how realistic things can look and how quickly we can turn things around. Things that would take so long in the past: being able to do them real time and have customization with the ability to adapt quickly — that’s exciting.

“Complexity is a challenge. When you can do so much, how do you make sure you have people who can not only create the content, but can program the content and create all the logic that needs to happen? Tying all the pieces together is challenging. We’re new in this space too: let’s say we’ve been involved with Unreal Engine for five years but I still consider that we’re new to it and we’re always learning. The challenge of creating a workflow and a pipeline is ever evolving.”

So finally Eric, do you see opportunities for crossover with other industries and other content formats such as gaming?

Eric Shanks: “The original crossover was into advertising. We’ve used a lot of our intellectual property and assets like our Cleatus robot and took it into being a centerpiece for Super Bowl advertising campaigns. We’ve created real time 3D animations our talent would interact with for a lot of our Super Bowl commercials.

“I think what you’re seeing now is crossover into gaming. And so for example, during the pandemic, we racked up I think the top five esports broadcasts on television with our virtual NASCAR event, alongside our partner iRacing. And with that event, a lot of people might have tuned in and not known it wasn’t an actual NASCAR race — except you see the virtual drivers in the virtual rig, driving on the virtual racetrack.

“And you’re seeing things like the NFL integration into Fortnite, and NASCAR itself just did a Rocket League integration. I think people are now doing it in a much smarter, more sustainable way, because the graphics and the technology allow for those brands to be integrated into environments in ways that the users buy into. The common dominator is the gaming engine, and there are a lot of conversations happening behind the scenes between different industries.”