Ultra-Low Latency: NEP’s Plan for Distributed Live OB Production

We have heard for over a year now about a radical new approach to live TV event production, deconstructing the on-site OB truck and its crew and distributing both in various ways to remote locations linked via new network connections.

But what do those network connections look like? How does this new pandemic-enforced distributed remote production workflow actually function, especially for large-scale sports productions? And what is the secret sauce, not employed before to any great degree in live TV, to enable crew on-site and their home operating colleagues to work together on the same show without getting tangled up in miscommunication and latency?

On a Haivision-hosted webinar, participants were given a close-up look at how NEP Group uses a combined workflow with EVS replay and highlight solutions, Haivision video streaming technologies, and the open source SRT protocol to enable remote replay operation with low latency high quality video.

The webinar participants were Jay Deutsch, Director Media Solutions, NEP Group; Selwyn Jans, Technical Video Solutions Engineer, Haivision; and Alex Redfern, SVP Solutions Architecture at EVS. The virtual session was hosted by Marcus Schioler, VP of Product Marketing for Haivision.

“I’m proud to be representing 4,300 technical professionals dedicated to live events, broadcast services, media solutions, engineering, operations and of course account management,” said Deutsch, referring to the NEP Group worldwide. “What we’re really here to talk about today is the difference between on-site production then [pre-pandemic], and on-site production now. We’ve been leveraging both Haivision and EVS to accomplish some of those goals.

“Talent, operators and management were all on-site in the traditional model. That has changed significantly in the past year and a half. There’s a new approach to mobile productions and we see this happening more and more every day.

“Technical operators are now at their home locations. The production mobile unit stays on-site, of course, because the technical infrastructure is still required. That didn’t go away. But the show talent is not really required to be on-site anymore. So, when we talk about on-site presence, we have really only a central critical staff on-site, which is a limited number. Everyone else, including EVS operators, can work from home with this model,” said Deutsch.

“EVS XT servers and IP Director infrastructure are typically found at the mobile unit at the production site. Core routing, multiviewers, comms, audio processing — all those requirements for a production are on-site. Now where this starts to get interesting is when we add the Haivision Makito X4 encoders and decoders, because they allow us to ultimately deliver video to the operators over public internet via SRT [Secure Reliable Transport].

“Let’s talk about the control plane,” said Deutsch. “Getting control of the EVS XT servers or IP Directors that are in the truck at the site: how do we extend that control to the remote operators? That’s a big part of this, obviously, as you need tangible control whether it’s LSM or an IP Director. Ultra-low latency monitoring is handled by Makito X4 decoders actually at the operators’ homes, to be able to deliver that ultra low latency, high quality video monitoring.

“A VPN appliance ultimately extends the resources from the truck to the home, including comms and even router panels. We have designed this to work over public internet, behind firewalls. I don’t want to say entirely ‘plug and play’, but it’s certainly as ‘plug and play’ as you can realistically get.

“Then you have talent and commentary also able to work from home, and able to contribute to productions from home, with return monitoring. You can’t have one without the other. Again, the VPN appliance extends truck resources to the home.

“In the last NFL Season there were quite a few A-level shows that were actually called from NEP remote kits with talent at their own homes. We had 24 kits in rotation during the 2020/2021 NFL Season,” he said.

Last Mile of Connectivity From the Home to the Local ISP

“How do we deliver all of the monitoring and video back to those operators? Well if we think about the mobile unit on-site, and having the ability to now create SRT, we deliver that SRT stream into our NEP managed cloud where Haivision’s SRT gateways are running, and we ultimately distribute that based on the regional location of the user.

“We can achieve minimal latency in terms of their monitoring and can adjust quality as we need to, depending on where they are and their internet speeds and connections. We’re very flexible in terms of leveraging public internet to accomplish this task.

“We want to get the highest quality we can. I wouldn’t say we have a fixed bitrate. That is going to change based on the network conditions. First and foremost what has to be done is a legit, honest internet speed check and test from that operator’s home to see what they can realistically sustain.

“It really all depends on the surrounding infrastructure; we don’t have much control over that last mile of connectivity from the home office set up to the local ISP. But we can run some tests to find out before a show: up speed, down speed, ping, jitter — all of those things are important in terms of determining what kind of bitrate you’re going to need and what your real constraints are to work within, leveraging the cloud backbone for transport but keeping in mind the challenge of the last mile. Having the best possible video quality, given network considerations, is what we are always trying to balance.

“And it’s all about reactivity,” said Deutsch. “It’s one thing to have a really nice quality program monitor, but reactivity is key when it comes to cueing up for playback, slow motion control, cutting highlights and things like that. You need a very reactive and interactive monitoring interface to do that.”

Extending the Communications Plane from the Home to the Truck

“Anyone who has taken on the endeavor of enabling remote operation has to really understand three critical planes that must be addressed. There’s the control plane, which is the tangible control for the LSM or IP Director operator — being able to actually extend that from the truck to the home operator.

“The second plane would be comms. This is critical. Without comms, you’re off on your own on an island and that just obviously isn’t going to work. So extending the comms plane from the home to the truck is critical — whether you use that as a frame from the truck and extend it via VPN, or use an intercom panel with which the operators are certainly familiar.

“And the third is the monitoring plane. An operator without the appropriate monitoring, or with very latent monitoring, is not going to be able to do their job very well. This is where Haivision comes into play, with the monitoring plane: how do we extend that plane, which exists in the truck with the routing and multiviewers on the monitor wall, to the operator at home? We need to get it there with as little latency as possible — ultra low latency,” he said.

Based in the New York Metropolitan area, Deutsch oversees all sales and client support for NEP’s growing suite of U.S. and global media solutions, including media asset management, remote/virtual editing, centralized production and cloud storage solutions. Throughout his career, Deutsch has helped clients establish systems and workflows in many client broadcast centers and studios, as well as for remote events and mobile units.

Prior to joining NEP, Deutsch spent nearly 10 years at EVS, leading technical sales efforts and also designed, tested and implemented new systems. He also spent four years in the broadcast industry as a lead workflow specialist, working with clients to create efficient workflows with remote broadcast, live event and post production media systems.