A holographic substitute for live sports events has been demonstrated at the Tokyo Olympics. While it’s a rudimentary and arguably hollow experience for now, it has the potential to get spectators located anywhere in the world as close as possible to a life-size representation of the action — live.
The technology has been in development for many years at Japanese national telco NTT but the pandemic has thrown the need for it into the spotlight.
Called Kirari and described as an “Ultra-realistic Communication Technology,” the system has been trialed during rounds of competition badminton at the Olympics.
The project, claimed as a world first, projected matches held at the Olympic’s Musashino Forest Sport Plaza at a venue (Miraikan – National Museum of Emerging Science and Innovation) 22 km away.
It requires the build or utilization of a similar sized spectator arena to that where the actual event is held but could transmit a realistic goggles-free 3D reproduction of a sports match or music performance anywhere in the world.
Here’s how it works.
At the venue in the Tokyo, a number of 4K cameras capture video of the match. The feeds are composited together to enable 180-degree panoramic displays as well as ultra-wide, surround image displays.
Unlike conventional chroma keying which relies on a green or blue screen to separate the background from the foreground, NTT has developed a means of real-time image segmentation that enables the extraction of just the moving players and the shuttlecock without using chroma key. This can happen regardless of background.
The extracted video plus additional audio, spatial and environmental information including depth and camera position and orientation, are transmitted as separate streams to the remote venue.
“We have succeeded with robust and accurate subject extraction,” NTT explains, “[using] a deep learning model optimized for the space of a badminton court by adding a pseudo-depth image to the input image.”
“Unlike conventional chroma keying which relies on a green or blue screen to separate the background from the foreground, NTT has developed a means of real-time image segmentation that enables the extraction of just the moving players and the shuttlecock without using chroma key.”
This makes it possible to extract players individually even in a fast-moving sport like badminton where players are separated in the front and back sides of the court.
For the shuttle, which moves both subtly and at very high speed in video, NTT says it can accurately extract the contours of the shuttle (travelling up to 300 km/h) as it deforms into various shapes by identifying its position using a deep learning model developed specifically for badminton. Opposing players and the shuttle are extracted independently, allowing for multi-layered projection at the remote venue.
The streams are synchronized together at the remote site using a proprietary technology called “advanced media streaming and synchronization” (NTT term this Advanced MMT or MPEG Media Transport).
At the remote site an arena has been set-up to mimic the dimensions of the actual venue. This includes arranging spectator seats at a height close to that of the actual competition venue. On the “court,” two half-mirrors reflect projected images.
NTT says, “This allows for multi-layered projection of the two layers of aerial images at different depths, with the players in front on the court in front, and the players at the back across the net on the opposite side of the court, each holographically reproduced.
READ MORE: Bringing an ultra realistic viewing experience to the people of the world – Kirari! (NTT Group)
In addition, “continuous, connected and natural shuttle movement” is reproduced by dynamically changing the display position of the shuttle image between the two layers of aerial images.
It’s also possible to display live match images in the form of giant signage floating above the court, and to enhance the live viewing experience with projection mapping on the court, according to the telco.
Allied to the images, NTT is using fiber-optic cables to transmit real-time cheering to remote venues. This experiment has had to be scaled down for Tokyo given the lack of fans in stadia but the telco hopes to use “5G or 6G cellular networks to deliver a similar experience without the need for a fixed fiber-optic connection.”
Kirari was originally developed to provide people who cannot get to the venue perhaps because of disability “with a sense of presence and unity”. If it works commercially then there’s also potential for sports rights holders to sell more than the host stadium ticket capacity.
The tech was tested as far back as 2017 when it transmitted Japanese music artists onto a stage at SXSW in Austin, TX over 6000 miles away. The bandwidth required for this was 1 Gbps, the latency was 75 milliseconds, and there was apparently no packet loss.
