SERVICES

Introduction: With the development of the virtual entertainment industry, more and more TV and online programs have begun to use virtual broadcasting technology on a large scale. Recently, XR technology has begun to appear in large-scale evening parties including the Spring Festival Gala, which is refreshing to the audience.

At the immersive virtual concert of THE9 girl group "City of Virtual Reality" last month, iQiyi, which bet on virtual production, once again demonstrated the "dreamlike" visual effects of XR virtual production technology. Similar to Jay Chou's "Mojito" at this year's Spring Festival Gala, XR technology can be said to be a revolution in the expressiveness and innovation of the content industry.

▼ THE9 girl group "City of Virtual Reality" immersive virtual concert

XR virtual performance "Mojito" completed remotely and synchronously at different locations on the Spring Festival Gala

Forbes predicted in 2019 that the XR market is expected to grow 8 times in the next two years, and the market size will exceed US$209 billion by 2022.

1. What is XR?

2. Why is it so popular recently?

3. How does XR work in production?

4. What is the connection and difference between XR and virtual production?

5. What is XR virtual production?

6. XR production case

7. In conclusion

Virtual performances and live broadcasts using XR technology

XR (Extended Reality) is a term used to describe an environment or interactive behavior that combines virtual and real elements. Although XR is generally considered to include AR (augmented reality), VR (virtual reality), and MR (mixed reality), it has a special meaning in film, television, broadcasting, and live entertainment production. At present, XR has become the mainstream of the virtual broadcasting industry.

Virtual environment expansion of XR virtual stage

In the industry description a few years ago, XR was more like a general description of AR and VR technology. It can be used to refer to an integrated system that includes the virtual and real worlds, and can even be equivalent to a specific motion capture technology, AR application, or VR game. But in the field of content production, it is specifically referred to as a workflow that includes LED displays, camera tracking systems, and powerful graphics engines. It is a virtual production method that places performers in the virtual world in real time without green screens and post-production processes.

▼ Sci-fi music stage created by XR

In XR production, high-quality LED screens (usually presented as ring screens or spherical screens) simultaneously display 3D virtual environments pre-loaded by graphics engines (usually using Unreal Engine or Notch), and these LEDs also form the background for filming or live events. When a precise camera tracking system is added, the camera begins to move around the real and virtual elements that are seamlessly combined in the LED environment, giving the audience who observes through the camera's perspective an illusion of fusion and immersion.

▼ Typical layout and installation of XR virtual stage

Benefits of XR production:

Immersive and real-time production and editing process

Actors, hosts, and producers can see the virtual environment on site, allowing them to adjust and guide performances or shooting on site, reducing the time and budget spent on post-production. Simply put, it is "post-production pre-positioning". In addition, virtual production solves the crew's transportation costs and environmental waiting costs for location shooting. Simply put, you can reproduce and modify the required environment at any time without waiting for the rain or sunset.

▼ Virtual weather broadcasting using XR technology

Real on-site interactive lighting:

The lighting is provided by the LED display screen on the XR site, which helps the seamless integration of real people and virtual environments and reduces the time for further lighting adjustments. This is particularly suitable for shooting real objects with high reflectivity, which requires real environmental reflection information and lighting without the need for post-synthesis.

▼ Interaction using tracked and positioned lighting props

No color overflow and matte synthesis:

In traditional green screen shooting environments, it often takes a lot of time to perform fine matte processing and fix color overflow problems, while XR does not require matte at all, and the scene is directly out of the lens.

▼ Switching from the XR virtual environment back to the real LED screen on site

Fast lens correction:

In traditional green screen shooting, the lens correction time of the virtual camera often takes several hours, while the correction of the virtual camera tracking system on the XR camera only takes a few minutes, which greatly reduces the interruption and preparation time on site and allows shooting across multiple scenes and session scenarios in a short time.

▼ Freely assembled and expandable LED modules

In fact, many people have doubts. Many XR screen recently are associated with virtual studios, evening parties, and virtual press conferences. Is XR more used for broadcasting activities? This is a wrong view. XR is not only used in live broadcasts and other activities, it can also be used for virtual production of film and television, namely VP (Virtual Production) that we often mentioned before. Virtual production can be simply regarded as the workflow of using VR and game technologies to shoot movies.

▼ LED virtual production site of the American TV series "The Mandalorian" Similar to the XR mentioned earlier, virtual production also relies on the background provided by the LED screen as the environment, but the ground is often built with real scenery materials, and all production processes are collected through cameras, without the need for expansion outside the LED screen. The XR virtual studio environment we often see has an LED floor screen, and requires virtual expansion synthesis of the LED external environment. In addition, virtual production relies heavily on the cooperation of the art department and virtual artists to create movie-level real and virtual environment quality.

▼ Typical XR virtual production LED and equipment installation site

In fact, virtual production relying on projection is not a new thing. As early as the 1930s to 1960s in Hollywood, many movies used rear projection technology. Actors stood in front of a screen with a picture projected on it, which was often used to shoot the moving environment outside the car window under a fixed lens. But at that time, it was only used as a supplement to conventional shooting due to the lack of a camera tracking system.

▼ Rear projection technology used by James Bond in the 007 movie

Industry application of XR:

Corporate activities:

Today, companies are increasingly considering XR as a new way to solve the problem of long-distance meetings. Virtual meetings, virtual presentations and even product launches can use XR technology to enhance the immersive experience of participants and make them truly feel the culture and concept of the brand.

▼ Guests remotely transmitted to the XR LED screen through green screen synthesis

▼ Indoor meeting virtual studio installation site

Education:

XR can immerse students in remote teaching in realistic scenes, allowing students to better understand and learn knowledge in simulated virtual situations and leave a deep impression.

▼ Display texture of ground LED to AR effect in camera

Movie:

xR provides opportunities for film shooting, reducing travel time for staff and contact risks during the epidemic. In post-production, LED screens can provide faster, easier and higher-quality alternatives to green screens, shortening production cycles and making more creative visual decisions in real time.

▼ LED virtual production site of "Solo: A Star Wars Story"

TV studio:

In the studio environment, LED screens provide high-resolution clear content and lighting in real time. Although cameras are needed to collect them in the end, the real-time content allows the host to better interact with the virtual environment instead of blindly guessing and stepping on the spot as before.

▼ Virtual studio of the xite Studio team

Music screen:

VJ artists can use XR to control and present live music videos, allowing the engine to generate real-time improvisations to create unlimited narrative possibilities and let the audience feel the wonderful world beyond the scope of LED.

▼ Application of XR technology on the VJ music interactive stage

Esports:

LED screens produce real reflections and shadows, which can reduce the workload of post-production. The more avant-garde and entertaining e-sports scene is very suitable for the assistance and interpretation of XR, allowing the virtual creatures and characters in the game to appear realistically in the arena, and the audience and performers can see the CG content and interact with it more realistically.

Virtual production of Disney + Star Wars series "The Mandalorian"

In 2019, Jon Favreau, the producer who finished filming "The Lion King", used XR technology for the first time in Disney's Star Wars series "The Mandalorian", shooting half of the scenes in the LED environment. Favreau's team built a 20-foot-high LED screen (270° ring screen) to provide actors and staff with photo-realistic environments for scenes such as frozen planets and barren deserts, saving a lot of time for set construction, prop switching, and traffic transfer in a 75-foot diameter space.

What is XR virtual production?

▼ Behind the scenes of the virtual production of the first season of "The Mandalorian"

Rapper Dave's live performance at the Brit Awards:

At the 2020 Brit Awards, Dave completed a fascinating virtual and real performance next to the piano set up by XR. With the help of Mo-Sys' camera tracking system, Notch's particle effects, and Disguise's projection server, realistic 3D animations were displayed on the piano. Combined with the precise camera movement around, the melody and visual experience of the whole song reached a climax.

▼ Behind-the-scenes footage of Dave's performance

Katy Perry's performance at the American Idol finals:

The live broadcast of the American Idol finals was produced and implemented by the technologically advanced XRStudios team for XR projects, using Disguise's media server, and the special effects were produced by the Silent Partner Studios team in Notch VFX. Many people are curious about this performance