360° 3D Video Produktion at the Elbphilharmonie:
»Ein (jazz-)musikalisches Raumerlebnis«
The architecture of the Elbphilharmonie as musical experience sphere: Chief conductor Geir Lysne placed soloists of the NDR big band and a 25-piece choir in various places on the audience galleries. A wonderful opportunity to make spectacular use of the possibilities of 360° video and spatial audio.
Syrinx Productions filmed the concert on 13.04.2019 in 360° 3D with two Insta360 Pro 2 cameras, the second of which served as a backup. In order to bring the events on the public galleries closer to the VR-viewers, five additional cameras were used to shoot close-ups of the soloists, which were latched into the 360° video as "virtual screens" at the appropriate points.
The sound mix was designed as "spatial audio": the localization of the instruments in the headphones should always correspond to the position in the 360° video for the VR viewer/listener , regardless of the respective viewing direction, i.e. remain fixed, when the head is turned. Almost all VR playback platforms, smartphone and desktop browsers are now capable of rendering such binaural signals from an Ambisonics master in real time using viewing direction data.
To create such an audio master, Facebook provides a toolset consisting of various plug-ins, an encoder and a player. However, the demands on an audio workstation are enormous if you want to use the toolset - as in this case - to mix an 85-channel multitrack recording. For example, every track that is to be spatially positioned with the "Spatializer" - that is the 360° spherical space panner plug-in with video display - must have a 16-channel layout, even if it only contains a mono signal, since the master bus is in Third-Order-Ambisonics (TOA) format.
Video producer Jürgen Klimmeck explains: "The Elbphilharmonie in Hamburg is ideal for 360° production because the stage is located in the centre of the hall and the audience sits around it. 20 years ago we pioneered the promising medium DVD-Audio for 5.1 music documentation and have been producing 3D videos since 2010.
But only the combination of 360° VR video in 3D with spatial audio creates the perfect illusion to be personally present at an event - immersively in the truest sense of the word! When you put on your VR headset, you won't fast forward, zap or even iron as you would on a mobile phone or TV.
The NDR publishes 2 pieces from this program via its Facebook page and the website of the NDR big band. A longer version with 3 pieces and more of the space effects can be seen and heard on the Syrinx-Youtube channel. The best way is of course to play them back via a VR headset, but the spatial audio effect can also be experienced on a mobile phone (swivel!) or PC (rotate the picture with the mouse!) with headphones.