This blog post was originally published at STMicroelectronics’ website. It is reprinted here with the permission of STMicroelectronics.
Imagine a movie experience where the screen melts away, transporting you directly into the heart of the action. This immersive feat is now a reality at Sphere, a next-generation entertainment medium in Las Vegas that measures 366 feet tall and 516 feet wide and features a 16K x 16K resolution LED display – the highest resolution LED screen on Earth – that wraps up, over and around the audience.
To achieve the ultra-high-resolution visuals displayed on Sphere’s screen, a specialized camera system – Big Sky – was created by the Sphere team. Sphere Entertainment and STMicroelectronics worked together to develop a custom 316-megapixel camera sensor for the Big Sky camera system, enabling an unparalleled cinematic experience.
The result? Audiences are taken on a breathtaking journey through Darren Aronofsky’s Postcard from Earth, a multi-sensory cinematic experience that was filmed with this custom-built technology. In this article, we’ll explore the technical details behind ST’s custom image sensor and Sphere’s Big Sky camera system, and the impact they’re having on the future of cinema.
The technology behind camera sensors
Image sensors are solid-state devices, absorbing photons through a matrix of millions of photodetectors and converting them into an electrical signal. In a nutshell, sensors act like the eyes of a machine, converting light into electrical signals for capturing images or live scenes.
How Image Sensors Work
At their core, image sensors are made up of millions of tiny light-sensitive elements (pixels) that capture light (photons) and convert it into electrical signals (electrons). They are essential for capturing a digital image, but they’re just one piece of the puzzle. Every camera needs a lens to focus light onto the image sensor, which then transforms it into raw digital information. This data is then processed and analyzed through a processing unit (such as a dedicated image sensor processor or an application processor). Finally, the interfacing stage occurs, where image data from the processing unit is converted into a format suitable for display or further analysis on a device.
The variety of image sensors
Sensors can be broadly categorized in two ways:
- Monochrome vs Color: the first captures the world in black and white, while the latter includes pixels sensitive to red, green and blue (RGB) similar to the way the human eye captures images.
Apart from these two standard structures, there are more advanced image sensors, called “multispectral”, which can discriminate multiple spectral ranges beyond the classic RGB distinction. - Global Shutter and Rolling Shutter: in a global shutter the pixels are exposed to light globally, at the same time. Once the exposure is finished, each pixel stores its information in a buffer, which is then read out sequentially. In the case of rolling shutter, pixels are not equipped with a buffer, so the image must be captured one row after the other.
The quality of image sensors
The image sensor is the component that captures light and ultimately determines final image quality. A high-quality sensor captures a wide range of light and dark areas (high dynamic range-HDR). In bright lighting conditions, a capable sensor combines high saturation capacity, collecting a large amount of light, with the ability to capture images with short exposure times. This avoids image saturation and the creation of large white spots.
Sensitivity and dark noise are crucial in low-light conditions: an HDR capable sensor maximizes the limited available light, while dark noise must be minimal to avoid overwhelming the captured signal. Finally, high-quality images rely on advanced pixel technologies that ensure uniform images without corner shading and excellent sharpness.
What’s next
The future of image sensors is all about pushing boundaries. Manufacturers constantly strive for even better image quality, higher resolutions, and lower power consumption. This will lead to even more innovative applications, from ultra-realistic virtual reality experiences to smarter robots and medical devices.
The human eye and a camera both capture images, but their capabilities differ in some aspects. While a camera’s image sensor can vary in size and technology, typically offering high resolution and precise detail, the human eye surpasses it in dynamic range, adjusting effortlessly to varying light levels. Cameras often struggle to replicate the eye’s ability to perceive details in both bright highlights and shadowy depths simultaneously. Additionally, the human eye has a wider field of view and can adapt to changing light conditions quickly, providing a more immersive and nuanced visual experience. Despite advancements in camera technology, the human eye remains a marvel of biological engineering, unmatched in its versatility and sensitivity. To capture images rivaling the human eye, STMicroelectronics leveraged advanced imaging technology, combining superior dynamic range and low noise for high-contrast images in any lighting condition.
ST and Sphere Studios work together to create the Big Sky image sensor
History
ST worked with Sphere Entertainment Co., a premier live entertainment and media company to create the world’s largest CMOS image sensor that is incorporated into Big Sky, the camera system specifically designed to capture high-resolution content for Sphere, a next-generation entertainment medium in Las Vegas.
We were first introduced to Sphere Entertainment in 2020 through Forza Silicon, a design house we have been working with since 2012. Forza Silicon has full turnkey capabilities and ST is a strategic foundry partner for CMOS image sensors. A cooperation between ST and Sphere started in 2021 when we welcomed the Sphere team to our Crolles site in France for a site tour.
The Big Sky camera project originally aimed to overcome the challenges of using a complex array of 11 to 13 separate 8K cameras to capture content for Sphere’s display. This configuration required significant time and effort for post-production with editing and stitching together content from the camera array for a single video. Working together with ST and Forza Silicon, Sphere was able to streamline the workflow and make it more efficient. ST has a dedicated business model that can handle very complex but relatively low volume wafer projects, while Forza Silicon has extensive expertise in various areas, including CMOS image sensors, which was crucial to the success of the project.
ST custom sensor and the Big Sky camera system
Sphere Sensor Wafer (Image credits © Sphere Entertainment Co.)
ST crafted for Big Sky a CMOS image sensor that merges a 316 Megapixel resolution with a fine 4.3 micrometer pixel pitch, delivering smooth, detailed images at a rapid 120 fps. With a die size that equates to 4 dice per 12 inch (300mm) wafer, it offers an impressive 87dB dynamic range, capturing the nuances of light and shadow. Coupled with an RGB Color Filter Array with Microlens, it accurately renders colors. Fabricated on one of ST’s specialized technology nodes with Backside Illumination, this sensor is a groundbreaking step towards redefining standard imaging in entertainment and cinematography.
The camera system boasts advanced features such as a giant fisheye lens, nearly a foot (30cm) in diameter, enabling the capture of expansive, wide-angle shots, along with smaller, more portable fisheye lenses. Additionally, the camera’s sensor, optimized for Sphere’s massive screen, utilizes larger pixels to enhance light capture and detail retention, ensuring unparalleled image quality.
The Big Sky camera system featuring ST custom sensor (Image credits © Sphere Entertainment Co.)
The Big Sky camera system, featuring Sphere’s CMOS image sensor, revolutionizes film production by replacing complex 8K camera arrays with a single portable camera unit that delivers a 165-degree field of view. Sphere’s design eliminates days of editing, offering remote control and fiber optic camera connectivity. Each of the dual 32TB SSDs efficiently stores 17 minutes of uncompressed 60fps video. This system not only streamlines the filmmaking process but also enhances the creative workflow with its integrated video editing tools.
Darren Aronofsky’s Postcard from Earth
Stepping into a world where the movie screen melts away, and you’re transported directly into the heart of the action. This is the magic of Postcard from Earth, the first cinematic production ever filmed with the Big Sky camera system and part of The Sphere Experience. The film premiered at Sphere in Las Vegas on October 6, 2023.
Directed by Academy Award nominee, Darren Aronofsky, this multi-sensory cinematic experience captivates audiences with its stunning 16k resolution, exclusively showcased on the 160,000 square-foot (15,000 m2) screen at Sphere in Las Vegas. The movie harnesses the technology of the Big Sky camera system and takes audiences on a journey through all seven continents, deep into the ocean, high into the mountains, and into the eye of a storm. The Big Sky camera system allowed the production team to capture a seamless panorama of Earth’s various landscapes.
The Big Sky camera system, used for Sphere and powered by STMicroelectronics’ 316-megapixel sensor, marks a significant turning point in cinematic history. Postcard from Earth serves as an example of the immersive experiences this technology can create. With its ability to capture 16K resolution and streamline filmmaking processes, Big Sky promises to usher in a new era where audiences can transcend the screen and live the stories unfolding before them.
