[Tech Day 2022] How ISOCELL Unlock the Future of Camera Experiences

Samsung is a leader in the development of high performance image sensors that improve the quality of photos and video. Recently, Samsung hosted its Tech Day 2022 event where they shared the latest improvements and future aspirations for various System LSI related technologies including the image sensor. Joonseok Kim, Vice President, Head of Project Management Team, discussed the pain points of smartphone photography and videography experiences and how Samsung continues to develop image sensors to improve the quality of content captured with them. Image sensor technology has come a long way, so much so that smartphone photo quality is now almost comparable to those taken with professional digital cameras. Sensor technology is key, but software for multi-frame noise reduction, HDR and AI, among other aspects, has also been critical to this evolution. How we use cameras has also changed a lot. It used to be primarily still images taken, but now, it’s all about video. Video is considered much more compelling and with popular social media apps, there’s greater consumer demand for it. Yet, there’s still a gap between the quality of still images and video, and closing that gap has proven to be difficult. Why Smartphone Video Hasn’t Yet Caught Up The first reason smartphone video quality lags is the presence of visual “noise,” especially in low light. This is caused by the lack of exposure time needed to capture enough light at 30 frames per second (fps) in video mode. The second problem is with high-dynamic range (HDR) images it’s difficult to show true colors of both an object and the background in complex light conditions like backlighting. To make this work, a smartphone needs to run multi-exposure and multi-frame fusion through system-on-chip (SoC), which uses much memory and power to be practical. The third is the lack of depth sensing capabilities. One of the best features of a DSLR camera is the “bokeh” – the pleasant-looking blur that appears when part of an image is out of focus. Using SoC a user can take photos where the subject stands out from a blurred background, but for video this method isn’t practical for the same reasons as that of the HDR’s. “We decided that we needed to solve these problems with sensors as opposed to software, and we’re taking a three-part approach. First, we’ll be making improvements in light and exposure sensitivity, which has been a big challenge especially for small and thin smartphone cameras,” said Kim. “Second, to increase the luminance of range we are working on 12-bit and 14-bit sensors, and towards even higher dynamic range sensors for superb HDR. And thirdly, we are developing ToF (Time-of-Flight) sensors that detect true image depth. Our goal is to provide a precise bokeh for smartphone video and other 3D applications.” The Advanced ISOCELL Pixel Technologies The science of creating pixels has made substantial progress in recent years. As a rule, high resolution image sensors need small, light-sensitive pixels. To capture as much light as possible, the pixel structure has evolved from front-side illumination (FSI) to a back-side illumination (BSI). This places the photodiode layer on top of the metal line, rather than below it. By locating the photodiode closer to the light source, each pixel is able to capture more light. The downside of this structure is that it creates higher crosstalk between the pixels, leading to color contamination. “To remedy such a drawback, Samsung introduced ISOCELL, its first technology that isolates pixels from each other by adding barriers. The name ISOCELL is a compound word from the words “isolate’ and ‘cell,’” Kim explained. “By isolating each pixel, ISOCELL can increase a pixel’s full well capacity to hold more light and reduce crosstalk from one pixel to another.”

Since its inception, the ISOCELL pixel technology has continuously evolved and its new generation of advancements are being applied to the latest image sensors. For instance, ISOCELL now adds optical walls made up of innovative low refractive material between the color filters. In addition to that, Kim shared “We are developing another innovative high refractive nano-structure to utilize the light of adjacent pixels to extreme levels. By applying these nano-photonics technology, we’re achieving high sensitivity that goes beyond the usual limits.” HDR Technologies for True-to-Life Colors Conventional image sensors support only 10-bit images at about 60dB dynamic range. But to capture minute details in most scenarios, sensors need to support at least 14-bit. For still images one solution is multi-exposure image capturing. This technique takes a long exposure image for the dark sections and then a short exposure image for bright sections. The two images are then merged together to create an HDR image. If you want a higher dynamic range, three or more exposures can be used. This method poses several problems for videos. First, power consumption. To capture a 30 fps HDR video, you would need to shoot at 60 fps to acquire dual exposure. To process such large image data in real-time would require a tremendous amount of power. The second problem is the challenge inherent in capturing moving objects – motion artifact. Multiple exposures means merging different images taken at different moments. Any high-speed movement makes the merging process very difficult and produces a broken or unnatural image. To help resolve this, Kim shared that, “instead we are building an in-sensor, single exposure HDR-technology. This enables the process required on SoC to be simpler, consume less power and reduce the motion artifact effect for HDR video.” A dynamic range is the ratio between the signal versus noise level. With ISOCELL technology, ISOCELL image sensors have very high full well capacity. Pixels in the newest ISOCELL image sensor have up to 70,000 electrons, allowing the sensor to reach huge signal range. Kim explained that, “To reduce noise, we perform two readouts: One with high gain to show the dark details and another with low gain to show the bright details. The two readouts are then merged in the sensor. Each read out has 10-bits. With the high conversion gain readout at 4x, it adds an additional 2-bits, producing 12-bit HDR image output. This technology is called Smart-ISO Pro also known as iDCG (intra-scene Dual Conversion Gain).”

The ISOCELL HM6 and GN5 image sensors currently available in the market comes with Smart-ISO Pro and are offering 12-bit HDR feature for sharp and detailed results. But 12-bit is still not enough. The full 14-bit or more is needed to capture eye-catching video. Samsung recently launched the ISOCELL HP3 that offers 14-bit HDR by enhanced iDCG. Furthermore, Samsung has a plan to provide 16-bit HDR sensors soon. ToF Technology for True Depth To measure the distance between a sensor and an object, Samsung has focused on ToF technology for the past five years. ToF measures the time it takes the emitted infrared light to bounce off an object and return to the sensor. Samsung’s indirect ToF (iTOF) provides good SNR, high resolution and high frame rate to create a 2D depth map and 3D point cloud.

Samsung has a plan to release a new generation of iToF sensor that has an image signal processor (ISP) integrated. The whole processing of depth information is done on the ISP within the sensor, rather than delegating to the SoC, so that the overall operation uses lower power consumption. In addition, the new solution offers improved depth quality even in scenarios such as low light environment, narrow objects or repetitive patterns. For future applications, Samsung’s ISP integrated ToF will help provide high quality depth image with little to no motion blur or lagging, at a high frame rate. All of these new innovations continue to deliver clearer and better quality imagery that can be captured with mobile devices. “With our future technology, we aim to keep changing and improving the ways our users capture images and video, and close the quality gap between the two. Our goal is to create solutions for the future through sensors that enhance human life, making it better, safer and healthier,” Kim shared to close the presentation.