[All About Exynos] ① Meet the GPU · ISP Development Leaders

We all have experienced being in awe at what the latest smartphones, tablets, and PCs released each year can do. There have been numerous technological advancements since the very first smartphones were developed, making modern smartphones utterly different from their predecessors. These days, the smart devices that we hold in our hands can perform just as many tasks as most PCs. The key technology that determines the performance of smart devices is the mobile processor. This mobile processor is the system semiconductor in charge of computations and operations of multimedia in the latest mobile devices. In most cases, it comes in the form of a system-on-chip (SoC). Various semiconductor technologies are utilized in the SoC, and system blocks, such as a central processing unit (CPU), graphics processing unit (GPU), and modem, are incorporated into a single chip. Simply put, SoC is a chip that combines all the key parts that make smartphones and tablets operate. Samsung Newsroom has met with the development leaders behind the seven IPs of the Exynos mobile processor, which is considered the “brain of smartphones.” Throughout the three stories in this series, Samsung Newsroom will introduce the roles and characteristics of each type of IP, which gives a smartphone its competitiveness, and the development direction going forward. This first story is on GPU and ISP, the second story will cover CPU and NPU and the third story will cover modems, connectivity, and iSE. Exynos: The Technology Intensive System Semiconductor

In the first installment of this series, Samsung Newsroom shares the importance of the graphics processing unit (GPU) and image signal processor (ISP) in Exynos mobile processor (SoC). Before meeting with the seven IP development leaders, Samsung Newsroom first met with the SoC Development Head and Senior Vice President Mingoo Kim, who oversees system-on-chip (SoC) design in the System LSI Business. Kim first explained the fundamental reason behind the concept of SoC, in which various features are all incorporated into a single chip. “When chips are divided, it becomes difficult to manage their power consumption completely,” Kim said. “When each feature consumes power separately, the battery efficiency also decreases. In addition, the bandwidth limit and transfer time latency will affect data transmission between chips, which can lead to a reduction in performance.” Consuming minimal power is very important for smartphones because, unlike desktop computers, smartphones are not continuously supplied with power. “SoC has high efficiency due to comprehensive power control. And as a single chip, it also takes up less space within a smartphone,” Kim explained. “When all functions are performed by a single chip, performance is significantly improved.” Modern mobile phones go beyond simply sending and receiving calls and texts and have evolved to perform a multitude of advanced functions, including taking videos, mobile gaming, and accessing financial services. The SoC, which is smaller than a thumbnail, has played a big role in making this possible. As the SoC is the epitome of all main types of information technology that exist today, Kim described it as the “flower of system semiconductors.” “SoC is not an easy field to work in, but it’s an optimistic field that any engineer would aspire to become a part of,” Kim said. “The role of SoC will be limitless in future industries, such as the metaverse, autonomous driving, and 6G.”

Samsung will continue to focus on developing proprietary IPs, including GPUs, NPUs, ISPs, modems, RF, and more. Samsung strives to take another step forward in chip design to become a platform solution company. “With our top competitiveness in SoC, we aim to see Exynos recognized as the best mobile processor brand available,” Kim said. “Through this feature series, we hope the role and importance of SoC, the characteristics and advantages of Exynos as well as its developmental direction will be shared with more people.” Expanding the Possibilities of Mobile Gaming: GPU for More Enhanced Graphics

In general, graphics processing requires large-scale computation, and it is faster and more efficient if processed in parallel (parallel processing). However, the structure of a central processing unit (CPU) is specialized for fast, serial processing. As such, graphics processing with a CPU would cause important computations to be delayed due to this constant computational performance by the CPU. One example of this can be seen when playing a game. It’s possible that your character may not be able to avoid an enemy due to the touch input being delayed while graphics are being rendered on the display. To address these issues, GPU was developed. Before there were GPUs, CPUs took care of everything. However, GPU was conceived to increase efficiency as a separate accelerator for similar computations that are frequently used. In short, the CPU is like a general-purpose calculator while GPU is a large-scale parallel calculator specializing in graphics processing. This is how GPU came to be, and it is one of the key components in graphics processing. It receives commands from the CPU and displays an object’s shape, location, color, and texture on the monitor. The Xclipse 920 equipped in Exynos 2200 is the first mobile GPU Samsung jointly developed with AMD, a U.S.-based global company that specializes in semiconductor products like GPU for PC and consoles. The name ‘Xclipse’ is a combination of the ‘X’ from Exynos and the word ‘eclipse.’ The name represents Samsung’s goal to go beyond the limits of mobile gaming and carry out performance on the level of console games, ushering in a new era of gaming.

In line with this, Samsung collaborated with AMD to develop a power-efficient, console-level GPU for mobile devices. AMD’s GPUs are designed specifically for PCs or consoles, meaning they had to be redesigned to fit the mobile environment. Specifically, it was redesigned to accommodate the memory bandwidth of mobile, which is relatively limited, in addition to managing heat dissipation. “Based on our rich knowledge of low-power design, acquired from developing mobile SoCs, we were able to successfully achieve power efficiency and miniaturization in the first-generation product,” said Vice President Sungboem Park, a mobile processor design expert who oversees GPU development. “We focused heavily on minimizing heat because mobile devices do not have fans like gaming consoles, while also maintaining performance so that the frames do not lag.” The main role of the GPU in Exynos is to display objects in a 3D virtual space on a 2D smartphone screen, a role that is especially important when playing graphically demanding games on mobile devices. In particular, the Xclipse 920 is the first mobile GPU to support hardware-accelerated ray tracing (RT). Ray tracing is a technology that produces realistic lighting effects by simulating light rays reflected from 3D objects. By being accelerated by hardware rather than software, Xclipse 920 is able to perform real-time computations faster. In addition, variable rate shading (VRS) technology adjusts the amount of GPU computation depending on changes in color, shade, movement, and other variables of the objects on the screen to reduce GPU overload.

As the number of game users grows and graphics become more detailed, the development direction of GPU is becoming increasingly more important. This includes increasing the performance level to that of high-performance consoles while lowering battery consumption. By improving these two crucial areas, users are able to experience high-fidelity graphics on mobile devices similar to how they are experienced on consoles. “In general, mobile tends to lag around five years or so behind consoles when it comes to graphics technology, however, we were able to incorporate the latest console technologies in Exynos 2200 mobile processor quickly through our collaboration with AMD. The SoC is applied to the Galaxy S22,” Park said. “We plan to continue to implement other features in the RDNA series by working closely with AMD going forward.”

“As the performance of smartphones has become enhanced overall, it’s true that it’s not easy to create a performance edge that is significant enough for consumers to notice,” Park answered when asked about the future direction of mobile GPUs. “From now on, the reason for flagship smartphone users to purchase the latest smartphones will most likely be dependent on gaming performance,” he said. This means that the possibility of the further advancement of GPUs, which determines a smartphone’s gaming performance, is high. He also predicted that mobile GPUs will become even more important in the fields of AR and VR as well. “For AR, GPUs will need to be equipped in lightweight devices, such as glasses, so a low-power design is very important,” Park said. “Performance requirements are much more demanding in the field of VR, as the entire visible virtual world needs to be rendered instantly. As such, we need to satisfy various development requirements while also generating more realistic images at a much faster rate than we currently can. So, the advancement of mobile GPU is crucial going ahead and will have limitless applications,” he emphasized. Elevating the Level of Satisfaction in Camera Performance: ISP for More Natural and Vivid Images

An ISP corrects raw data transferred from an image sensor and creates a photo or video to a form preferred by the user. ISP also corrects the potential physical limitations of a camera module – comprised of an optical system and an image sensor – interpolates red, green, and blue (R/G/B) and removes noise. In addition, it carries out post-processing, such as adjusting the brightness of a video and emphasizing detailed areas. Simply put, through fine-tuning and post-processing, ISP generates the picture or video that is most desirable by the users.

In earlier smartphone models, ISP was equipped as a separate chip. However, due to market demand, embedded ISP soon became the norm. “At first, we cooperated with research institutes overseas to develop a high-performing ISP that can be used for digital cameras,” said Jongseong Choi, a project leader (PL) in the Multimedia Development Team who has been working in the video processing field for more than 20 years. “As a result, our first embedded ISP solution was utilized in the Galaxy S4’s main camera.” Since 2012, Exynos mobile processor has been contributing to a substantial advancement in smartphone cameras’ video image quality through the internalization of ISP, enabling performance at the level of digital single-lens reflex (DSLR) cameras.

With a high-performing ISP, consumers are able to enjoy higher video image quality as well as faster processing speeds. “It’s difficult to explain in specific numbers how great a photo is because photography is subjective, but we are conducting various research studies such as video evaluation and ISP tuning technique using deep learning to create natural and sharp photos and videos,” Choi explained. In addition, ISP also determines the speed of the burst rate in continuous shooting and is responsible for the fast processing of high-resolution photos and videos. The high-performing ISP equipped in the latest Exynos can process up to 200 million pixels. Up to seven image sensors are supported, and videos and images from up to four image sensors can be processed simultaneously. Also included is the ability to apply different parameters per each element, such as sky, bushes, skin, and so on by incorporating scene segmentation technology that recognizes the scenes being captured with the help of NPU. The AI feature detects and marks a person's face and adjusts the brightness, focus, and color of the video based on the face's coordinates.

When asked about the future direction of ISP, Choi answered that development will be focused on maintaining low-power usage and improving video image quality. “The data transferred from the image sensor is processed in real-time by ISP, but the amount of data is increasing exponentially,” Choi said. “So, a significant amount of power is consumed in writing the data into memory and then in retrieving it later. ISP in Exynos, however, is designed to save data in memory only once, thereby minimizing power consumption,” he emphasized. “The era of videos has arrived faster than expected and, because of this, we’re focusing on improving the video image quality,” Choi said. “In particular, we’re making efforts to increase our competitiveness by improving the video image quality, even those taken in dark, low-light environments.” _{* All images shown are provided for illustrative purposes only and may not be an exact representation of the product or images captured with the product. All images are digitally edited, modified, or enhanced.}