STM32 ISP IQTune is the first mass-market software tool for general-purpose microprocessors designed to help developers tune their image signal processor (ISP) for a particular CMOS image sensor and its lens without needing the services of a third party or extensive expertise. In essence, the desktop application analyzes image quality, color accuracy, and statistic extraction, then talks to a program running on the embedded system (called ISP IQTune embedded application), which interfaces with the ISP’s registers to ensure proper white balance, color accuracy, and more. Available on Linux, with macOS and Windows versions coming soon, STM32 ISP IQTune supports the STM32MP2’s ISP at launch, and we are committing to adding new devices as they arrive.
The central place of the ISP
Tuning, not optional
Smartphones have indirectly and unknowingly made the public aware of the impact of ISPs on image quality. Indeed, while most consumers don’t know what ISP stands for, they know that image quality may vary significantly from one phone to the next based on the processing that happens on it. The reason is that two phones may use the same image sensor and, in some cases, the same lens, but the way the ISP processes the raw data from the CMOS significantly impacts the final product. For instance, the ISP will interpret the values from the sensor to determine colors and exposure. It may also apply filters to remove noise or smooth certain areas, among many other things.
Expertise previously required
Consequently, tuning the ISP to a specific sensor is crucial because of its central role in the image processing pipeline. The engineering challenge stems from the fact that there are so many parameters and tweaks to apply that it can be a nightmare, which is why, in many instances, small teams just rely on recommended settings from the sensor supplier. For example, knowing how to adjust the balance of colors to ensure an optimal color accuracy can demand a lot of trial and error, which may slow development operations. Additionally, not all sensor makers provide guidelines on how to tweak the ISP for their components, leaving engineers at the mercy of expensive third-party consultants.
Alternative granted
STM32 ISP IQTune is unique because it solves this issue by automating the complex task of tuning the ISP for a particular sensor and lens. Its automatic calculations mean that developers don’t have to guess or spend time experimenting with myriads of settings. Moreover, its graphical user interface makes the whole process much more accessible, even to those with no experience, thus significantly lowering the barrier to entry. Some companies may hire an expert to optimize the final image before production. However, for the first time, teams have a free alternative that may be enough for production if the image quality they obtain meets their specifications.
The pivotal role of STM32 ISP IQTune
Seeing is believing
To make ISP tuning more accessible, STM32 ISP IQTune offers a live preview. Users connect their MPU board to the PC running the application using the USB port, and the system ensures automatic feedback after saving each setting. Moreover, since the live preview is an information dump directly from the pipeline and not a simulation or approximation, developers have the assurance that what they see on their screen is accurate. Once the process is completed, the system generates a YAML file for the STM32MP2 for initialization. We are also working with the libcamera framework on Linux to help with implementation. We even offer X-LINUX-ISP, which bundles all the tools, middleware, and application examples in one software package.
The devil is in the details
Behind the scenes, STM32 ISP IQTune enables developers to tweak many aspects of the image processing pipeline with various degrees of complexity. For something more straightforward, we offer a 2A algorithm. The “2A” comes from the fact that the code controls automatic exposure and automatic white balance. That alone already has a tremendous impact on the final image and will help developers get 80% of the way to their desired results. We also offer the ability to tweak our pixel correction algorithm, which corrects artifacts that may crop up during processing.
Additionally, STM32 ISP IQTune offers much more granular tuning capabilities. For instance, it’s possible to adjust the black level, modify the resolution based on the pipeline, adapt the bayer matrix and other filters, or change contrast and gamma correction settings, among others. This is easily possible because the STM32 ISP IQTune desktop application offers a GUI that opens all these settings to the user. The software then sends the new configuration to the ISP IQTune embedded application running on the MPU, which will configure the device’s registers and then update the live preview to show the final result. Put simply, this unique solution is only possible because ST made both the hardware and software.
The supporting role of hardware
The need for customization
ST designed the ISP inside the STM32MP2. Named DCMIPP for digital camera memory interface pixel processor, it reflects our desire to prioritize and optimize for computer vision applications and other embedded systems that use a camera. It also explains why we created one. Indeed, an ISP for a consumer camera would be bigger because it would account for features like high-dynamic range, extreme noise reduction, or lens shading. And while these are essential in a digital single-lens reflex or smartphone camera, they would be useless in a machine-learning application. Conversely, ST’s ISP applies certain filters that significantly boost the accuracy of a neural network but would not satisfy a consumer taking a picture.
Working smarter, not bigger
By customizing our DCMIPP to our needs, we ultimately optimized the space the IP took on the silicon. For instance, we managed to shrink the unit responsible for the demosaicing process by skipping a few inconsequential steps in computer vision applications. Similarly, we output and perform only a few lines of statistical extraction at a time instead of storing the entire frame because it helps save space without incurring a penalty on the machine learning algorithm that won’t need the entire frame at once anyway. Put simply, while it is critical to tune the ISP, it is also essential to have an ISP tailored for computer vision, or engineers risk wasting resources and increasing inaccuracies.
Getting what we put in
Additionally, while DCMIPP and the STM32 ISP IQTune software will work with any image sensor, we also offer ST BrightSense, a portfolio of CMOS sensors. Whether a team is looking for a global or rolling shutter, or both, and whether they are looking for RGB or monochrome, we offer devices ranging from 0.38 to 5.1 megapixels. We even have automotive-grade sensors for modern computer vision applications in cars, and partners, like [Xvisio Technology, are already adopting ST BrightSense components in spatial computing systems]. This is important because while we aim to be flexible, we also know that not all sensors are created equal, and no amount of tuning will fix a poorly performing device.
