Enhancing Image Quality of Optim Endoscope with 10xEngineers' Soft-ISP
Problem Statement
The client provided us with images from their current endoscope and images from their competitor’s endoscope, which were considered the target image quality. The objective was to match or exceed the target image quality while working under the constraint of fitting the upgraded pipeline in the same FPGA chip of the endoscope system.
Solution
A subjective analysis of the client’s endoscope images and an exploration of the endoscope hardware was performed, focusing on the FPGA-based ISP pipeline and firmware. This assessment identified key areas for improvement, such as sharpening, saturation, color correction, and contrast enhancement. After examining the architecture, it was determined that the modules of the bridge chip should be tuned and new modules added in FPGA.
For prototype and image quality analysis, the proposed ISP-pipeline model (Soft-ISP) was first implemented in software to achieve the desired image quality and optimize it for implementation within the current available FPGA resources. To obtain raw sensor image hardware modifications were required, so a reverse ISP pipeline was implemented to extract raw data from the endoscope’s JPG output.
Processing the endoscope image using the ISP-pipeline model significantly improved the overall image quality. Imatest results demonstrated a 45% reduction in ΔE00 color error, enhancing color accuracy, and with MTF50 increasing from 0.0864 to 0.1588 image improves image sharpness. Subjective comparisons of the before-and-after images, as well as a comparison with the competitor’s target image, showed that the same hardware resources surpassed the target image quality in detail and contrast.
Approach and Methodology
- Initial Image Analysis and RAW Extraction: An analysis of both the client’s endoscope and the target images’ sharpness, contrast, color correction matrix, and saturation enhancement was identified as key areas for improvement. Client-provided datasheets were reviewed to understand the existing architecture of the ISP pipeline implemented in the FPGA and sensor modules tuned through firmware. This assessment enabled the determination of whether the necessary modules were already implemented, required modification, or needed to be designed from scratch.
- Soft-ISP Development: To demonstrate potential improvements, an ISP-pipeline model (Soft-ISP) was developed in Python to simulate how the same sensor images could be enhanced using an upgraded ISP. This provided the client with a preview of the potential image quality improvements that could be achieved by processing sensor data with the ISP-pipeline model.
- Reverse Engineering for RAW Image Extraction: The endoscope outputs images in JPG or MP4 video format, but verifying the ISP pipeline results required sensor RAW data, which was not accessible without hardware modification. To overcome this limitation, a reverse pipeline was implemented to extract Bayer RAW images from the endoscope’s YUV-422 output. This process included YUV format decompression, color space conversion (YUV to RGB), and mosaicking the RGB image to generate a Bayer image.
- Hardware Constraints and Optimization: To meet the hardware constraint of not making changes to the existing endoscope, several sensor modules, including denoising and auto white balance, were fine-tuned through firmware. These adjustments, complemented by the FPGA-implemented Soft-ISP, ensured that the solution could be implemented within the existing FPGA resources. This upgraded ISP-pipeline enhanced image quality without requiring any hardware modifications, fully aligning with the client’s constraints while still achieving significant improvements.
Image Quality Analysis
Subjective Image Quality
For subjective image quality assessment, images and videos of an anatomical head model were captured using both the client’s and the competitor’s endoscope. The extracted raw data from the client’s endoscope was processed through the Soft-ISP to evaluate the improvements made. Subjectively, the processed images achieved improved contrast, with enhanced visibility in dark areas and a reduction in overexposed regions. Additionally, in terms of image detail and color accuracy, the processed images even surpassed the target image quality.
Objective Image Quality
For objective image quality assessment, a test setup was created using a black box with a single LED light source to replicate the endoscope’s real-world conditions. A nano-ColorChecker and micro RezChecker charts were used for quantitative analysis of image quality improvements, which was performed using Imatest.
The analysis focused on three key image quality metrics:
- Color Accuracy
- Sharpness
- Noise
Color accuracy was measured through ColorChecker analysis, achieving a 42% improvement by reducing the ΔE00 error from 15.58 to 8.99. Saturation and white balance also improved, with a 10.85% enhancement in color saturation and a 54.17% reduction in white balance errors.
Sharpness was measured using edge analysis through the MTF function, focusing on the rising slope and MTF50 values. The MTF50 value nearly doubled, increasing from 0.0863 to 0.1588 for vertical edges, while horizontal sharpness improved by 83.21%. The number of pixels required for the MTF function’s 10%-90% contrast transitions was reduced from an average of 11.2 to 2.1, indicating a significant improvement in sharpness.
Noise is estimated during color checker analysis at various exposure levels, results show how in dark areas SNR of -1 is amplified to 23.
