Upgrade to CMOS cameras without changing optics

Adimec Adaptive Resolution Introduction

For many applications the optical format, often driven by an initial choice of an existing (legacy) image sensor, is a given.  These legacy systems tended to be CCDs, many of which are going the way of the cassette tape. Upgrading cameras can also mean a costly upgrade in optics. Trends in CMOS imagers are to lower read noise, increase pixel throughput and add more and smaller pixels. This opens the possibility to mimic an optical format by digital image down-scaling, i.e. Adaptive Resolution. With Adaptive Resolution, the improved performance of CMOS cameras can be utilized without changing the optics.

Image scaling

Image scaling involves a 2D re-sampling of the intrinsic sensor image. Physical pixels are digitally converted to virtual pixels of a different size.

Adaptive Resolution Feature

Here we provide a basic introduction to the Adaptive Resolution (AR) feature of Adimec cameras. It will also describe the situations where AR can be applied.

Adaptive Resolution again is an image scaling feature, which rescales the images from the sensor to a new desired image output format. In order to remove confusion between the various options that influence camera resolution we will show them side by side.  Specifically, we consider: No Scaling, Region of Interest (ROI), Zoom, Binning and Adaptive Resolution.

 

 

Scaling features off is the baseline condition of the sensor and thus has no relative advantages or disadvantages. ROI can provide for faster frame speed but you lose the full sensor FoV.  Zoom allows small regions to be enlarged to better investigate an area.  Depending on the implementation and whether the interface or sensor is the limiting factor, binning can increase frame rate, increase full well, as well as reduce noise due to averaging.  The key disadvantage to binning is that you can only compress or combine whole pixels (e.g. 2×2, 4×4).

Then we have Adaptive Resolution.  The advantage of AR is that you can keep the legacy optics in your system while profiting from the noise improvements of CMOS over CCD.  You can rescale by a ratio, allowing resizing to any format. AR calculates the partial impact of the applicable sensor pixels to get the correct output pixel. Due to ratio scaling, pixel size simulation is possible. (Example: rescale 3,45×3,45 um pixels to behave as 5,5×5,5 um pixels.)  Due to ratio scaling, full sensor FoV can be rescaled to desired Image output (Example 4096×3072 => 1920×1080). The disadvantage is that this is only digital, and that image deformation is possible: but you can scale X and Y in same factor to prevent deformation.  This may still be a bit confusing so let’s consider the following four examples.

Practical use cases for Adaptive Resolution

1. Rescale the image size to application desired image output format

To display an image on a monitor/tv the image does not need to contain more pixels than the display resolution, the best approach is to provide the display with an image that has exactly the same resolution. With AR you can reshape the image size to match your display, without losing the sensor FoV.

*NOTE: It’s possible to display a square sensor to an HD (rectangular) display, but then the image will be deformed however you can do an ROI on the sensor as well as AR to get an HD rectangular display.

Example:

Adimec TMX50: sensor 2464×2056 pixels

Sensor FoV on a HD display: 2464×2056 => 1920×1080

Compress ratio 1.283(X) : 1.904 (Y) => note that image will be deformed

Largest sensor ROI without deformation: 2464×1386

Compress ratio 1.283(X) : 1.283(Y)

2. Replace old (or discontinued) sensors

Most camera systems are built to specification, with an original camera designed in. When this camera sensor is discontinued by the manufacturer or there appears a new (better) camera sensor on the market, the designed camera system needs a lot of redesigns to be able to work with the new camera sensor.

Usually the system has a dedicated lens system and software build for the old sensor FoV, and pixel size. If a replacement sensor has another pixel format and/or new sensor size, then the lens assembly and/or software must be changed, which usually prohibits a change due to the costs involved.

With Adimec AR the lens assembly and/or interface firmware do not need to be changed (as long as the new sensor is equal or bigger)

Example:

A system is designed with a sensor: 1920×1080 resolution and 5.5×5.5 um pixels => output image 1920×1080

Replaced with:

TMX55 (4096×2160) 3.45×3.45 um pixels -> output image 1920×1080 with same FoV:

5.5/3.45 = 1.666 compression ratio

Sensor ROI = (1920*1.666)x(1080×1.666) = 3200×1800 => Image output resolution 1920×1080.

3. Create a virtual pixel format, with better performance compared to the original sized pixel

Adaptive resolution calculates the bigger virtual pixels out of the original smaller sensor pixels. This is a cumulative calculation, like with binning, and because of the calculation the virtual pixel has additional benefits:

• Cumulative full well capacity
• Less noise due to averaging noise over multiple pixels

The impact of each pixel taken into the calculation is based on its overlap in the AR calculated output pixel. So, if the pixel overlaps by 40% with the new pixel, AR uses 40% of its value in the calculation of the new pixel.

Example

TMX50 => sensor 2464×2056 pixels, 3.45×3.45 pixel size

Simulate 10×10 um pixel size:

10/3.45 = 2.9 compression ratio

When using the full sensor (2464×2056) => Image output 850×710, with 10×10 um simulated pixels

While it is beyond the scope of this blog to show how in this case the virtual full well is 114kel with a dark noise of 8.9e^–, which results in a dynamic range of 82dB.

4. Downscaling the image at camera results in lower interface bandwidth

Because these calculations are done in the camera, the output image over the camera interface has a lower image size and results in a higher possible frame rate before reaching the limit of the interface bandwidth, without losing sensor FoV.

Example:

TMX50 => 2464×2056 (8bit)

Desired image output: 1920×1080 while keeping full FoV of sensor.

(see first example) => 2464×1386 => sensor usage without deformation

If the PC/Frame grabber does the rescaling:

Interface bandwidth image data is 2464x1386x8 = 27.3 Mb/s per image or 57 fps.

If the camera does the rescaling with AR:

Interface bandwidth image data is 1920x1080x8 = 16.6 Mb/s per image or 73 fps.

This results in a 60% decrease of data bandwidth, and so a higher fps, if the interface is the limiting factor.

Summary

Adaptive Resolution is a subsampling approach to imaging that allows for the creation of virtual pixels.  It takes advantage of cumulative full well capacity and less noise due to averaging noise over multiple pixels.  In addition, you can realize savings on design costs by the overarching benefit of matching the resolution of the new sensor to an existing set of optics.