Sony has just unveiled two new sensors,the IMX500 and IMX501. Extremely compact, they are distinguished above all by the integration of a layer dedicated to artificial intelligence. They inaugurate a new type of sensor, named Intelligent Vision Sensor by Sony. A look back at the features of these two innovative sensors.
Two new sensors signed Sony
In the smartphone and hybrid market,Sony and its Image and Sensing Solutions division are doing well by providing nearly 70% of the sensors on the market. For titre example,Sony has been the exclusive provider of iPhone photo sensors since 2011.
Sony sensors can also be found in many hybrid devices, such as nikon, Panasonic or Olympus. Faced with an ever-increasing increase in demand, the brand was struggling to honour its order book at the end honorer of commande 2019.
To establish its dominance in this market, the Image and Sensing Solutions entity has been focusing for several years on the design of faster sensors, producing better quality images capteurs and able to gather more information – which is analyzed for artificial intelligence solutions.
Over the past 2 years,the brand has unveiled several sensors, especially for autonomous cars to allow them to better “see” their environment (other vehicles, pedestrians, signs, etc.).
With these two new sensors,Sony is taking a step closer in the field of AI – and its use in everyday life. The IMX500 and 501 are thus the first sensors to integrate a computational chip and a memory chip dedicated to artificial intelligence. .
The goal: to allow AI calculations directly at the sensor level, without the use of an external processor or a remote cloud service. . They inaugurate a new range of sensors, dubbed “Intelligent Vision Sensors” by Sony.
A new architecture of photo and video sensors
The architecture of these new sensors is very different from that of the “classic” sensors. To understand why, let’s go back for a moment to the sensor-processor torque of our cameras.
Regardless of the type of camera, the sensor and processor are 2 inseparable elements.Thus, the processor converts the signal sent by the sensor into digital information.
In the case of a smartphone, the data is then sent to a remote cloud service for data analysis. l’analyse The goal: to offer a complementary service – the recognition of the faces of a photo, for example.. This is the solution used by Google or Apple within their photo galleries, for example. In some cases, THE analysis by the AI (in the cloud) can be done after the fact. When importing a photo on Facebook, Twitter or Instagram, for example.
With these two new sensors, the IMX500 and IMX501,Sony is changing this technical paradigm. Thus,the Japanese brand has associated the DSP (Digital Signal Processing, a function provided by the processor)and the ISP (Image Signal Processing).
In en practice, they differentiate themselves from “classic” sensors by placing the logical layer, dedicated to AI (as well as a memory chip) directly under the pixel layer.
From this point of view, these two new sensors take up the architecture inaugurated with the “stacked CMOS” (CMOS stacked) sensors, designed by Sony. For the record, this technology relies on the integration of a DRAM chip into the sensor to speed up the processing of information and increase the responsiveness of the device.
Smarter sensors, better tracking moving subjects
According to Sony, placing the dedicated AI layer and dedicated memory just below the pixel layer has multiple advantages. They must be able to process sensor data and perform l’IA “in-house” ANALYSIs by AI almost en instantly. Sony thus puts forward the figure of 3.1 milliseconds for the recognition of objects.
In the field, the sensor takes the photo and analyzes the scene directly to recognize the type of subject. This information is directly “translated” into metadata.
This is done without the need for an external cloud service Cloud and without the need for a very powerful processor. Thus bound, the eye and brain of the device can communicate better and be much more effective. In the field, they are designed to better detect and track subjects accurately, especially when they are in motion.
Several “formats” for the output of the image are offered to developers, depending on the desired scope:
- A “normal” photo
- Metadata (in the form of labels indicating the type of subject)
- A “raw” ISP output image (YUV/RGB)
- An image with only the “specific area of interest” that shows only the main detail of a photo (the face of a person or animal, for example).
Faster, more privacy and better energy management
As mentioned above, all calculations are done locally, without having to go through an external service. Therefore, this architecture should provide a win-win and performance- by reducing latency between the device and the cloud en server.
This should reduce bandwidth and energy consumption: instead of sending the entire image, the device would only send its metadata, just a few bytes instead of several MB.
Finally, according to Sony, this architecture would also offer better guarantees in terms of privacy. We will remain cautious on this point, given the propensity of some integrators to communicate user data to third parties.
What areas of application for IMX500 and 501?
According to information provided by Sony, the IMX500 will be offered “naked nu“. For example, it can be integrated into an entry-level or mid-range smartphone.
On the left, the Sony IMX500; on the right, the Sony IMX501. The latter includes a complete integrated circuit, allowing it to be used in different industrial areas
The IMX501, en on the other hand, has an integrated circuit. It can be deployed for robotic uses, for example. But also to analyze and track the movement of objects or people.
Thus, these sensors could prove very relevant for stores without cash. The cameras would thus be able to track customers in the aisles. And to perform a much more accurate visual “tracking” when the customer puts an object in his basket. They could also be used for marketing purposes, by more precisely identifying the typology of customers interested in such a particular type of product.
Sony IMX500 and 501: two sensors with very high added value
For now, this new architecture comes to take place on an entry-level sensor gamme of 12 Mpx (type 1/2.3 inch, or 6.17 x 4.55 mm). It is still a long way from a full-size sensor (36 x 24 mm).
However, this new architecture, totally new, comes deeply to differentiate the Sony IMX500 and IMX501 from the existing sensors. Over the next few years,Sony will probably deploy this technology on larger sensors, opening the door to smartphones – and hybrid devices? capteurs Smarter.
The integration of AI functions at the sensor level therefore offers a very high added value to Intelligent Vision Sensor sensors. AI models are fully customizable depending on the type of use being considered.
In fact,the d’application scopes of this new type of sensors are almost limitless. No doubt they will find a place in many areas,from autonomous cars to shopping malls.
With this new type of sensor,vehicles could automatically assess the driver’s attention and warn the driver in case of inattention or drowsiness. Located in autonomous cars, they could analyze the presence of pedestrians on the side of a road.
Thus, these sensors Sony IMX500 and IMX 501 are the “proof of concept“. These sensors allow Sony to inaugurate a new line of smarter sensors and demonstrate their potential.
In doing so, these two new sensors further enroll Sony in the field of artificial intelligence. They should thus enable Sony – and its Image and Sensing Solutions entity – to establish their dominance in the sensor market. And this,in a context where competition is becoming more en and more pressing.