Update, May 17, 2023
Since publishing this blog post, ST released the ISM330ISN. Users can find it on the STEVAL-MKI233KA and take advantage of this step-by-step guide on our Community to experiment with it. The device is similar to the ISM330IS but is optimized to run anomaly detection on NanoEdge AI. The walkthrough covers platform selection and how to log data, benchmark results, and how to deploy the resulting library. Consequently, even those with no experience in machine learning applications can obtain results and learn from this particular setup. And to go even further, we provide an on-demand webinar and a whitepaper to help teams better understand the challenges that this technology can solve and how to obtain results faster.
The SENSOR + TEST 2022 conference took place last month in Nuremberg, Germany, where attendees got to check out the ISM330IS, the first sensor with an intelligent sensor processing unit (ISPU). ST announced the technology in early 2022, and its demonstration is highly symbolic as it testifies to its soon availability. In a nutshell, the ISPU is a C-programmable embedded digital signal processor (DSP) capable of running machine learning and deep learning algorithms. It is thus the next evolution of AI at the Edge, or what ST calls “The Onlife Era”. Indeed, the ISM330IS includes a floating point unit for single-bit precision computations, a first in a motion sensor.
Table of Contents
From an Idea to a New Generation of Sensors
What Challenges Did ST Overcome?
It is critical to look at significant innovations over the past four years to understand how we got here. It started when ST published a paper studying the feasibility of a machine learning core inside an inertial sensor. Prior to this research, sensors only collected data. As a result, all computations had to take place on the microcontroller. The reasons behind this system are relatively straightforward. Inertial sensors are small and low-power devices. Adding a powerful processor would not only run afoul of those constraints but create significant integration and manufacturing challenges. However, despite combining the DSP and The accelerometer along with the gyroscope, ST did not compromise processing capabilities, memory storage, or sensing accuracy.
What Did the First Sensor With a Machine Learning Core Accomplish?
The 2018 ST paper was groundbreaking because it solved these challenges to promote what would later be called the LSM6DSOX, the first inertial sensor ever to house eight decision trees in parallel to run machine learning algorithms. As a result, it became possible to run applications locally at a fraction of the power consumption. New applications began to crop up, especially after integrating the device into the SensorTile.box. For instance, it was possible to create a baby crying detector for infants left in cars. Similarly, two University College London projects made waves by automating sit-stand tests and creating a more accessible digital stethoscope.
What Other Sensors Got a Machine Learning Core?
The LSM6DSOX was also the beginning of a new community of developers. ST provided a machine learning core repository on GitHub and made the Unico GUI software tool more accessible to assist programmers looking to take advantage of the machine learning core in the LSM6DSOX. Moreover, we released even more powerful sensors. The LSM6DSRX saw better performance to target more demanding applications like virtual reality headsets. We also launched the LSM6DSV16X with an enhanced machine learning core and a better performance-per-watt ratio for systems with more stringent power constraints. Consequently, ST’s sensors with a machine learning core partly contributed to the advent of the next automation age , and the ISM330IS hansels a significant new chapter in this saga.
From a New Processing Core to New Applications
What Is the ISPU Made Of?
The ISPU of the ISM330IS offers 8 KB of RAM for data and 32 KB of RAM for the application. It also has a 32-bit RISC Harvard architecture running at 10 MHz, a four-stage pipeline, a floating-point unit, and a set of 16-bit length instructions optimized for neural network processing. Moreover, while the processor can throw an interrupt in only four clock cycles (Arm Cortex cores usually do it in 15), it can also process a 16-bit multiplication in one cycle. It uses SPI or I2C to communicate with the host MCU. Developers only need to load C code to the ISPU’s volatile memory at startup from the host processor.
The FPU enables applications to run inferences at the edge with greater flexibility. Once conditions are met, the program throws an interrupt to the microcontroller. Similarly, the architecture enables the ISPU to improve performance compared to the previous generation of devices while still operating at microwatt levels. It is thus a significant leap from the decision tree of the previous machine learning core and a more efficient system. Furthermore, despite the presence of significant computational power, the ISM330IS still fits inside a market-standard LGA package measuring 3 mm x 2.5 mm x 0.83 mm. As a result, designers don’t have to significantly alter their PCB to adopt the new part.
How Does the ISM330IS Stand Out?
Machine learning applications are increasingly common and many associate them with high memory requirements, cloud computing servers, or highly-parallel GPU architectures. And many programs do require expensive computational operations, making them incompatible with edge processing. However, companies also know that not all deep learning systems need this power. As a result, running inferences on mobile devices, such as smartphones, or on local industrial installations, is gaining traction. Image recognition, anomaly detection, and predictive maintenance all demand reliable AI performance in a tight power envelope. Similarly, cameras for wireless home security systems use AI to recognize faces or pets, and an ISPU could offer the intelligence for always-on displays on mobile systems.
The ISM330IS is a new answer to this challenge as it only requires 0.59 mA in high-performance mode. Comparatively, the ISM330DHCX needs up to 1.5 mA under similar conditions. The latter has a more powerful gyroscope, among other things, which does partly explain the difference. However, the numbers also show the optimizations that went into the new device and the efficiency of the processing core. Indeed, it is rare for low-power microcontrollers to have an FPU as they often demand a lot of energy. Yet, the ISM330IS still manages to keep its power consumption low enough to fit battery-powered systems.
