Nomination of the LSM6DSOX, First Inertial Sensor with Machine Learning

Check out how to configure the decision tree in a few simple steps, with our Unico GUI, to use machine learning to detect a user glancing at a phone. We also have a new IDE, STM32CubeIDE, to make development operations a lot more practical.

Update June 28, 2019:

The AMA Association for Sensors and Measurement (AMA) nominated the LSM6DSOX as one of the four finalists of its AMA Innovation Award 2019 and recognized it as an outstanding project during the SENSOR+TEST conference that took place a few days ago in Nuremberg, Germany. The nomination promotes some of the most advanced devices in the industry and the jury appreciated the machine learning capabilities of the LSM6DSOX as it offloads the main microcontroller and thus saves tremendous energy. The decision tree enables the analysis, classification, and identification of patterns to make the sensor a lot smarter than competing components, thus transforming embedded systems.

Update February 11, 2019:

We are now announcing the commercial availability of the LSM6DSOX. Following the release a few months ago of the LSM6DSO, the first sensor ever with 16 finite state machines, the LSM6DSOX builds on the paper we described last year to offer 16 FSM and a decision tree, thus bringing machine learning to sensors. Developers can start prototyping with the STEVAL-MKI197V1, an adapter board that can go on the STEVAL-MKI109V2 or the STEVAL-MKI109V3 motherboards to facilitate developments. We also updated our Unico GUI to help them customize their applications. Finally, engineers can configure certain parameters of the decision tree with Weka, an open source collection of machine learning algorithms

Original article published on April 11, 2018:

During Embedded World 2018, Dr. Marco Castellano presented a lecture on Embedded Algorithms for Motion Detection and Processing, and it was our pleasure to sit down with him and learn how machine learning will disrupt sensors and IoT. Indeed, sensors continue to shape the IoT revolution since the vast majority of applications connect low-power devices to a cloud as a way to transmit sensors’ information. According to a study by IHS-Markit, the global sensors market will total $11.84 billion by 2022, and we expect pressure, temperature, image, magnetic, light, and motion sensors will be among the most popular. Hence, leading the sensors’ market means shaping the IoT revolution.

However, Dr. Castellano’s paper, written in cooperation with Marco Bianco’s team, suggests that the trends of today may be very different from those of tomorrow. Sensors tend to focus on performance, since the public is always asking for more megapixels, millimeter accuracy, and ultimate precision. However, new research papers challenge this vision by showing that it’s possible to have much more efficient sensor applications with a much lower power consumption, and still enjoy outstanding performance by moving part of the intelligence into the sensor and reducing application processor tasks.

Ultimately, sensors can offer better results and more significant energy savings when they become a lot smarter. It sounds almost too good to be true, but this technological breakthrough is at the heart of ST’s new inertial sensor, the LSM6DSOX prototype, that includes a Machine Learning Processor (MLP) and a Finite State Machine (FSM).

A Sensor with Machine Learning

Very simply, Machine Learning is a field of computer science that studies how huge datasets can teach a machine and how this machine will, in turn, apply what it learned to make decisions. Machine Learning is an incredibly vast topic, and it serves a lot of different applications. However, when a machine must draw a conclusion from various data points, it often uses a decision tree. A decision tree is a structure made of nodes, each node possessing one or more branches. As the system processes the information, it will travel through different branches until it reaches a leaf, which represents the conclusion the machine will return.

One of the most revolutionary aspects of the LSM6DS0X sensor is that it has a Machine Learning Processor baked into the sensor itself with up to eight configurable decision trees. What this means is that instead of using a host microcontroller (MCU) to run an algorithm and draw conclusions from the available data, which demands a lot of energy, the decision tree in the sensor can run an inductive algorithm at a fraction of the power consumption. As a result, the system can recognize specific activities, carry positions, or count steps, by simply observing the users’ movements and inferring from pre-defined patterns. Dr. Castellano, who joined ST in 2008 and specializes in algorithm integration as well as low-power arithmetic circuits, explained:

“We have custom hardware to run a decision tree, whereas an MCU runs general-purpose instructions for any kind of application. Our hardware is thus able to implement a decision tree with hundreds of nodes and run algorithms that would saturate a traditional Cortex M0 and simply wouldn’t compile.

A Sensor with Finite State Machines

The LSM6DSOX also relies on a Finite State Machine, a model that uses a certain number of pre-defined states, and a series of inputs to offer outputs. In its most basic form, it uses sequential logic to make decisions. For instance, we could imagine an FSM that changes from State A to State B if the input is 1, but remains in State A if it is 0. FSMs require very little computational throughput and very little memory, making them extremely efficient at very specific tasks. The tradeoff is that unlike a more polyvalent computing system, an FSM can’t perform certain calculations.

Dr. Castellano and co-authors (R. Bassoli, M. Bianco, A. Cagidiaco, C. Crippa, M. Ferraina, M. Leo, S.P. Rivolta) realized that Finite State Machines would be excellent at running deductive algorithms to ensure sensors can detect whether a product is facing up or down, in free fall, picked up by the user, or even monitoring certain fitness activities, among other things. The LSM6DSOX has thus up to 16 independent FSMs that are very simple in nature, but highly configurable, such as threshold or timer comparisons. Hence, if developers want to detect a wrist tilt to turn the screen of a smartwatch on or off, they just need to configure the FSM and determine when specific outputs will generate an interrupt that will turn the screen on or off depending on the user’s movements.

Furthermore, the MLP and FSM of the LSM6DSOX are not independent silos and can inform one another. For instance, it’s possible to run the result of an FSM through one of the components’ decision tree to rule out false positives that would otherwise throw a wrongful interrupt.

A Sensor from All of ST

We are leading efforts in Machine Learning in low-power applications, such as with Orlando, a SoC with a neural network, and since companies often know us through our MEMS, an inertial sensor like the LSM6DSOX just makes sense. Having MLP and FSM modules means algorithm processing requires 20 to 100 times less current than traditional solutions, opening the door to much smaller and smarter devices that don’t need more power or additional external hardware units to achieve similar results. Dr. Castellano’s team worked on realistic architectures, which explains why the LSM6DSO, a similar sensor with the FSM module, should be available this year.

However, beyond the product, it’s also important to underline that this new type of sensor was possible because all of ST got together. Indeed, the dataset that trained the Machine Learning module comes from ST employees themselves. We organized a worldwide campaign to track people’s movements and teach the system to recognize certain activities from the patterns collected. This was a crucial endeavor because the quality of the dataset determines the accuracy and performance of the algorithms, and ultimately the relevance of the results that shape the user’s experience. It is thus a truly special feeling to see all the efforts of our people come together in the LSM6DSOX prototype to create the sensors of tomorrow.