What if a database for embedded systems, such as eXtremeDB, were the only thing standing between critical sensor data and a massive catastrophe? The scenario may seem far-fetched. And yet, how many engineers value the thousands, if not millions, of data points collected by their IoT systems? Many engineers carefully review the type of memory they use. Yet, they often gloss over how to structure it, which led McObject, a member of the ST Partner Program, to create the eXtremeDB embedded database management system (DBMS), and eXtremeDB/rt, the first hard real-time DBMS for microcontrollers. And as of a few weeks ago, McObject released eXtremeDB/rt 2.0 so developers can create a persistent database.
What’s new?
eXtremeDB/rt 2.0 supports persistent databases thanks to two new middleware: xFile and xFlash. The former works with the filesystem or the block I/O interface exposed by the flash translation layer (FTL) to read and write to a transactional persistent database. And, as the name suggests, xFlash utilizes unmanaged NAND memory to ensure deterministic performance, requiring no support from the bare-metal application or a real-time operating system. xFlash will even implement wear leveling and garbage collection while minimizing write amplification to extend the lifespan of the flash. This is particularly important when deterministic runtimes are required, as it offers stronger guarantees of real-time execution to developers.
The new version of eXtremeDB/rt also includes extensive improvements, such as an enhanced deadline management system that provides greater granularity for developers enforcing stricter deadlines and dynamically adjusting to runtime conditions. McObject is also adding a lightweight RESTful server for easier remote access and bringing support for Zephyr. Put simply, the company is looking to make eXtremeDB more accessible, whether through an API or by joining popular projects that facilitate the use of a real-time operating system. It also helps McObject reach new commercial projects that are increasingly relying on Zephyr.
The shortcomings of current systems
How expensive can a text file get?
Returning to the premise of this blog post — that losing sensor data could result in a catastrophe — let’s first consider the cost of losing that data. We already have real-world examples of software bugs corrupting sensor logs, leading to massive recalls and costing billions of dollars. Multiple car companies have faced this issue, but others have as well. Moreover, as the world increasingly relies on more sensor data for mission-critical applications, it’s easy to see that beyond the monetary costs, corruption of sensor logs could lead to physical harm. Which leads to the question very few ask: Is a text file the best place to store sensor data?
How tedious can a text file get?
The vast majority of IoT applications store sensor data in a CSV, TSV, or JSON file, but there are issues. If the IoT system experiences a critical crash during a write operation, it is impossible to ensure the file’s consistency. For instance, failing to write the footer of a JSON file renders the entire log unusable. Similarly, if two processes attempt to write to the same log simultaneously, the risk of corruption increases significantly. The lack of replication can also burden systems. For instance, syncing with a gateway that has been offline for a while can require a large bandwidth, which most IoT systems simply don’t have.
eXtremeDB as a solution to protect data, applications, customers, and companies
An age-old idea
Interestingly, those issues are far from new and explain why computer scientists invented databases in the first place. Some may be familiar with the acronym ACID, which stands for Atomicity, Consistency, Isolation, and Durability. It’s a concise way to explain the benefits of using a database. In a nutshell, databases circumvent the pitfalls of using a simple text file by defining a unit of work and implementing mechanisms to make the whole process vastly more efficient and resilient against errors and crashes. McObject explained that some IoT devices with very basic needs may still prefer to use a plain text file. However, many who underestimate the pitfalls of a JSON or CSV document could benefit from a database.
A new solution for embedded systems
McObject specifically built eXtremeDB for embedded systems, meaning that developers shouldn’t view it as a traditional relational database, like MySQL. Consequently, while eXtremeDB SQL can co-exist alongside eXtremeDB, it isn’t the primary setup. Most installations will use eXtremeDB on its own, which can utilize a combination of in-memory and on-disk databases, each optimized to minimize I/O operations and CPU bottlenecks by employing only a few instructions of what is called a navigational API through which an application directly operates the database in lieu of SQL.
eXtremeDB stands out as the only actual real-time database in the industry, meaning it offers deterministic transactions compatible with a real-time operating system (RTOS). Competing solutions may focus on so-called real-time analytics, but they don’t provide deterministic performance. Consequently, developers can utilize eXtremeDB for mission-critical applications, such as advanced driver assistance systems (ADAS).
To make eXtremeDB more accessible, McObject provides APIs in a wide range of programming languages, including C, C++, C#, Rust, Lua, and Python. Additionally, the company worked with ST to integrate its solution into the STM32Cube ecosystem. Concretely, it means that not only does eXtremeDB support both STM32 microcontrollers and microprocessors, but that McObject baked in heterogeneous processing to take advantage of all the cores available. The software company also leverages STM32 cryptocores to accelerate AES encryption, and even supports standard security libraries, like WolfSSL, the eponymous member of the ST Partner Program.
