App-driven intelligence with IoT data

Solution Overview

From manufacturing and transportation to logistics and supply chain, the ability to perform real-time analysis on live IoT data during operation, and combine it with other data sources post-operation, is valuable across many industries. This demo shows how to leverage the MongoDB developer data platform to go from zero to lift-off in a rocket launch example.

A typical rocket launch spans an eight-hour period from the time the initial countdown begins until the rocket payload is in orbit. During this window, approximately one million metrics are generated per second by sensors capturing the rocket’s performance. While the metrics make up the bulk of the data in this scenario, there are two other sources of data: notes and weather data.

• Notes are created by both rocket engineers and an automated system. The rocket engineers create notes when they want to mark a time period or situation that they want to remember to revisit after the launch. An automated system is continuously watching the metrics as they stream in, creating notes whenever parameters reach thresholds that are out of bounds.

• Weather data is retrieved from a third party, and stored in an Amazon S3 bucket, and analyzed in combination with the launch data post-launch.

This solution shows how the various components of MongoDB's developer data platform can be used together to support application-driven analytics on IoT data by highlighting a rocket launch, with analysis happening during the launch and after. It uses an open source data set from a Blue Origin rocket launch along with fictional data.

Other Applicable industries and use cases:

Other industries that leverage IoT sensor data for both real-time and post-event analytics include:

• Manufacturing: Oftentimes manufacturing machinery is loaded with IoT sensors that operate for hours at a time, similar to a rocket launch. It’s important that analysis can be done while the machinery is operating and post-operation.

• Logistics: From the modes of transportation to the packages themselves, IoT sensors enable supply chain optimization both while in-transit and post-transit.

• eCommerce: In addition to warehouses and shipping logistics, retailers use real-time insights to drive personalization or improve business processes.

Reference Architectures

Without MongoDB:

Without MongoDB, this solution often requires a variety of bolt-on or disparate databases to support the variety of data types being generated, and data needs to be moved and transformed to reach the systems that are used to action this data (like a BI tool).

Data Model Approach

Review of the Atlas cluster deployed for the demonstration, and overview of the document model for two main Atlas collections used to store the launch data: launchData and notes. There are two main collections used in the demo:

• launchData
• notes

In general, the rocket metrics are produced by the rockets in 4-element tuples having the structure below:

• device, timestamp, metric, value

In the real-world scenariov that inspired this demonstration, an intermediate app server collected these tuples and aggregated them by device and timestamp. When all the metrics for a given device and timestamp were received, these metrics were written to the launchData collection as a single document. This document model groups the metrics into documents that match the way the data is read and analyzed by the application (by device and time). This follows the MongoDB schema design principle that data that is accessed together is stored together. This document model also was designed to work with MongoDB time-series collections. The time and metrics fields are top-level fields and the meta data (device name) is encapsulated in a field called “meta.”

Code Snippet

This data can be found in the file aerospace.archive.gz and restored to an Atlas cluster using mongorestore as below:

Building the Solution

This rocket launch solution was recorded over a three-part livestream. This GitHub provides all the code, data, and links to recordings you need to get started building out application-driven analytics that revolve around time-stamped IoT sensor data.

1. Development Environment Setup

Prior to watching any of the livestreams, you’ll want to go through the setup steps listed in GitHub readme. This will help you get started with an Atlas free tier account (if you don’t already have one) and download Compass.

2. Pre-launch: Rocket launch setup and basic analytical queries with MongoDB’s aggregation framework

The Livestream 1 recording will walk you through the data captured in a rocket launch and how to write basic aggregation queries with MongoDB’s query API. Then you’ll perform basic analytics on the IoT sensor data with Atlas Charts. GitHub folder.

3. During-launch: Embedding visualizations and building search capabilities in a React app for real-time analytics

Livestream 2 will show you how to embed charts into a React app for real-time analytics, and how to create a search capability for finding insights for specific notes, automated or manually generated, of interest. GitHub folder.

4. Post-launch: Finding insights across multiple application and third-party data sources

Lastly, Livestream 3 will show you how to find post-operation insights by combining rocket launch data with weather data from the same time frame and analyzing it in Tableau. GitHub folder.

Technologies and Products Used

MongoDB developer data platform

• Atlas Database with dedicated analytics nodes
• Time Series with window functions
• Aggregation Pipelines
• Atlas SQL Interface
• Atlas Search
• Atlas Charts with embedded charts
• Atlas Data Federation
• Atlas Trigger

Partner technologies

• Amazon S3
• Tableau

Key Considerations

• Modeling data across multiple data types
• Analyzing time series data with window functions
• Leveraging search indexes
• Doing in-place aggregation pipelines for minimizing ETL processes
• Finding insights from hot data (live application) and cold data (third party in Amazon S3)
• Integrating serverless functions to react to real-time data

Author

• Jay Runkel, MongoDB