Atlas Stream Processing is Now Generally Available!

We're thrilled to announce that Atlas Stream Processing—the MongoDB-native way to process streaming data—is now generally available, empowering developers to quickly build responsive, event-driven applications!

Our team spent the last two years defining a vision and building a product that leans into MongoDB’s strengths to overcome the hard challenges in stream processing. After a decade of building stream processing products outside of MongoDB, we are using everything that makes MongoDB unique and differentiated—the Query API and powerful aggregation framework, as well as the document model and its schema flexibility—to create an awesome developer experience.

It’s a new approach to stream processing, and based on the feedback of so many of you in our community, it’s the best way for most developers using MongoDB to do it.

Let’s get into what’s new.

This post is also available in: Deutsch, Français, Español, Português, Italiano, 한국어, 简体中文.

What's new in general availability?

Graphic calling out the new features in general availability. Under the Dev Experience Improvements category, there is

  1. Production Readiness
    Ready to support your production workloads, ensuring reliable and scalable stream processing for your mission-critical applications.

  2. Time Series Collection Support
    Emit processor results into Time Series Collections. Pre-process data continuously while saving it for historical access later in a collection type available in MongoDB Atlas built to efficiently store and query time series data.

  3. Development and Production Tiers
    Besides the SP30 cluster tier available during the public preview, we’re introducing an SP10 tier to provide flexibility and a cost-effective option for exploratory use cases and low-traffic stream processing workloads.

  4. Improved Kafka Support
    Added support for Kafka headers allows applications to provide additional metadata alongside event data. They are helpful for various stream processing use cases (e.g., routing messages, conditional processing, and more).

  5. Least Privilege Access
    Atlas Database Users can grant access to Stream Processing Instances and enable access to only those who need it. Read our tutorial for more information.

  6. Stream Processor Alerting
    Gain insight and visibility into the health of your stream processors by creating alerts for when a failure occurs. Supported methods for alerting include email, SMS, monitoring platforms like Datadog, and more.

Why Atlas Stream Processing?

Atlas Stream Processing brings the power and flexibility of MongoDB's document model and Query API to the challenging stream processing space. With Atlas Stream Processing, developers can:

  • Effortlessly handle complex and rapidly changing data structures

  • Use the familiar MongoDB Query API for processing streaming data

  • Seamlessly integrate with MongoDB Atlas

  • Benefit from a fully managed service that eliminates operational overhead

Customer highlights

Read what developers are saying about Atlas Stream Processing:

At Acoustic, our key focus is to empower brands with behavioral insights that enable them to create engaging, personalized customer experiences. To do so, our Acoustic Connect platform must be able to efficiently process and manage millions of marketing, behavioral, and customer signals as they occur. With Atlas Stream Processing, our engineers can leverage the skills they already have from working with data in Atlas to process new data continuously, ensuring our customers have access to real-time customer insights.

John Riewerts, EVP, Engineering at Acoustic

Atlas Stream Processing enables us to process, validate, and transform data before sending it to our messaging architecture in AWS powering event-driven updates throughout our platform. The reliability and performance of Atlas Stream Processing has increased our productivity, improved developer experience, and reduced infrastructure cost.

Cody Perry, Software Engineer, Meltwater

What's ahead for Atlas Stream Processing?

Graphic calling out some of the things that are ahead for Atlas Stream Processing, including more regions, advanced networking, additional sources & sinks, and richer metrics & observability

We’re rapidly introducing new features and functionality to ensure MongoDB delivers a world-class stream processing experience for all development teams. Over the next few months, you can expect to see:

  1. Advanced Networking Support
    Support for VPC Peering to Kafka Clusters for teams requiring additional networking capabilities

  2. Expanded Cloud Region Support
    Support for all cloud regions available in Atlas Data Federation

  3. Expanded Cloud Provider Support
    Support for Microsoft Azure

  4. Expanded Data Source and Sink Support
    We have plans to expand beyond Kafka and Atlas databases in the coming months. Let us know which sources and sinks you need, and we will factor that into our planning

  5. Richer Metrics & Observability
    Support for expanded visibility into your stream processors to help simplify monitoring and troubleshooting

  6. Expanded Deployment Flexibility
    Support for deploying stream processors with Terraform. This integration will help to enable a seamless CI/CD pipeline, enhancing operational efficiency with infrastructure as code. Look out for a dedicated blog in the near future on how to get started with Atlas Stream Processing and Terraform.

So whether you're looking to process high-velocity sensor data, continuously analyze customer data to deliver personalized experiences, or perform predictive maintenance to increase yields and reduce costs, Atlas Stream Processing has you covered. Join the hundreds of development teams already building with Atlas Stream Processing. Stay tuned to hear more from us soon, and good luck building!

Login today or check out our introductory tutorial to get started.

← Previous

Elevating Database Performance: Introducing Query Insights in MongoDB Atlas

Today, at .local NYC, MongoDB Atlas introduced the new Query Insights tab, enhancing how users monitor, manage, and optimize their database performance directly within the Atlas UI. This new feature offers developers deeper insights into their database’s performance, with a more powerful query analysis tool and detailed namespace-level metrics for faster issue resolution and enhanced performance. Applications and workloads change over time, making it increasingly difficult to track inefficient queries that strain a database's resources. Metrics can spike for various reasons, and developers need the right tooling to determine the source of the problem so they can quickly identify and resolve the issue. MongoDB Atlas's Query Insights directly tackles these challenges by enhancing MongoDB's observability capabilities with two crucial features: Namespace Insights and an upgraded Query Profiler. This post is also available in: Deutsch , Français , Español , Português , Italiano , 한국어 , 简体中文 . Query Insights delivers performance optimization through actionable intelligence The introduction of MongoDB Atlas Query Insights demonstrates MongoDB’s commitment to advanced database management. This feature enhances our platform’s observability capabilities with detailed and actionable insights. This feature integrates Namespace Insights and an upgraded Query Profiler within a new dynamic interface, helping boost database performance by streamlining diagnostics and reducing troubleshooting times. The newly added Namespace Insights provides users with collection-level latency statistics and a comprehensive view of how the hottest collections on a cluster perform over time. This enables developers to answer "Who or what is causing the problem?” which is instrumental in identifying performance trends and prioritizing query optimizations. The enhanced cluster-centric Query Profiler introduces a more comprehensive view of slow and inefficient queries over a broader period. Having an overall view of data across the entire cluster facilitates more straightforward navigation between nodes and a longer lookback period to identify trends. This ultimately reduces troubleshooting time, thereby enhancing developer productivity and improving overall database performance. Key benefits of Query Insights Query Insights brings MongoDB Atlas users several new benefits, including: Granular telemetry: Faster identification and resolution of database issues with namespace-level latency statistics Improved observability: It is easier to spot performance trends, identify root causes, and debug applications Enhanced productivity: Reduced troubleshooting time thanks to a more comprehensive view of slow operations Try it out! The Query Insights page provides more granular insights into database performance by providing collection and operation-level details. The Namespace Insights page provides metrics for the top 20 collections by total latency. Hover over the charts to see how collections perform relative to each other over time. This information makes it easier to answer the question: “who/what is causing the problem?” Use the Query Profiler to view specific slow operations. Click on a point in the scatter plot to bring up additional metadata about each slow operation. Click on View More Details to see more metrics and metadata about each slow operation, including the app name, the operation, the plan summary, execution stats, etc. Empowering users for peak performance The launch of Query Insights in MongoDB Atlas underscores MongoDB’s commitment to enhancing our platform's observability capabilities. By providing users with the necessary tools and insights for optimal database performance, MongoDB enables developers to spend less time debugging and more time creating—lowering the total cost of ownership and maximizing efficiency, adding significant value to our users' operations. Sign up for MongoDB Atlas , our cloud database service, to see Query Insights in action, and for more information, see Monitor Query Performance .

May 2, 2024
Next →

The Converged AI and Application Datastore for Insurance

In the inherently information-driven insurance industry, companies ingest, analyze, and process massive amounts of data, requiring extensive decision-making. To manage this, they rely on a myriad of technologies and IT support staff to keep operations running smoothly but often lack effectiveness due to their outdated nature. Artificial intelligence (AI) holds great promise for insurers by streamlining processes, enhancing decision-making, and improving customer experiences with significantly less time, resources, and staff compared with traditional IT systems. The convergence of AI and innovative application datastores is transforming how insurers work with data. In this post, we’ll look at how these elements are reshaping the insurance industry and offering greater potential for AI-powered applications, with MongoDB at the heart of the converged AI and application datastore. Scenario planning and flexible data layers One of the primary concerns for IT leaders and decision-makers in the insurance industry is making smart technology investments. The goal is to consolidate existing technology portfolios, which often include a variety of systems like SQL Server, Oracle, and IBM IMS. Consolidation helps reduce inventory and prepare for the future. But what does future-proofing really look like? Scenario planning is an effective strategy for future-proofing. This involves imagining different plausible futures and investing in the common elements that remain beneficial across all scenarios. For insurance companies, a crucial common thread is the data layer. By making data easier to work with, companies can ensure that their technology investments remain valuable regardless of how future scenarios unfold. MongoDB’s flexible developer data platform offers a distinct architectural advantage by making data easier to work with, regardless of the cloud vendor or AI application in use. This flexibility is vital for preparing for disruptive future scenarios, whether they involve regulatory changes, market shifts, or technological advancements. Watch now: The Converged AI and Application Datastore: How API's, AI & Data are Reshaping Insurance The role of AI and data in insurance Generative AI is revolutionizing the insurance sector, offering new ways to manage and utilize data. According to Celent's 2023 Technology Insight and Strategy Survey, 33% of companies across different industries have AI projects in planning, 29% in development, and 19% in production (shown in Figure 1 below). This indicates a significant shift towards AI-driven solutions by insurers actively experimenting with gen AI. Figure 1: Celent Technology Insight and Strategy Survey 2023 However, there's tension between maintaining existing enterprise systems and innovating with AI. Insurance companies must balance keeping the lights on with investing in AI to meet the expectations of boards and stakeholders. The solution lies in integrating AI in a way that enhances operational efficiency without overwhelming existing systems. However, data challenges need to be addressed to achieve this, specifically around access to data. According to a Workday Global Survey , only 4% of respondents said their data is fully accessible, and 59% say their enterprise data is somewhat or completely siloed. Without a solid data foundation, insurers will struggle to achieve the benefits they are looking for from AI. Data architectures and unstructured data When adopting advanced technologies like AI and ML, which require data as the foundation, organizations often grapple with the challenge of integrating these innovations into legacy systems due to their inflexibility and resistance to modification. A robust data architecture is essential for future-proofing and consolidating technology investments. Insurance companies often deal with a vast amount of unstructured data, such as claim images and videos, which can be challenging to manage. By leveraging AI, specifically through vector search and large language models, companies can efficiently process and analyze this data. MongoDB is ideal for managing unstructured data due to its flexible, JSON-like document model, which accommodates a wide variety of data types and structures without requiring a predefined schema. Additionally, MongoDB’s flexibility enables insurers to integrate seamlessly with various technologies, making it a versatile and powerful solution for unstructured data management. For example, consider an insurance adjuster assessing damage from claim photos. Traditionally, this would require manually reviewing each image. With AI, the photos can be converted into vector embeddings and matched against a database of similar claims, drastically speeding up the process. This not only improves efficiency but also enhances the accuracy of assessments. The converged AI and application datastore with MongoDB Building a single view of data across various systems is a game-changer for the insurance industry. Data warehouses and data lakes have long provided single views of customer and claim data, but they often rely on historical data, which may be outdated. The next step is integrating real-time data with these views to make them more dynamic and actionable. A versatile database platform plays a crucial role in this integration. By consolidating data into a single, easily accessible view, insurance companies can ensure that various personas, from underwriters to data scientists, can interact with the data effectively. This integration allows for more responsive and informed decision-making, which is crucial for staying competitive in a rapidly evolving market. This can be achieved with a converged AI and application datastore, as shown in Figure 2 below. This is where operational data, analytics insights, and unstructured data become operationally ready for the applications that leverage AI. Figure 2: Converged AI and application datastore reference architecture The convergence of AI, data, and application datastores is reshaping the insurance industry. By making smart technology investments, leveraging AI to manage unstructured data, and building robust data architectures, insurance companies can future-proof their operations and embrace innovation. A versatile and flexible data platform provides the foundation for these advancements, enabling companies to make their data more accessible, actionable, and valuable. The MongoDB Atlas developer data platform puts powerful AI and analytics capabilities directly in the hands of developers and offers the capabilities to enrich applications by consolidating, ingesting, and acting on any data type instantly. Because MongoDB serves as the operational data store (ODS)—with its flexible document model—insurers can efficiently handle large volumes of data in real-time. By integrating MongoDB with AI/ML platforms, insurers can develop models trained on the most accurate and up-to-date data, thereby addressing the critical need for adaptability and agility in the face of evolving technologies. With built-in security controls across all data, whether managed in a customer environment or through MongoDB Atlas, a fully managed cloud service, MongoDB ensures robust security with features such as authentication (single sign-on and multi-factor authentication), role-based access controls, and comprehensive data encryption. These security measures act as a safeguard for sensitive data, mitigating the risk of unauthorized access from external parties and providing organizations with the confidence to embrace AI and ML technologies. If you would like to learn more about the convergence of AI and application datastores, visit the following resources: Video: The Converged AI and Application Datastore: How API's, AI & Data are Reshaping Insurance Paper: Innovation in Insurance with Artificial Intelligence The MongoDB Solutions Library is curated with tailored solutions to help developers kick-start their projects

July 18, 2024