Venkatesh Shanbhag

Partner Solution Architect

Partner

Yemi Falokun

ISV CoE Solution Architect at Google

In the digital age, leveraging natural language for database queries represents a leap toward more intuitive data management. [Vertex AI Extensions](https://cloud.google.com/vertex-ai/generative-ai/docs/extensions/overview), currently in **private preview**, help in interacting with MongoDB using natural language. This tutorial introduces an approach that combines Google [Gemini's](https://ai.google.dev/models/gemini) advanced natural language processing with MongoDB, facilitated by Vertex AI Extensions. These extensions address key limitations of large language models (LLMs) by enabling real-time data querying and modification, which LLMs cannot do due to their static knowledge base post-training. By integrating MongoDB Atlas with Vertex AI Extensions, we offer a solution that enhances the accessibility and usability of the database. 

MongoDB's dynamic schema, scalability, and comprehensive querying capabilities render it exemplary for Generative AI applications. It is adept at handling the versatile and unpredictable nature of data that these applications generate and use. From personalized content generation, where user data shapes content in real time, to sophisticated, AI-driven recommendation systems leveraging up-to-the-minute data for tailored suggestions, MongoDB stands out. Furthermore, it excels in complex data analysis, allowing AI tools to interact with vast and varied datasets to extract meaningful insights, showcasing its pivotal role in enhancing the efficiency and effectiveness of Generative AI applications.

## Natural language to MongoDB queries

Natural language querying represents a paradigm shift in data interaction, allowing users to retrieve information without the need for custom query languages. By integrating MongoDB with a system capable of understanding and processing natural language, we streamline database operations, making them more accessible to non-technical users.

### Solution blueprint

The solution involves a synergy of several components, including MongoDB, the Google Vertex AI SDK, Google Secrets Manager, and OpenAPI 3 specifications. Together, these elements create a robust framework that translates natural language queries into MongoDB Data API calls. In this solution, we have explored basic CRUD operations with Vertex AI Extensions. We are closely working with Google to enable vector search aggregations in the near future.

### Components involved

1. **MongoDB**: A versatile, document-oriented database that stores data in JSON-like formats, making it highly adaptable to various data types and structures
2. **Google Vertex AI SDK**: Facilitates the creation and management of AI and machine learning models, including the custom extension for Google Vertex AI 
3. **Vertex AI Extensions:** Enhance LLMs by allowing them to interact with external systems in real-time, extending their capabilities beyond static knowledge 
4. **Google Secrets Manager**: Securely stores sensitive information, such as MongoDB API keys, ensuring the solution's security and integrity
5. **OpenAPI 3 Specification for MongoDB Data API**: Defines a standard, language-agnostic interface to MongoDB that allows for both easy integration and clear documentation of the API's capabilities

### Description of the solution

The solution operates by converting natural language queries into parameters that the MongoDB Data API can understand. This conversion is facilitated by a custom extension developed using the Google Vertex AI extension SDK, which is then integrated with Gemini 1.0 Pro. The extension leverages OpenAPI 3 specifications to interact with MongoDB, retrieving data based on the user's natural language input. Google Secrets Manager plays a critical role in securely managing API keys required for MongoDB access, ensuring the solution's security.

![architecture diagram of the Vertex AI extensions][1]

## Prerequisites

Before you start, make sure you have the following:

1. Secure your access to the Google Cloud [project](https://console.cloud.google.com/) or to create a new project.
2. If you are new to MongoDB Atlas, you can sign up to MongoDB either through the [Google Cloud Marketplace](https://console.cloud.google.com/marketplace/product/mongodb/mdb-atlas-self-service?utm_source=mongodb_devrel&utm_medium=static_site&utm_campaign=devrel&utm_id=mongodb) or with the Atlas [registration page](https://account.mongodb.com/account/register).
3. Vertex AI Extensions are not publicly available. Please sign up for the Extensions [Trusted Tester Program](https://cloud.google.com/vertex-ai/generative-ai/docs/extensions/overview).
4. Basic knowledge of [OpenAPI specifications](https://swagger.io/specification/) and how to create them for APIs will be helpful.
5. You’ll need a Google [Cloud Storage bucket](https://cloud.google.com/storage/docs/creating-buckets) for storing the OpenAPI specifications.

Before we begin, also make sure you:

**Enable MongoDB Data API**: To enable the Data API from the Atlas console landing page, open the [Data API](https://www.mongodb.com/docs/atlas/app-services/data-api/generated-endpoints/) section from the side pane, enable the Data API, and copy the URL Endpoint as shown below.

![Navigating to Data API on MongoDB][2]

To create an API key, click on **Users** and then click on **Create API Key**.

![Creating API key for Data API][3]

**Create a secret for your API key**: To secure the MongoDB Data API key, store it in Google Cloud [Secret Manager](https://cloud.google.com/secret-manager/docs/creating-and-accessing-secrets#:~:text=Go to the Secret Manager page in the Google Cloud console.&text=On the Secret Manager page,example%2C my-secret )). To create a new secret on the Google Cloud Console, [navigate](https://console.cloud.google.com/security/secret-manager) to Secrets Manager, and click on **CREATE SECRET**. Paste the secret created from MongoDB to the secret value field and click on **Create**.

![Creating secret on Google Cloud console][4]

**Develop API specification:** We will define the interactions between Google Vertex AI and your MongoDB Data API using OpenAPI 3.0 specs. You can download the specs from the [GitHub repository](https://github.com/mongodb-partners/MongoDB-VertexAI-extensions). This specification outlines how natural language queries will be translated into MongoDB operations.

## Create Vertex AI extensions

This tutorial uses the MongoDB default dataset from the **sample_mflix** database, **movies** collection. We will run all the below code on the Enterprise Colab notebook.

1. Vertex AI Extensions is a platform for creating and managing extensions that connect large language models to external systems via APIs. These external systems can provide LLMs with real-time data and perform data processing actions on their behalf.

```python
from google.colab import auth
auth.authenticate_user("GCP project id")
!gcloud config set project {"GCP project id"}
```

2. Install the required Python dependencies.

```python
!gsutil cp gs://vertex_sdk_private_releases/llm_extension/google_cloud_aiplatform-1.44.dev20240315+llm.extension-py2.py3-none-any.whl .
!pip install --force-reinstall --quiet google_cloud_aiplatform-1.44.dev20240315+llm.extension-py2.py3-none-any.whl[extension]
!pip install --upgrade --quiet google-cloud-resource-manager
!pip install --force-reinstall --quiet langchain==0.0.298!pip install pytube
!pip install --upgrade google-auth
!pip install bigframes==0.26.0
```

3. Once the dependencies are installed, restart the kernel.

```python
import IPython
app = IPython.Application.instance()
app.kernel.do_shutdown(True)  # Re-run the Env variable cell again after Kernel restart
```

4. Initialize the environment variables.

```python
import os
## This is just a sample values please replace accordingly to your project
# Setting up the GCP project
os.environ['PROJECT_ID'] = 'gcp project id'  # GCP Project ID
os.environ['REGION'] =  "us-central1" # Project Region
## GCS Bucket location
os.environ['STAGING_BUCKET'] =  "gs://vertexai_extensions"
## Extension Config
os.environ['EXTENSION_DISPLAY_HOME'] =  "MongoDb Vertex API Interpreter"
os.environ['EXTENSION_DESCRIPTION'] =  "This extension makes api call to mongodb to do all crud operations"


## OPEN API SPec config
os.environ['MANIFEST_NAME'] =  "mdb_crud_interpreter"
os.environ['MANIFEST_DESCRIPTION'] =  "This extension makes api call to mongodb to do all crud operations"
os.environ['OPENAPI_GCS_URI'] =  "gs://vertexai_extensions/mongodbopenapispec.yaml"


## API KEY secret location
os.environ['API_SECRET_LOCATION'] = "projects/787220387490/secrets/mdbapikey/versions/1"


##LLM config
os.environ['LLM_MODEL'] = "gemini-1.0-pro"
```

5. Download the Open API specification from [GitHub](https://github.com/mongodb-partners/MongoDB-VertexAI-extensions) and upload the [YAML file](https://github.com/mongodb-partners/MongoDB-VertexAI-extensions/blob/main/open-api-spec/mdb-data-api.yaml) to the Google Cloud Storage bucket. 

```python
from google.cloud import aiplatformfrom google.cloud.aiplatform.private_preview import llm_extension

PROJECT_ID = os.environ['PROJECT_ID']
REGION = os.environ['REGION']
STAGING_BUCKET = os.environ['STAGING_BUCKET']

aiplatform.init(
    project=PROJECT_ID,
    location=REGION,
    staging_bucket=STAGING_BUCKET,
)
```

6. To create the Vertex AI extension, run the below script. The manifest here is a structured JSON object containing several key components:

```python
mdb_crud = llm_extension.Extension.create(
display_name = os.environ['EXTENSION_DISPLAY_HOME'],
description = os.environ['EXTENSION_DESCRIPTION'],  # Optional         manifest = {             "name": os.environ['MANIFEST_NAME'],
             "description": os.environ['MANIFEST_DESCRIPTION'],
             "api_spec": {
                 "open_api_gcs_uri": os.environ['OPENAPI_GCS_URI'],
             },             "auth_config": {
                 "apiKeyConfig":{
             "name":"api-key",
            "apiKeySecret":os.environ['API_SECRET_LOCATION'],
            "httpElementLocation": "HTTP_IN_HEADER"
            },
            "authType":"API_KEY_AUTH"
            },
        },
)
```

7. Validate the Created Extension, and print the Operation Schema and Parameters.

```python
print("Name:", mdb_crud.gca_resource.name)print("Display Name:", mdb_crud.gca_resource.display_name)print("Description:", mdb_crud.gca_resource.description)
import pprint
pprint.pprint(mdb_crud.operation_schemas())
```

## Extension in action

Once the extension is created, navigate to [Vertex AI UI](https://console.cloud.google.com/vertex-a) and then Vertex UI Extension on the left pane. 

![Navigating to Vertex AI extensions][5]

All the created extensions will be available for selection. Select **MongoDB Vertex API Interpreter**. For this tutorial, we will consider two user journeys.

### User Journey 1

**Retrieve data through Vertex AI extensions:** Imagine needing to quickly find out when a classic film was released without navigating through the database manually. By asking Vertex AI, "Find the release year of the movie 'A Corner in Wheat' from VertexAI-POC cluster, sample_mflix, movies," you get the specific release year, as the system performs a findOne() operation to retrieve this detail.

![Testing Vertex AI extensions for find query][6]

In addition to using the UI, we can also use the Notebook to query MongoDB using natural language processing. We are using Gemini to convert natural language to the actual operation schema for calling the Vertex AI extension. Update the environment variable and run the below script.

```python
## Please replace accordingly to your project
## Operation Ids
os.environ['FIND_ONE_OP_ID'] = "findone_mdb"




## NL Queries
os.environ['FIND_ONE_NL_QUERY'] = "Find the release year of the movie 'A Corner in Wheat' from VertexAI-POC cluster, sample_mflix, movies"


## Mongodb Config
os.environ['DATA_SOURCE'] = "VertexAI-POC"
os.environ['DB_NAME'] = "sample_mflix"
os.environ['COLLECTION_NAME'] = "movies"


### Test data setup
os.environ['TITLE_FILTER_CLAUSE'] = "A Corner in Wheat"

from vertexai.preview.generative_models import GenerativeModel, Tool

fc_chat = GenerativeModel(os.environ['LLM_MODEL']).start_chat()
findOneResponse = fc_chat.send_message(os.environ['FIND_ONE_NL_QUERY'],
    tools=[Tool.from_dict({        "function_declarations": mdb_crud.operation_schemas()
   })],
)

findOneResponse
```

From the converted schema, query the extension.

```python
response = mdb_crud.execute(
    operation_id = findOneResponse.candidates[0].content.parts[0].function_call.name,
    operation_params = findOneResponse.candidates[0].content.parts[0].function_call.args
)

response
```

### User Journey 2

**Retrieve data through Vertex AI extensions:** A film historian wants a list of all movies released in a specific year — say 1924 — to study the cinematic trends of that era. They could ask, "Give me movies released in the year 1924 from VertexAI-POC cluster, sample_mflix, movies," and the system would use the find() method to list all movies from 1924, providing a comprehensive snapshot of that year's cinematic output.

![testing Vertex AI extensions for find query 2][7]

In addition to using the UI, we can also use the Notebook to query MongoDB using natural language processing. 

Here’s the Gemini integration for natural language to extension schema conversion:

```python
## This is just a sample values please replace accordingly to your project
## Operation Ids
os.environ['FIND_MANY_OP_ID'] = "findmany_mdb"


## NL Queries
os.environ['FIND_MANY_NL_QUERY'] = "give me movies released in year 1924 from VertexAI-POC cluster, sample_mflix, movies"




## Mongodb Config
os.environ['DATA_SOURCE'] = "VertexAI-POC"
os.environ['DB_NAME'] = "sample_mflix"
os.environ['COLLECTION_NAME'] = "movies"
os.environ['YEAR'] = 1924
fc_chat = GenerativeModel(os.environ['LLM_MODEL']).start_chat()
findmanyResponse = fc_chat.send_message(os.environ['FIND_MANY_NL_QUERY'],
    tools=[Tool.from_dict({        "function_declarations": mdb_crud.operation_schemas()
   })],
)

findmanyResponse
```

Execute the extension to get the response.

```python
response = mdb_crud.execute(
    operation_id = findmanyResponse.candidates[0].content.parts[0].function_call.name,
    operation_params = findmanyResponse.candidates[0].content.parts[0].function_call.args
)

response
```

## Step by step extension execution

![User queries with natural language][8]

![LLM converts natural language accordingly to extension schema and executes extension][9]

![Extension fetches the document from MongoDB and returns the response][10]

![LLM converts the extension response to natural language and responds back to user][11]

## Conclusion

In conclusion, integrating MongoDB with natural language querying capabilities revolutionizes data interaction, enhancing accessibility and intuitiveness for database queries. Leveraging the Google Gemini Foundation Model alongside a custom Vertex AI extension not only enriches the data retrieval experience but also upholds data security and integrity. We are closely working with the Google Cloud team to add even more query patterns to Vertex AI extensions. Watch out for more in this space. 

### Further reading

1. [Get started](https://www.mongodb.com/docs/atlas/getting-started/) with MongoDB Atlas on Google Cloud.
2. Connect models to APIs by using [Vertex AI extensions](https://cloud.google.com/vertex-ai/generative-ai/docs/extensions/overview).


  [1]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt5f6e6aa6cea13ba1/661471b70c47840e25a3437a/1.png
  [2]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltae31621998903a57/661471cd4180c1c4ede408cb/2.png
  [3]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt58474a722f262f1a/661471e40d99455ada032667/3.png
  [4]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt9cd6a0e4c6b2ed4c/661471f5da0c3a5c7ff77441/4.png
  [5]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt3ac5e7c88ed9d678/661472114180c1f08ee408d1/5.png
  [6]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt39e9b0f8b7040dab/661472241a0e49338babc9e1/6.png
  [7]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/bltfc26acb17bfca16d/6614723b2b98e9f356100e6b/7.png
  [8]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt7bdaa1e8a1cf5a51/661472517cacdc0fbad4a075/8.png
  [9]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt272144a86fea7776/661472632b98e9562f100e6f/9.png
  [10]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt62198b1ba0785a55/66147270be36f54af2d96927/10.png
  [11]: https://images.contentstack.io/v3/assets/blt39790b633ee0d5a7/blt4a7e5371abe658e1/66147281be36f5ed61d9692b/11.png

By integrating MongoDB Atlas with Vertex AI Extensions, we offer a solution that enhances the accessibility and usability of the database.

Yemi Falokun

Content by Yemi Falokun

Harnessing Natural Language for MongoDB Queries With Google Gemini