Using LlamaIndex for Web Content Indexing and Querying

In the evolving landscape of information technology, data accessibility and management are paramount.

Enter LlamaIndex, a simple, flexible data framework for connecting custom data sources to large language models (LLMs). This blog post introduces you to the capabilities of LlamaIndex and illustrates its use through a sample project that leverages its ability to extract and query data from a web page – in this case, Abraham Lincoln’s Wikipedia page.

A fully runnable, companion code repository for this blog post is available here.

What is LlamaIndex used for?

LlamaIndex is particularly useful for developers looking to integrate web scraping, data indexing, and natural language processing (NLP) capabilities into their applications. LlamaIndex’s integration with machine learning models and its ability to work with various data loaders makes it a versatile tool in the field of data processing and analysis, as well as RAG (Retrieval-Augmented Generation) based applications.

Key Features

• VectorStoreIndex: Allows for efficient indexing of text documents into a vector space model, facilitating quick and accurate retrieval of information based on queries.
• ServiceContext: Integrates with language models like Mistral AI, enhancing the querying process with advanced NLP capabilities.
• Extensibility: LlamaIndex supports various data loaders, adapting to different sources and formats of web content.

Setting Up Your Project with LlamaIndex

Prerequisites

To get started, ensure you have Python 3.x installed, along with the llama_index and dotenv Python packages.

Configuration

1. Install the necessary libraries using pip:

   pip install llama_index python-dotenv

2. Create a .env file at your project’s root directory and include your Mistral AI API key:

   MISTRAL_API_KEY=YOUR_MISTRAL_API_KEY

   This key is essential for accessing the language model services used by LlamaIndex. For more information about how to use Mistral AI’s API hosting the open source Mistral models, you can checkout my blog post on building a chatbot with Mistral 8x7B.

Implementation

With LlamaIndex, you can easily load, index, and query web content. Here’s a breakdown of how to do this using the Wikipedia page of Abraham Lincoln as an example:

1. Data Loading: Use the community offered custom data loader BeautifulSoupWebReader to load the desired web page content.
2. Index Creation: Employ VectorStoreIndex to transform the loaded documents into a searchable index.
3. Querying: Utilize the query engine provided by VectorStoreIndex in conjunction with ServiceContext, integrated with Mistral AI, to execute queries against the indexed data.

Practical Application: Extracting Information about Abraham Lincoln

In our example project, we load the Wikipedia page of Abraham Lincoln using BeautifulSoupWebReader. This data is then indexed using VectorStoreIndex. With the indexed data, we perform queries like “What is this web page about?” or “What is one interesting fact about Abraham Lincoln?” The integration of Mistral AI through ServiceContext allows for sophisticated, context-aware responses.

Sample Code Snippet

To illustrate the practical use of LlamaIndex, let’s walk through a sample project. This project demonstrates how to index and query content from Abraham Lincoln’s Wikipedia page using LlamaIndex.

Setup

Load Environment Variables:

Use the dotenv package to load the environment variables from your .env file:

from dotenv import load_dotenv
load_dotenv()

Usage

Define the URL for Data Loading:

Specify the URL of the web page you want to index. In this case, it’s the Wikipedia page of Abraham Lincoln.

URL = "https://en.wikipedia.org/wiki/Abraham_Lincoln"

Load the Document Using BeautifulSoupWebReader:

Use the BeautifulSoupWebReader to fetch and parse the content of the specified URL.

from llama_index import download_loader

# ... previous code ...

BeautifulSoupWebReader = download_loader("BeautifulSoupWebReader")
loader = BeautifulSoupWebReader()
documents = loader.load_data(urls=[URL])

Create and Use the VectorStoreIndex for Querying:

Initialize the VectorStoreIndex and use it to create a query engine, integrated with the Mistral AI model through ServiceContext.

from llama_index import VectorStoreIndex, ServiceContext
from llama_index.llms import MistralAI

# ... previous code ...

service_context = ServiceContext.from_defaults(llm=MistralAI())
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(service_context=service_context)

query = "What is this web page about?"
response = query_engine.query(query)
print(f"RESPONSE:\n{response}")

Query for an Interesting Fact:

As a practical example, query the engine for an interesting fact about Abraham Lincoln.

# ... previous code ...

query = "What is one interesting fact about Abraham Lincoln?"
response = query_engine.query(query)
print(f"RESPONSE:\n{response}")

Example Output:

If the app runs successfully, then you can expect to see similar output in the terminal such as the following:

QUERY:
What is this web page about?
RESPONSE:
This web page is about Abraham Lincoln, the 16th President of the United States. The information provided on the page covers various aspects of his life, including his family and childhood, early career and militia service, time in the Illinois state legislature and U.S. House of Representatives, emergence as a Republican leader, presidency, assassination, religious and philosophical beliefs, health, and legacy.
QUERY:
What is one interesting fact about Abraham Lincoln?
RESPONSE:
Abraham Lincoln worked at a general store in New Salem, Illinois, during 1831 and 1832. When he interrupted his campaign for the Illinois House of Representatives to serve as a captain in the Illinois Militia during the Black Hawk War, he planned to become a blacksmith upon his return. However, instead, he formed a partnership with William Berry to purchase a New Salem general store on credit. As licensed bartenders, Lincoln and Berry were able to sell spirits, and the store became a tavern. However, Berry became an alcoholic and the business struggled, causing Lincoln to sell his share.

Conclusion

LlamaIndex is a powerful tool for developers who need to connect custom data sources to LLMs. Its ability to integrate with advanced NLP models, like those offered by Mistral AI and other AI platform companies, elevates its capability, making it an excellent choice for a variety of projects, including projects involving web scraping and data analysis.

By following the steps outlined in this post, you can start leveraging the power of LlamaIndex in your projects and unlock new possibilities in data processing and information retrieval.