Build a PDF Vector Database with FastEmbed and Teradata

Written By James Armitage

Help yourself to the complete Python script, pdf_to_vector_db.py used in this post.

PDFs contain a huge amount of valuable business knowledge, but searching through them can be difficult, especially when users know the concept they are looking for rather than the exact words used in the document. By converting PDF content into vector embeddings and storing them in Teradata Vantage, we can enable semantic search, making it possible to find information based on meaning rather than simple keyword matching.

This Python script demonstrates how to build a simple PDF-based vector store by extracting text from documents, generating embeddings with FastEmbed, and loading the results into Teradata for future AI and Retrieval-Augmented Generation (RAG) use cases.

The script example below uses PyMuPDF to read the PDF files. Its extract_chunks function processes each page and divides the text into chunks of approximately 180 words. Each chunk keeps track of its source page, making it easier to identify where search results came from.

Next, FastEmbed converts the text chunks into numerical vectors using the BAAI/bge-small-en-v1.5 model. These vectors capture the meaning of the text, so related content can be found even when the search terms do not exactly match the document.

Once the text and embeddings are ready, the script connects to Teradata Vantage using the teradatasql driver. It creates a pdf_embeddings table and loads each record with its filename, page number, text, and embedding.

The script currently stores the embeddings as comma-separated values in a VARCHAR column. This creates the foundation for a Teradata-based vector store and can later be extended to use native vector functionality for similarity searches.

Rather than reviewing the entire script at once, let's break it down into the key steps involved in extracting, vectorizing, and loading PDF content into Teradata.

1. Import Required Modules

The script uses a small number of Python modules to handle PDF extraction, embedding generation, file management, and connectivity to Teradata. PyMuPDF is used to read PDF content, FastEmbed generates vector embeddings, and the teradatasql driver provides direct access to Teradata Vantage. Each module is imported as shown below.

import os
from fastembed import TextEmbedding
from pathlib import Path
import pymupdf as fitz
import teradatasql

2. Configure Environment Variables

Rather than hard-coding credentials and paths, the script retrieves them from environment variables. This keeps secrets out of source control and makes the solution portable across environments.

TD_HOST = os.environ["TD_HOST"]
TD_USER = os.environ["TD_USER"]
TD_PASSWORD = os.environ["TD_PASSWORD"]
TD_DATABASE = os.environ["TD_DATABASE"]
PDF_PATH = os.environ["PDF_PATH"]

3. Initialize the Embedding Model

Once the environment is configured, the next step is to load a lightweight embedding model. FastEmbed provides a simple way to generate vector representations of text suitable for semantic search and RAG applications.

model = TextEmbedding(
    model_name="BAAI/bge-small-en-v1.5"
)

4. Discover Input PDF Files

The script scans the supplied directory and identifies all PDF documents ready for processing.

pdf_files = [
    f
    for f in os.listdir(PDF_PATH)
    if f.lower().endswith(".pdf")
]

5. Extract and Chunk PDF Content

Rather than embedding an entire document, the script breaks each PDF page into manageable chunks. Smaller chunks typically improve retrieval accuracy because only the most relevant content is returned during a search.

def extract_chunks(
    pdf_path,
    words_per_chunk=180
):

    chunks = []

    with fitz.open(pdf_path) as document:

        for page_number, page in enumerate(
            document,
            start=1
        ):

            words = page.get_text("text").split()

6. Generate Vector Embeddings

Each chunk is converted into a dense vector representation. These embeddings capture semantic meaning, enabling similarity searches that go far beyond traditional keyword matching.

texts = [chunk["text"] for chunk in chunks]

embeddings = model.embed(texts)

7. Create the Teradata Table

Next, a table is created to hold both the source text and its corresponding embedding vector.

cursor.execute(f"""
CREATE TABLE pdf_embeddings (
    chunk_id INTEGER GENERATED ALWAYS AS IDENTITY,
    source_file VARCHAR(512),
    page_number INTEGER,
    chunk_text CLOB,
    embedding VARCHAR(8000)
)
""")

8. Load Embeddings into Teradata

Finally, the generated embeddings are batch loaded into Teradata where they can be searched using vector functions and combined with enterprise data.

cursor.executemany(
    """
INSERT INTO pdf_embeddings
(source_file, page_number, chunk_text, embedding)
VALUES (?, ?, ?, ?)
""",
    records
)
`

Why Store Embeddings in Teradata?

At this point, the PDF content has been transformed into vector embeddings and loaded into Teradata Vantage. This means the documents are no longer limited to traditional keyword searches. Instead, users can search based on meaning and context, making it possible to find relevant information even when the exact wording differs from the source document.

For existing Teradata customers, this approach allows unstructured document content to sit alongside structured enterprise data in a single platform. PDF-derived knowledge can be combined with operational, financial, customer, or transactional data, creating new opportunities for AI-powered search, retrieval-augmented generation (RAG), and intelligent applications.

Because the embeddings are stored within Teradata, governance, security, and data management processes already in place for enterprise data can be extended to AI workloads without introducing another specialist platform.

James Armitage