PDF chunking for RAG by section, not by character count

The default recipe in many tutorials is to extract the text, split it every 1,000 characters with 200 characters of overlap and embed the pieces. It is quick to write and works on clean prose. PDFs are rarely clean prose.

A PDF stores glyphs at positions, not paragraphs. Plain text extraction flattens that into one long string, and character splitting then cuts the string at arbitrary points. Several things go wrong at once.

• Tables become rows of numbers without their header. A chunk that contains "12.5 14.0 9.8" with no column names is useless to retrieve and dangerous to quote.
• Two column layouts interleave. If the extractor reads straight across the page, sentences from the left column alternate with sentences from the right.
• Section boundaries disappear. A chunk that starts at the end of "Termination" and continues into "Liability" matches queries about both and answers neither well.
• Headers, footers and page numbers repeat on every page and pollute every chunk with the same tokens.
• Page information is lost, so the answer cannot point to where it came from.

The fix is not a smarter character count. The fix is to parse the layout first and chunk along the structure the author already gave the document.

A layout aware parser returns blocks with a type, the text, a bounding box and a page number, in reading order. Headings, paragraphs, lists, tables, form fields and checkboxes are separate blocks. Repeating headers and footers can be identified and dropped. Two column pages come back in the order a person reads them.

With that structure, a section is simply everything between one heading and the next heading of the same or higher level. Sections nest, so each block knows its section path, for example ["Master Services Agreement", "9. Termination", "9.2 Termination for cause"].

Chunking by section means you start with these natural units and only split further when a section is larger than your token budget. When you split, you split at paragraph or list boundaries inside the section, never in the middle of a sentence and never inside a table.

In PdfParse you request this in the parse call. The response includes a chunks array where every chunk carries its text, the first and last page, the section path and a bounding box.

{
  "chunks": [
    {
      "text": "9.2 Termination for cause. Either party may terminate this Agreement on written notice if the other party materially breaches...",
      "page_start": 14,
      "page_end": 15,
      "section_path": ["Master Services Agreement", "9. Termination", "9.2 Termination for cause"],
      "bbox": [72.0, 410.5, 540.0, 742.0]
    }
  ]
}

RAG chunking with metadata is part of the Growth plan and above. The pdf parsing page shows the full chunk object next to the Markdown output.

Most vector stores let you attach metadata to each vector. Store the section path, not just the file name.

The section path does three jobs. It gives the retriever a filter, so a question about termination can be limited to chunks under a Termination heading. It gives the language model context, because a paragraph that says "the fee is waived" means something very different under "Onboarding" than under "Late payment". It also gives the person reading the answer a breadcrumb they can follow.

A common and effective trick is to prepend the section path to the text you embed, while storing the raw text separately for display.

def embedding_text(chunk: dict) -> str:
    path = " > ".join(chunk["section_path"])
    return f"{path}\n\n{chunk['text']}"

This makes short chunks much easier to retrieve. A two sentence clause has little semantic signal on its own, and its headings often carry the words a user types in a query.

Embedding models and context windows are measured in tokens, so measure chunks in tokens too. Characters per token vary by language, by the amount of numbers and code, and by the tokenizer.

Pick a maximum per chunk based on two constraints. The embedding model has a maximum input length, and longer inputs tend to blur several topics into one vector. The generation step has a context window, and you want room for several retrieved chunks plus the question and instructions.

For most document question answering, a budget somewhere in the range of a few hundred to around a thousand tokens per chunk is a reasonable starting point. Treat that as a starting point to test, not as a rule. Dense legal text often retrieves better with smaller chunks, narrative reports with larger ones.

In the parse call you set the strategy and the budget. Sections under the budget stay whole. Sections over it are split at paragraph boundaries.

import os
from pdfparse_ai import PdfParse

client = PdfParse(api_key=os.environ["PDFPARSE_API_KEY"])

result = client.parse(
    "msa.pdf",
    outputs=["json", "markdown"],
    chunking={"strategy": "section", "max_tokens": 800},
)

for chunk in result["chunks"]:
    print(chunk["page_start"], " > ".join(chunk["section_path"]), len(chunk["text"]))

Keep a minimum as well. Tiny chunks, such as a heading alone or a one line list item, are noise in the index. Merge them into the next chunk in the same section.

Overlap exists because fixed size splitting cuts sentences, and a little repeated text on each side reduces the damage. When you split on section and paragraph boundaries, sentences are not cut, so most of that reason goes away.

Overlap still has costs. Every repeated token is embedded, stored and possibly sent to the model twice. Near duplicate chunks also crowd the top results, so the retriever returns three versions of the same paragraph instead of three different pieces of evidence.

A good default with structural chunking is no character overlap at all, plus context carried by metadata. The section path already tells the model where the chunk sits. If you find answers that need the previous paragraph, add one paragraph of overlap only when a section is split, not between sections.

Tables are where character chunking does the most harm and where structural chunking gains the most. A table should never be split in a way that separates rows from their header.

Small and medium tables go into a single chunk as Markdown, with the section path and the table caption if there is one. Markdown tables keep row and column relationships readable for the language model.

Large tables need a different approach. Split them by groups of rows and repeat the header row in every piece, so each chunk is a valid table on its own. Tables that continue across pages should be merged into one logical table before splitting, otherwise the page break becomes an accidental split point.

In the PdfParse response, every table also appears in tables with its page, rows, columns and a CSV rendering. That gives you a second option for numeric questions: retrieve the chunk to find the right table, then load its CSV and compute the answer instead of asking the model to read numbers. The pdf table extraction page covers merged cells and multi page tables in detail.

def table_chunks(table: dict, rows_per_chunk: int = 25) -> list[str]:
    lines = table["csv"].splitlines()
    header, body = lines[0], lines[1:]
    pieces = []
    for start in range(0, len(body), rows_per_chunk):
        block = [header] + body[start:start + rows_per_chunk]
        pieces.append("\n".join(block))
    return pieces

Below is a complete pipeline in Node. It parses a PDF with section chunking, builds the embedding text with the section path, and stores page and bounding box next to each vector. The embedding call and the vector store are interfaces, so you can plug in whatever your stack uses.

import { PdfParse } from "pdfparse-ai";
import { createHash } from "node:crypto";

const client = new PdfParse({ apiKey: process.env.PDFPARSE_API_KEY });

type StoredChunk = {
  id: string;
  documentId: string;
  text: string;
  embeddingText: string;
  sectionPath: string[];
  pageStart: number;
  pageEnd: number;
  bbox: number[];
};

export async function indexDocument(
  file: string,
  documentId: string,
  embed: (texts: string[]) => Promise<number[][]>,
  upsert: (rows: { chunk: StoredChunk; vector: number[] }[]) => Promise<void>
) {
  const result = await client.parse({
    file,
    outputs: ["json", "markdown"],
    chunking: { strategy: "section", max_tokens: 800 },
  });

  const chunks: StoredChunk[] = result.chunks.map((c: any, i: number) => ({
    id: createHash("sha256").update(`${documentId}:${i}:${c.text}`).digest("hex"),
    documentId,
    text: c.text,
    embeddingText: `${c.section_path.join(" > ")}\n\n${c.text}`,
    sectionPath: c.section_path,
    pageStart: c.page_start,
    pageEnd: c.page_end,
    bbox: c.bbox,
  }));

  const vectors = await embed(chunks.map((c) => c.embeddingText));
  await upsert(chunks.map((chunk, i) => ({ chunk, vector: vectors[i] })));
  return chunks.length;
}

The chunk id is derived from the document id, the position and the text. Re-indexing the same document produces the same ids, so an upsert replaces vectors instead of duplicating them.

For large collections, submit files with POST /v1/batches (up to 500 files per job on Growth and above) and index each document when the parse.completed webhook arrives.

An answer that cannot be checked is hard to trust in legal, finance and compliance work. Because each chunk keeps its pages and bounding box, the answer can cite its evidence precisely.

Ask the model to reference chunk ids in its answer, then map those ids back to pages in your interface. A viewer can open the PDF at page_start and draw a highlight over bbox. Coordinates are in PDF points from the page, so scale them by the rendered width divided by the page width returned in pages.

• Show the section path above each cited passage
• Link the page number to the viewer at that page
• Highlight the bounding box on the rendered page
• Show the raw chunk text, not the text with the prepended path

Citations also make errors visible. If an answer cites a chunk that does not support it, a reviewer sees it immediately, and that example becomes a test case.

Chunking decisions should be measured, not argued. Build a small evaluation set from your own documents before changing anything.

• Write a few dozen realistic questions per document family, including questions about tables and about clauses deep in long documents.
• For each question, record the page and section that contain the answer.
• Run retrieval only, without generation, and check whether the right section appears in the top k results.
• Track recall at k and the rank of the first correct chunk for each chunking configuration.
• Read the failures. They usually point to one cause, such as a table without a header, an orphaned heading or a section that is far too long.

Compare configurations one change at a time: fixed size against section chunks, different token budgets, with and without the section path in the embedding text, with and without overlap. Only once retrieval finds the right evidence is it worth tuning prompts for generation.

Rerun the set whenever you change the parser, the embedding model or the chunking settings. Store the results with the configuration so you can see which change moved which number.