The OCR Step a RAG Pipeline Needs Before It Indexes a Scanned Document

A user uploads a signed contract to your chat-with-your-documents app and asks what the payment terms are. The contract went through an office scanner, so the PDF is four pictures of pages with no text layer. Your loader pulls an empty string from each page, the chunker gets nothing, and the vector store gets nothing. Nothing errors. The chatbot answers from whatever else is in the index, and the user has no way to tell. OCR for RAG fixes the gap: read each page as an image before it reaches the chunker, and keep track of where every piece of text sat on the page.

Filestack OCR does the reading. It returns every word on a page image with its position, and the document conversion in front of it turns each PDF page into an image in the same request. We built a small ingest module around the two that drops blank pages, cuts the text into one chunk per clause, and keeps each chunk’s page and position so the answer can show where it came from. It’s one of the steps an intelligent document processing platform handles.

 

Key takeaways

• A scanned PDF or a phone photo has no text layer, so a text extractor returns empty pages without an error. OCR reads the page images instead.
• ocr takes images. Render each PDF page to PNG with output=format:png,page:N in the same request, and send photos to ocr as they are.
• A blank page returns empty text. Drop it, and mark a file with no readable page as unreadable instead of indexing nothing.
• OCR groups a page into text areas. On a contract, chunking by text area keeps each clause and its heading together.
• Store each chunk’s page number and its box as fractions of the page, and a citation can highlight the exact paragraph on any rendering of that page.

Why a scanned page indexes as blank

A scanner saves each page as a picture. Some scanners add a text layer, and many don’t. A phone photo never has one. Libraries for PDF text extraction read the text layer, so for these files they return an empty string per page:

pdftotext contract-scan.pdf -

That prints nothing for the contract above. A loader that only reads the text layer usually hands back each empty page as a valid page, so the pipeline carries on as if the contract had been indexed.

There are a few ways to get text out of a page image:

Running Tesseract yourself means rendering each page and tuning the OCR, as in OCR on PDFs in Python. We built the ingest on the last option, because the positions come back with the words, and the citation at the end of this post needs them. Keep text layer extraction for PDFs that have a text layer, and send the rest through OCR.

Read every page of the upload

OCR runs on Start, Grow and Scale, and every request needs a signed policy with the read and convert calls. An unsigned request returns 403 security required for tasks: ocr.

The OCR API takes an image and returns every text area, line and word on it. Each one carries a four-point bounding_box in the image’s pixels, and a flat text field holds the whole page. The OCR response reference has the full shape.

A PDF sent straight to ocr returns 400 ... unsupported mimetype, so the ingest puts the page conversion in front of it in the same URL, output=format:png,page:3/ocr/HANDLE, and OCR reads page 3. To find out how many pages there are, output=docinfo:true returns:

{"dimensions":{"height":792,"width":612},"numpages":4}

A page number past the end returns 400 ... Page does not exist, so the ingest loops up to numpages. This contract’s US Letter pages render at 612 by 792 pixels by default, and density:200 in the conversion renders them at 1700 by 2200. On this contract both read the same text, and the default returns sooner.

Photos go to ocr as they are. A phone photo of the contract’s page 1, taken at an angle on a desk, reads the same clauses as the scan without being straightened first.

The code is plain Node with no packages. It needs Node 20.6 or later for --env-file, and a .env file holding your API key and app secret, both from the Developer Portal:

FILESTACK_API_KEY=YOUR_API_KEY
FILESTACK_APP_SECRET=YOUR_APP_SECRET

filestack.mjs signs a short-lived URL for one file and fetches JSON from it:

import { createHmac } from 'node:crypto';

const CDN = 'https://cdn.filestackcontent.com';

// A five-minute signed URL that can read and convert this one file
export function signedUrl(tasks, handle) {
  const expiry = Math.floor(Date.now() / 1000) + 300;
  const policy = Buffer.from(JSON.stringify({ call: ['read', 'convert'], handle, expiry }))
    .toString('base64')
    .replace(/\+/g, '-')
    .replace(/\//g, '_');
  const signature = createHmac('sha256', process.env.FILESTACK_APP_SECRET)
    .update(policy)
    .digest('hex');
  return `${CDN}/security=p:${policy},s:${signature}/${tasks}/${handle}`;
}

export async function getJson(tasks, handle) {
  const res = await fetch(signedUrl(tasks, handle));
  if (!res.ok) throw new Error(`${tasks} ${handle}: ${res.status} ${await res.text()}`);
  return res.json();
}

// The page as an image, for showing a citation
export const pageImage = (handle, page) => signedUrl(`output=format:png,page:${page}`, handle);

ingest.mjs is the step itself. It reads every page, four at a time for a PDF, and returns the chunks:

import { getJson } from './filestack.mjs';
import { chunkPage } from './chunk.mjs';

const PARALLEL_PAGES = 4;

// One OCR response per page. A photo is one page. A PDF is rendered page by page first.
async function readPages(handle, mimetype) {
  if (mimetype.startsWith('image/')) return [await getJson('ocr', handle)];
  if (mimetype !== 'application/pdf') throw new Error(`No OCR path for ${mimetype}`);

  const { numpages } = await getJson('output=docinfo:true', handle);
  const pages = new Array(numpages);
  let next = 0;
  const worker = async () => {
    while (next < numpages) {
      const i = next++;
      pages[i] = await getJson(`output=format:png,page:${i + 1}/ocr`, handle);
    }
  };
  await Promise.all(Array.from({ length: Math.min(PARALLEL_PAGES, numpages) }, worker));
  return pages;
}

export async function ingest(handle, mimetype) {
  const pages = await readPages(handle, mimetype);
  const chunks = [];
  const emptyPages = [];
  pages.forEach((ocr, i) => {
    if (!ocr.text.trim()) return emptyPages.push(i + 1);
    for (const chunk of chunkPage(ocr, i + 1)) chunks.push({ handle, ...chunk });
  });
  // Nothing readable: tell the user, and index nothing
  const status = chunks.length ? 'ready' : 'unreadable';
  return { status, pages: pages.length, emptyPages, chunks };
}

Any other file type throws, so a Word document or a spreadsheet fails loudly. Convert those to PDF first, or send them to a loader that reads their text directly.

Drop empty pages before they reach the vector store

A page with nothing on it returns this, with no page_width or page_height:

{"document":{"text_areas":[]},"text":"","text_area_percentage":0}

The contract came through a duplex scanner, so pages 2 and 4 are the blank backs of the two sheets. ingest records those page numbers in emptyPages and builds no chunks from them.

A file where every page is empty gets status: 'unreadable' and no chunks. That’s the case to show the user (“We couldn’t read any text in this file”), so they can upload a clearer copy. Indexing it as an empty document is what lets the chatbot answer without it.

Chunk by clause and keep the box

OCR groups the lines on a page into text areas. On the contract, each clause heading is one text area and the clause under it is another. Text areas arrive in the grouping OCR chose, which isn’t always reading order. On this contract they run top to bottom, so each heading comes just before its clause. For pages in columns or tables, sort the areas by their boxes first or use your own chunker.

chunk.mjs makes one chunk per paragraph. It joins a short one-line area, such as a heading, to the paragraph after it, and drops anything under five words, such as the page footer:

const MIN_WORDS = 5;
const HEADING_WORDS = 6;
const MAX_WORDS = 150;

const countWords = (lines) => lines.reduce((n, line) => n + line.words.length, 0);

// One chunk per paragraph. A short one-line text area, such as a heading, joins the paragraph
// after it. Each chunk keeps its page number and its box as fractions of the page.
export function chunkPage(ocr, page) {
  const chunks = [];
  let lines = [];
  for (const area of ocr.document.text_areas) {
    lines.push(...area.lines);
    const heading = area.lines.length === 1 && area.lines[0].words.length <= HEADING_WORDS;
    if (heading && countWords(lines) < MAX_WORDS) continue;
    chunks.push(toChunk(lines, ocr, page));
    lines = [];
  }
  if (lines.length) chunks.push(toChunk(lines, ocr, page));
  return chunks.filter((chunk) => chunk.text.split(/\s+/).length >= MIN_WORDS);
}

function toChunk(lines, { page_width, page_height }, page) {
  const points = lines.flatMap((line) => line.bounding_box);
  const xs = points.map((p) => p.x);
  const ys = points.map((p) => p.y);
  const share = (n, of) => Math.round((n / of) * 1000) / 1000;
  return {
    page,
    text: lines.map((line) => line.text).join('\n'),
    box: {
      left: share(Math.min(...xs), page_width),
      top: share(Math.min(...ys), page_height),
      width: share(Math.max(...xs) - Math.min(...xs), page_width),
      height: share(Math.max(...ys) - Math.min(...ys), page_height),
    },
  };
}

On the contract, “3. Fees” and the clause under it land in one chunk, the one a question about payment terms should find.

The box is stored as fractions of the page, not pixels. OCR’s pixel positions belong to the image it read, which is 612 pixels wide for a rendered page and 4032 for a phone photo. Fractions place the same box on any rendering of the page, at any size.

Here’s the chunk for the fees clause:

{
  "handle": "Ga7wOd3rTD28bVr663iQ",
  "page": 1,
  "text": "3. Fees\nThe Customer will pay a fixed monthly fee of USD 18,400, invoiced on the first\nbusiness day of each month. Invoices are payable within forty five (45) days\nof receipt.",
  "box": {
    "left": 0.096,
    "top": 0.383,
    "width": 0.608,
    "height": 0.08
  }
}

Embed text, and store handle, page and box as metadata next to the vector. Whatever chunker you use, keep the same metadata.

Run it

try-ingest.mjs runs the ingest on one file and writes its chunks to a JSON file:

import { writeFile } from 'node:fs/promises';
import { ingest } from './ingest.mjs';

// node --env-file=.env try-ingest.mjs HANDLE MIMETYPE
const [handle, mimetype] = process.argv.slice(2);
const started = Date.now();
const { status, pages, emptyPages, chunks } = await ingest(handle, mimetype);
const seconds = ((Date.now() - started) / 1000).toFixed(1);

console.log(`${status}, ${pages} pages, empty pages ${JSON.stringify(emptyPages)}, ${chunks.length} chunks, ${seconds}s`);
for (const chunk of chunks) console.log(`  page ${chunk.page}  ${chunk.text.split('\n')[0]}`);
await writeFile(`${handle}.chunks.json`, JSON.stringify(chunks, null, 2));

node --env-file=.env try-ingest.mjs YOUR_HANDLE application/pdf

For the scanned contract, a phone photo of its page 1 and a blank page, it prints:

ready, 4 pages, empty pages [2,4], 11 chunks, 1.3s
  page 1  MASTER SERVICES AGREEMENT
  page 1  2. Term
  page 1  3. Fees
  page 1  4. Late payment
  page 1  5. Service levels
  page 3  6. Termination
  page 3  7. Liability
  page 3  8. Confidentiality
  page 3  9. Governing law
  page 3  Priya Raman
  page 3  Tomas Lindqvist
ready, 1 pages, empty pages [], 5 chunks, 1.2s
  page 1  MASTER SERVICES AGREEMENT
  page 1  2. Term
  page 1  3. Fees
  page 1  4. Late payment
  page 1  5. Service levels
unreadable, 1 pages, empty pages [1], 0 chunks, 0.7s

The contract that indexed as nothing is now 11 chunks, with the fees clause third. The first chunk on page 1 starts with the title because the title, the agreement number and the “1. Services” heading are all one-line areas, so they join clause 1.

Show the page behind each answer

When retrieval returns a chunk, the answer can show the page it came from with the paragraph highlighted, so the user can check it against the signed copy. For the payment terms question, that’s page 1 with the fees clause marked. citation.mjs builds that from the chunk’s metadata:

import { pageImage } from './filestack.mjs';

const percent = (n) => `${(n * 100).toFixed(1)}%`;

// The page a retrieved chunk came from, with the chunk highlighted
export function citationHtml(chunk) {
  const { left, top, width, height } = chunk.box;
  return `<figure class="citation">
  <div class="page">
    <img src="${pageImage(chunk.handle, chunk.page)}" alt="Page ${chunk.page} of the source document">
    <mark style="left:${percent(left)};top:${percent(top)};width:${percent(width)};height:${percent(height)}"></mark>
  </div>
  <figcaption>Page ${chunk.page}</figcaption>
</figure>`;
}

.citation .page { position: relative; display: inline-block; }
.citation img { display: block; max-width: 100%; }
.citation mark { position: absolute; background: rgb(255 200 0 / 35%); outline: 2px solid rgb(230 160 0); outline-offset: 4px; }

 

The image URL is signed for five minutes, so build the HTML when you render the answer, not when you index. If your application has a domain whitelist, the page image loads only on pages served from an address on the list. Anywhere else, the image request returns 403 Forbidden.

Start the ingest when an upload finishes

In your app, run ingest once per file, as soon as the upload completes. Your app learns that from the picker’s upload callback, which returns the handle and MIME type, or from upload webhooks. In the fp.upload event, the handle is the last part of text.url and the MIME type is text.type.

Store the chunks when the ingest returns. If ingest throws, for example on a 5xx from the CDN, retry the whole file, since nothing from it is indexed yet. The OCR response is Cache-Control: private and a repeat request runs OCR again, so re-reading a file costs another request per page. Each page is one OCR request, blank pages included. The monthly allowance is 5,000 OCR units, then $0.03 a unit.

If your image uploads already run through file processing workflows, an ocr task in the workflow reads each image at upload time. The result arrives on the fs.workflow webhook under results, keyed by the name you give the task, and the OCR JSON under data has the same shape chunkPage reads. PDFs still need the page rendering above.

Running the model and vector store on your own machine

The chunks are plain text with metadata, so nothing ties them to a hosted model. Embed them with a local embedding model, for example one served by Ollama, store them in a vector database on your own machine, and answer with a local model. The pages still go through OCR on our side, so what stays local is the index, the questions and the answers. The original files stay in your Filestack storage, and citationHtml renders the cited page from them.

Next, point try-ingest.mjs at a scanned PDF from your own users and compare its chunk count with what your current loader indexed for the same file.

Read More →