OCR Bounding Boxes in a React Review Screen

Every product that reads receipts gets the same support ticket sooner or later. The total in the expense report is wrong, and nobody can see why. The user has a number and a photo, and nothing connects the two. An OCR bounding box is that connection: the exact spot on the photo where each value was read. Draw it over the image, and the reviewer checks a field by looking at it.

On this faded receipt, OCR reads the date as 2025-09-28. The receipt says 2026. With the box drawn around the word it came from, a reviewer sees the mismatch in a second and types the right year.

Key takeaways

• Every word, line and block in the ocr response carries a four-point bounding_box in the pixels of the image OCR read. page_width and page_height give the size of that image.
• An SVG with viewBox="0 0 page_width page_height" laid over the image scales every box with the image, so there is no coordinate math in the component.
• Phone photos often store their pixels sideways with an EXIF orientation tag. Put rotate=deg:exif in front of both the OCR call and the image URL, so the boxes match the upright photo.
• On a tilted photo, find the amount for a label along the label’s own baseline, not at the nearest height.
• Every OCR call runs the task again, so read each receipt once and store the result with the reviewer’s corrections.

What the OCR response gives a review screen

The response is a tree. document.text_areas holds blocks of text, each block holds lines, and each line holds words. Every one of those carries its text and a bounding_box of four points, clockwise from the top left of the text as written:

{
  "text": "TOTAL",
  "bounding_box": [
    { "x": 473, "y": 881 },
    { "x": 603, "y": 866 },
    { "x": 606, "y": 895 },
    { "x": 476, "y": 910 }
  ]
}

Four points rather than a rectangle, because a photographed receipt is rarely square to the camera. That box comes from the tilted receipt photo further down, and its top edge climbs 15 pixels across the word.

The top level adds page_width and page_height, the size of the image the boxes are measured in, and text, every line joined with newlines. Each word has its text and its box and nothing else. There’s no per-word confidence score, so a review screen shows every extracted field for checking rather than only the doubtful ones.

The call is one segment in front of the file handle of the OCR API, and it needs a signed policy. OCR runs on Start, Grow and Scale.

https://cdn.filestackcontent.com/security=p:POLICY,s:SIGNATURE/ocr/HANDLE

The policy needs the read and convert calls. A request with no policy returns 403 security required for tasks: ocr, and a policy without convert returns 403 ... call is not allowed: (call: convert). The route below signs both. The OCR response reference has the full tree and the plain text form.

Ways to show a reviewer where a value came from

The rest of this article builds the last row in Next.js. The server reads each receipt once and finds the fields, and a React component draws them.

Run OCR once on the server and keep the result

The app secret signs the policy, so the OCR call happens in a route handler and never in the browser. These files use Next.js 16 with the App Router and filestack-js 4.3.9.

npx create-next-app@latest receipt-review --ts --app --no-tailwind
cd receipt-review
npm install filestack-js

.env.local holds the API key and the app secret, both from the Developer Portal. The secret is under Security.

NEXT_PUBLIC_FILESTACK_API_KEY=YOUR_API_KEY
FILESTACK_APP_SECRET=YOUR_APP_SECRET

lib/filestack.ts signs a policy for one handle and builds the CDN URL:

import { createHmac } from 'node:crypto';

type Policy = { call: string[]; expiry: number; handle?: string };

export function sign(policy: Policy) {
  const encoded = Buffer.from(JSON.stringify(policy))
    .toString('base64')
    .replace(/\+/g, '-')
    .replace(/\//g, '_');
  const signature = createHmac('sha256', process.env.FILESTACK_APP_SECRET!)
    .update(encoded)
    .digest('hex');
  return { policy: encoded, signature };
}

// A CDN URL for one handle, signed for that handle only.
export function signedUrl(tasks: string, handle: string, seconds = 3600) {
  const expiry = Math.floor(Date.now() / 1000) + seconds;
  const { policy, signature } = sign({ call: ['read', 'convert'], handle, expiry });
  return `https://cdn.filestackcontent.com/security=p:${policy},s:${signature}/${tasks}/${handle}`;
}

Scoping the policy to the handle means a URL copied out of the page opens that one receipt and nothing else, so the route signs the image URL as well as the OCR call. The secure file upload page covers policies and the other security settings.

lib/ocr.ts makes the call and flattens the tree for the component:

import { signedUrl } from './filestack';

export type Point = { x: number; y: number };
export type Word = { text: string; bounding_box: Point[] };
export type Line = { text: string; bounding_box: Point[]; words: Word[] };
export type OcrResult = {
  page_width: number;
  page_height: number;
  text: string;
  document: { text_areas: { text: string; bounding_box: Point[]; lines: Line[] }[] };
};

export async function runOcr(handle: string): Promise<OcrResult> {
  // rotate=deg:exif first, so the boxes match the upright image the browser shows
  const res = await fetch(signedUrl('rotate=deg:exif/ocr', handle, 300));
  if (!res.ok) throw new Error(`ocr ${handle} failed: ${res.status} ${await res.text()}`);
  return res.json();
}

export const allLines = (ocr: OcrResult) => ocr.document.text_areas.flatMap((area) => area.lines);

export function wordBoxes(ocr: OcrResult) {
  return allLines(ocr).flatMap((l) => l.words.map((w) => ({ text: w.text, box: w.bounding_box })));
}

The OCR response is Cache-Control: private, and a repeat request runs the task again. Each run uses one unit of the 5,000 a month on Start, Grow and Scale, then $0.03 a unit. So the route reads a receipt once and keeps the result. lib/receipts.ts is the stand-in for your database:

// Stand-in store. Replace these functions with your database.
import type { OcrResult } from './ocr';
import type { Field } from './fields';

export type Receipt = { handle: string; ocr: OcrResult; fields: Field[] };

const g = globalThis as unknown as { receipts?: Map<string, Receipt> };
const receipts = (g.receipts ??= new Map<string, Receipt>());

export const getReceipt = (handle: string) => receipts.get(handle);
export const saveReceipt = (receipt: Receipt) => receipts.set(receipt.handle, receipt);

app/api/receipts/route.ts takes a handle, runs OCR the first time it sees it, and returns what the screen draws. The image URL is signed fresh on every request, and the OCR result comes from the store:

import { runOcr, wordBoxes } from '@/lib/ocr';
import { extractFields } from '@/lib/fields';
import { signedUrl } from '@/lib/filestack';
import { getReceipt, saveReceipt } from '@/lib/receipts';

export async function POST(req: Request) {
  const { handle } = await req.json();
  if (typeof handle !== 'string') return Response.json({ error: 'handle required' }, { status: 400 });

  // OCR runs once per receipt. Every later view reads the stored result.
  let receipt = getReceipt(handle);
  if (!receipt) {
    try {
      const ocr = await runOcr(handle);
      receipt = { handle, ocr, fields: extractFields(ocr) };
      saveReceipt(receipt);
    } catch (err) {
      return Response.json({ error: String(err) }, { status: 502 });
    }
  }

  return Response.json({
    imageUrl: signedUrl('rotate=deg:exif', handle),
    width: receipt.ocr.page_width,
    height: receipt.ocr.page_height,
    words: wordBoxes(receipt.ocr),
    fields: receipt.fields,
  });
}

A handle that doesn’t exist makes the OCR call fail with a 400, and the route passes the message on with a 502 so the screen can show it.

Turn the phone photo upright before reading it

Phones and cameras often save a portrait photo as landscape pixels plus an EXIF tag that says “rotate this 90 degrees”. Browsers apply the tag, so the user sees the receipt upright. OCR reads the stored pixels, so the boxes come back in the sideways frame, with page_width and page_height swapped. Lay those boxes over the upright photo and they land nowhere near the text.

 

rotate=deg:exif turns the pixels upright and resets the tag. With it in front of ocr, the boxes are measured on the upright image. With it in front of the image URL too, the browser shows exactly the pixels the boxes were measured on. That is why both URLs in the route start with it.

A resize in front of ocr works the same way. The boxes come back in the resized pixels, which page_width reports. As long as the image URL and the OCR URL share the chain in front of the task, the boxes fit.

Find the amount on the same printed row

Words alone don’t make fields. Something has to decide that 28.89 is the total. On a receipt the label sits on the left and the amount on the right, often in separate text areas, so the app pairs them by position. For checking the values once they’re paired, the receipt validation rules cover totals that don’t add up and dates in the future.

The obvious rule is “the amount at the same height as the label”. That works on a flat scan. On this photo, tilted about 7 degrees, it pairs Subtotal with 2.14 and Tax with 28.89, because the right-hand column sits higher than the labels it belongs to.

 

The four points fix it. The label’s top edge gives the direction its row runs in. Walk along that direction, and the first amount within three quarters of a text height of the line is the one on the same row. lib/fields.ts:

import { allLines, type Line, type OcrResult, type Point } from './ocr';

export type Field = { name: string; value: string; boxes: Point[][]; corrected?: boolean };

const center = (box: Point[]) => ({
  x: box.reduce((s, p) => s + p.x, 0) / 4,
  y: box.reduce((s, p) => s + p.y, 0) / 4,
});
const MONEY = /^\$?\d[\d,]*\.\d{2}$/;

// The amount on the same printed row as a label, to its right.
// Rows are measured along the label's own baseline, so a tilted photo still lines up.
function amountRightOf(lines: Line[], label: Line) {
  const [p0, p1, , p3] = label.bounding_box;
  const len = Math.hypot(p1.x - p0.x, p1.y - p0.y);
  const along = { x: (p1.x - p0.x) / len, y: (p1.y - p0.y) / len };
  const across = { x: -along.y, y: along.x };
  const height = Math.hypot(p3.x - p0.x, p3.y - p0.y);
  const c = center(label.bounding_box);

  return lines
    .filter((line) => line !== label && MONEY.test(line.text))
    .map((line) => {
      const d = center(line.bounding_box);
      const dx = d.x - c.x, dy = d.y - c.y;
      return { line, ahead: dx * along.x + dy * along.y, off: Math.abs(dx * across.x + dy * across.y) };
    })
    .filter((m) => m.ahead > 0 && m.off < height * 0.75)
    .sort((a, b) => a.ahead - b.ahead)[0]?.line;
}

export function extractFields(ocr: OcrResult): Field[] {
  const lines = allLines(ocr);
  const fields: Field[] = [];

  const merchant = lines[0];
  if (merchant) {
    fields.push({ name: 'Merchant', value: merchant.text, boxes: merchant.words.map((w) => w.bounding_box) });
  }

  const date = lines.flatMap((l) => l.words).find((w) => /^\d{4}-\d{2}-\d{2}$/.test(w.text));
  fields.push({ name: 'Date', value: date?.text ?? '', boxes: date ? [date.bounding_box] : [] });

  for (const name of ['Subtotal', 'Tax', 'Total']) {
    const label = lines.find((l) => l.text.toLowerCase().startsWith(name.toLowerCase()));
    const amount = label && amountRightOf(lines, label);
    fields.push({
      name,
      value: amount?.text ?? '',
      boxes: amount ? amount.words.map((w) => w.bounding_box) : [],
    });
  }
  return fields;
}

Each field keeps the boxes of the words it came from. That is what the screen highlights, and what a correction replaces. A field the rules can’t find comes back empty with no boxes, and the screen marks it Not found. The label names and the date pattern match this receipt layout, so swap in the ones your documents use.

Draw the boxes with an SVG viewBox

The image is shown at whatever width the layout gives it, and the boxes are in OCR pixels. An SVG laid over the image with viewBox="0 0 page_width page_height" maps one onto the other. The browser scales every polygon with the image, so a box at x 1065 on a 1600 pixel page lands at the right spot on a 480 pixel image or a 1200 pixel one, with no scale factor in your code.

preserveAspectRatio="none" stretches the SVG to the image exactly. That’s safe here because the image and the boxes come from the same pixels, so their proportions match.

Let a reviewer click a box and correct the value

components/ReviewScreen.tsx uploads a photo, asks the route for the review data, and draws it. Every word gets a thin blue outline. Extracted fields are filled green, and the selected field turns amber. To fix a value, the reviewer selects its row and either types the right value or clicks the word on the receipt that holds it.

'use client';

import { useState } from 'react';
import * as filestack from 'filestack-js';

type Point = { x: number; y: number };
type Field = { name: string; value: string; boxes: Point[][]; corrected?: boolean };
type Word = { text: string; box: Point[] };
type Review = { handle: string; imageUrl: string; width: number; height: number; words: Word[]; fields: Field[] };

const API_KEY = process.env.NEXT_PUBLIC_FILESTACK_API_KEY!;
const points = (box: Point[]) => box.map((p) => `${p.x},${p.y}`).join(' ');

export default function ReviewScreen() {
  const [review, setReview] = useState<Review | null>(null);
  const [selected, setSelected] = useState<string | null>(null);
  const [status, setStatus] = useState('');

  async function onChange(event: React.ChangeEvent<HTMLInputElement>) {
    const input = event.target.files?.[0];
    if (!input) return;
    setStatus('Uploading');
    const security = await fetch('/api/filestack/policy').then((r) => r.json());
    const file = await filestack.init(API_KEY, { security }).upload(input).catch(() => null);
    if (!file) return setStatus('Upload failed');

    setStatus('Reading');
    const res = await fetch('/api/receipts', {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ handle: file.handle }),
    });
    const data = await res.json();
    if (!res.ok) return setStatus(`Could not read this file: ${data.error}`);
    setReview({ handle: file.handle, ...data });
    setStatus('');
  }

  function edit(name: string, change: Partial<Field>) {
    setReview((r) => r && { ...r, fields: r.fields.map((f) => (f.name === name ? { ...f, ...change } : f)) });
  }

  // Clicking a word on the receipt puts it in the selected field
  function pickWord(word: Word) {
    if (selected) edit(selected, { value: word.text, boxes: [word.box] });
  }

  async function save(field: Field) {
    const res = await fetch(`/api/receipts/${review!.handle}`, {
      method: 'PATCH',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify(field),
    });
    if (!res.ok) return setStatus(`Could not save ${field.name}`);
    const fields: Field[] = await res.json();
    setReview((r) => r && { ...r, fields });
  }

  const active = review?.fields.find((f) => f.name === selected);

  return (
    <div>
      <input type="file" accept="image/*" onChange={onChange} />
      {status && <p>{status}</p>}
      {review && (
        <div style={{ display: 'flex', gap: 24, alignItems: 'flex-start' }}>
          <div style={{ position: 'relative', width: 480 }}>
            <img src={review.imageUrl} alt="Uploaded receipt" style={{ width: '100%', display: 'block' }} />
            {/* viewBox uses the OCR page size, so the browser scales every box with the image */}
            <svg
              viewBox={`0 0 ${review.width} ${review.height}`}
              preserveAspectRatio="none"
              style={{ position: 'absolute', inset: 0, width: '100%', height: '100%' }}
            >
              {review.words.map((w, i) => (
                <polygon
                  key={i}
                  points={points(w.box)}
                  onClick={() => pickWord(w)}
                  style={{ fill: 'transparent', stroke: '#3b82f6', strokeWidth: 2, cursor: selected ? 'pointer' : 'default' }}
                >
                  <title>{w.text}</title>
                </polygon>
              ))}
              {review.fields.flatMap((f) =>
                f.boxes.map((box, i) => (
                  <polygon
                    key={`${f.name}-${i}`}
                    points={points(box)}
                    style={{
                      pointerEvents: 'none',
                      fill: f.name === selected ? 'rgba(250, 204, 21, 0.45)' : 'rgba(34, 197, 94, 0.25)',
                      stroke: f.name === selected ? '#ca8a04' : '#16a34a',
                      strokeWidth: 3,
                    }}
                  />
                )),
              )}
            </svg>
          </div>

          <table>
            <tbody>
              {review.fields.map((f) => (
                <tr key={f.name} onClick={() => setSelected(f.name)} style={{ background: f.name === selected ? '#fef9c3' : undefined }}>
                  <th style={{ textAlign: 'left', paddingRight: 12 }}>{f.name}</th>
                  <td>
                    <input value={f.value} onChange={(e) => edit(f.name, { value: e.target.value })} />
                  </td>
                  <td>{f.boxes.length === 0 ? 'Not found' : f.corrected ? 'Corrected' : ''}</td>
                </tr>
              ))}
            </tbody>
          </table>
          {active && <button onClick={() => save(active)}>Save {active.name}</button>}
        </div>
      )}
    </div>
  );
}

The field highlights are drawn after the word outlines and ignore pointer events. A click on a highlighted word still reaches the word polygon underneath, so a reviewer can pick any word, including one that already belongs to a field. Hovering a word shows its text in a tooltip, which helps on faded print.

Saving sends the field to app/api/receipts/[handle]/route.ts, which replaces it in the store and marks it corrected:

import { getReceipt, saveReceipt } from '@/lib/receipts';
import type { Field } from '@/lib/fields';

export async function PATCH(req: Request, { params }: { params: Promise<{ handle: string }> }) {
  const { handle } = await params;
  const receipt = getReceipt(handle);
  if (!receipt) return Response.json({ error: 'not found' }, { status: 404 });

  const update: Field = await req.json();
  receipt.fields = receipt.fields.map((f) =>
    f.name === update.name ? { ...update, corrected: true } : f,
  );
  saveReceipt(receipt);
  return Response.json(receipt.fields);
}

The stored OCR result never changes. Corrections live on the fields, so you can always tell what OCR read from what a person confirmed, and report how often each field needed fixing.

 

Put the pieces together and run it

The upload needs a policy too. app/api/filestack/policy/route.ts signs one for picking and storing:

import { sign } from '@/lib/filestack';

export async function GET() {
  // Check your own session here, so the route never signs for an anonymous caller.
  const expiry = Math.floor(Date.now() / 1000) + 300;
  return Response.json(sign({ call: ['pick', 'store'], expiry }));
}

Five minutes covers one upload. The component fetches a fresh policy each time the reviewer chooses a file, so a page left open over lunch still uploads. It’s the same pattern used when signing policies for a React picker.

app/page.tsx renders the component:

import ReviewScreen from '@/components/ReviewScreen';

export default function Home() {
  return (
    <main>
      <h1>Receipt review</h1>
      <ReviewScreen />
    </main>
  );
}

Run npm run dev and open the page from a host your application allows. Uploads from a browser are checked against the Whitelisted Domains list under Security in the Developer Portal, so add your dev host there, 127.0.0.1:3000 for example. From a host that isn’t on the list, the upload fails and the screen shows Upload failed. If you’d rather start from a ready-made picker, the React file upload components return the same handle to pass to the route.

Choose a receipt photo, and the screen draws a box on every word with the five fields highlighted. A portrait photo straight off a phone shows upright with its boxes in place. Select a row, fix the value, and save. Reloading the review for the same handle reads from the store, so OCR runs once however many times the receipt is opened.

Next, swap the five receipt fields for the ones your documents carry, and point the save route at your database. The same boxes work for invoices, forms and ID documents. Once your rules rarely need a correction, an intelligent document processing pipeline can read those documents with no reviewer in the loop. The OCR bounding box data comes back with every OCR call, so the review screen needs no extra request to draw it.

OCR bounding box FAQ

What is an OCR bounding box?

It is the set of coordinates that marks where a piece of text sits on an image. In the Filestack OCR response, every word, line and block has a four-point bounding_box in the pixels of the image OCR read.

Why does the bounding box have four points instead of a rectangle?

A photographed receipt is rarely square to the camera. Four corner points, clockwise from the top left of the text as written, follow a tilted word where a rectangle can’t.

How do you draw OCR bounding boxes over an image in React?

Lay an SVG over the image with viewBox="0 0 page_width page_height" and draw each box as a polygon. The browser scales every box with the image, so the component needs no coordinate math.

Why are the OCR boxes in the wrong place on a phone photo?

Phones often store a portrait photo as sideways pixels with an EXIF orientation tag. OCR reads the stored pixels, so put rotate=deg:exif in front of both the OCR call and the image URL, and the boxes match the upright photo.

Does Filestack OCR return a confidence score for each word?

No. Each word has its text and its bounding box only, so a review screen shows every extracted field for checking.

How much does Filestack OCR cost?

OCR is included on the Filestack Start, Grow and Scale plans with 5,000 units a month, and each unit above that costs $0.03. Every OCR call uses a unit, so read each document once and store the result.

Read More →