How to Auto-Close Orders from Proof of Delivery Photos with OCR

It’s mid-afternoon on route 14, and the driver is at stop 7. They hand over the pallet, get the delivery note signed, lay it on the truck bed and take a photo. The proof of delivery photo goes into the app, and the truck pulls away. Later that day, someone in the office opens the photo, reads the order number and marks the order delivered. If the photo is blurred, cut off or of the wrong note, nobody knows until then. By that time the driver is several stops away, and the customer gets a phone call.

That office step is what intelligent document processing software for shippers is meant to remove, and two Filestack features cover it between them. OCR reads the order number off the photo while the driver is still at the door, so a bad photo goes straight back for a retake. Document Detection keeps a straightened, shadow-free copy of the note for anyone who reads it later. We built a small Node service around the two to show how they fit together. It’s one of the narrower intelligent document processing solutions, with one document and one decision.

 

Key takeaways

• ocr reads a photographed delivery note as the driver took it, angle and shadow included, so point it at the uploaded photo.
• Find the order number by its pattern and by the label beside or above it. A second number with the same pattern, such as a back order, has a different label.
• Check the photo inside the driver’s request. An unreadable photo, a missing note or the wrong order goes back to the driver as a retake while they are still at the stop.
• doc_detection with store keeps a cropped, straightened, shadow-free copy of the note. Store the original photo alongside it.
• A photo with no note in it still returns a crop from doc_detection, so the OCR text, not the crop, decides whether a note is there.

What the photo has to prove

The photo has one job. It has to show the signed note for the order the driver is delivering. There are three ways it can fail:

• The order number can’t be read.
• There’s no note in the photo at all.
• The note belongs to a different order.

Each one has the same fix, which is another photo, and the driver is the only person who can take it. So the check can’t wait for someone in the office. It runs in the request the driver’s app makes when they tap “Delivered”, and the service below answers inside that request.

Four ways to match a photo to an order

There are four usual ways to tie the photo to the order:

We went with OCR, because it reads any printed note, including the ones a supplier or a customer prints. The catch is in the last column. Besides the order number, a delivery note carries a customer account, a PO number and sometimes a second number in the same format as the order. So the service needs a rule for which number is the order, and that rule gets a file of its own.

Read the photo as the driver took it

The first thing the service does is read the photo. OCR and Document Detection run on Start, Grow and Scale, and both need a signed policy on every request. For ocr the policy needs the read and convert calls, and an unsigned request returns 403 security required for tasks: ocr.

The photo at the top of this post is taken at an angle, on a ribbed truck bed, with a shadow across the page. The OCR API reads it as it is, so there’s no crop or straighten step first. It returns every text area, line and word on the page, each with a four-point bounding_box in the photo’s pixels, plus a flat text field. The OCR response reference has the full shape.

We wrote the service in plain Node with no packages. It needs Node 20.6 or later for --env-file, and a .env file holding your API key and app secret, both from the Developer Portal:

FILESTACK_API_KEY=YOUR_API_KEY
FILESTACK_APP_SECRET=YOUR_APP_SECRET

filestack.mjs is the only file that talks to Filestack. It signs each request and makes the two calls the service needs, one to read the photo and one to store a clean copy of the note:

import { createHmac } from 'node:crypto';

const CDN = 'https://cdn.filestackcontent.com';

function signedUrl(tasks, handle, calls) {
  const expiry = Math.floor(Date.now() / 1000) + 300;
  const policy = Buffer.from(JSON.stringify({ call: calls, handle, expiry }))
    .toString('base64')
    .replace(/\+/g, '-')
    .replace(/\//g, '_');
  const signature = createHmac('sha256', process.env.FILESTACK_APP_SECRET)
    .update(policy)
    .digest('hex');
  return `${CDN}/security=p:${policy},s:${signature}/${tasks}/${handle}`;
}

async function getJson(url) {
  const res = await fetch(url);
  if (!res.ok) throw new Error(`${res.status} ${await res.text()}`);
  return res.json();
}

// Every line and word on the photo, with its position on the page
export const readText = (handle) => getJson(signedUrl('ocr', handle, ['read', 'convert']));

// Crops, straightens and cleans up the note, then stores it as a new file
export async function storePage(handle) {
  const url = signedUrl('doc_detection/store', handle, ['read', 'convert', 'store']);
  return (await getJson(url)).handle;
}

The OCR response is Cache-Control: private, and a repeat request runs the task again. So the service reads each photo once and keeps what it needs from the response.

At this point the service can turn the photo into lines of text with positions. It can’t tell which of those lines is the order yet.

Pick out the order number by its label

Matching ORD- followed by digits finds every order-shaped number on the note, and the note at the top of this post has two: the order being delivered, and a back order that follows on the next delivery. The label next to each number is what tells them apart.

Notes print that label in one of two places. Some put it on the same line, as in Order No: ORD-2026-58213. Others put it in a box above the value. In that layout, the flat text field lists the labels and values out of order. On the second carrier’s note below, it reads CONSIGNMENT, ROUTE, HL-77-40912, E3, so the flat text alone can pair a value with the wrong label.

 

The bounding boxes settle it. A value belongs to a label that sits directly above it, in an overlapping column, less than two lines away. find-order.mjs takes the OCR response and looks for that label. It accepts an order number only when the label is on the same line just before it, or directly above it:

// Order numbers as this carrier prints them, such as ORD-2026-58213
const ORDER = /\bORD-\d{4}-\d{5}\b/;
const ORDER_LABEL = /^order\s*(no\.?|number|#)?\s*:?$/i;
const NAME_LABEL = /^(print\s+)?name\s*:?\s*/i;

const linesOf = (ocr) => ocr.document.text_areas.flatMap((area) => area.lines);

function box(line) {
  const xs = line.bounding_box.map((p) => p.x);
  const ys = line.bounding_box.map((p) => p.y);
  return { left: Math.min(...xs), right: Math.max(...xs), top: Math.min(...ys), bottom: Math.max(...ys) };
}

// The label sits directly above the value: overlapping columns, less than two lines apart
function labelAbove(label, value) {
  const a = box(label);
  const b = box(value);
  const lineHeight = b.bottom - b.top;
  return a.left < b.right && b.left < a.right && a.bottom <= b.top + lineHeight / 2 && b.top - a.bottom < lineHeight * 2;
}

export function findOrderNumber(ocr) {
  const lines = linesOf(ocr);
  const labels = lines.filter((line) => ORDER_LABEL.test(line.text.trim()));
  for (const line of lines) {
    const match = line.text.match(ORDER);
    if (!match) continue;
    const before = line.text.slice(0, match.index).trim();
    if (ORDER_LABEL.test(before)) return match[0];
    if (!before && labels.some((label) => labelAbove(label, line))) return match[0];
  }
  return null;
}

export function findReceivedBy(ocr) {
  const line = linesOf(ocr).find((l) => NAME_LABEL.test(l.text));
  return line ? line.text.replace(NAME_LABEL, '').trim() || null : null;
}

On the truck bed note, the back order line starts with “Back order”, which isn’t an order label, so it never matches. On the second carrier’s note, the ORDER NUMBER box sits right above its value, so that one matches as well.

To fit your own notes, change ORDER to your order number format and ORDER_LABEL to the label your notes print. If your notes come from several carriers, add each carrier’s label to the pattern.

findReceivedBy reads the printed name next to the signature, not the signature. The stored photo and page show the signature itself.

Answer the driver before they leave the stop

The driver’s app uploads the photo the way it uploads anything else, then sends the photo’s handle to the service for that stop. The service reads the photo, checks the order number against the stop, and answers in the same request. Here’s server.mjs:

import { createServer } from 'node:http';
import { readText, storePage } from './filestack.mjs';
import { findOrderNumber, findReceivedBy } from './find-order.mjs';

// Stand-in for your dispatch database: the order each stop on today's route expects
const stops = new Map([
  ['route-14-stop-7', { order: 'ORD-2026-58213', status: 'out_for_delivery' }],
]);

async function closeStop(stopId, handle, lastTry) {
  const stop = stops.get(stopId);
  if (!stop) return [404, { error: 'unknown stop' }];
  if (stop.status === 'delivered') return [200, { action: 'closed', order: stop.order, proof: stop.proof }];

  const ocr = await readText(handle);
  const order = findOrderNumber(ocr);
  let problem = null;
  if (!ocr.text) problem = 'No delivery note in the photo';
  else if (!order) problem = 'Order number not readable';
  else if (order !== stop.order) problem = `This note is for ${order}`;

  if (problem && !lastTry) return [422, { action: 'retake', reason: problem }];

  const proof = await storePage(handle);
  Object.assign(stop, {
    status: problem ? 'needs_review' : 'delivered',
    reason: problem,
    proof,
    photo: handle,
    receivedBy: findReceivedBy(ocr),
    at: new Date().toISOString(),
  });
  console.log(stopId, stop);
  return [200, { action: problem ? 'review' : 'closed', order: stop.order, proof }];
}

createServer(async (req, res) => {
  const match = req.url.match(/^\/stops\/([\w-]+)\/proof$/);
  if (req.method !== 'POST' || !match) {
    res.writeHead(404).end();
    return;
  }
  let raw = '';
  for await (const chunk of req) raw += chunk;
  let input = {};
  try {
    input = JSON.parse(raw) ?? {};
  } catch {}
  const { handle, lastTry = false } = input;
  if (typeof handle !== 'string') {
    res.writeHead(400, { 'Content-Type': 'application/json' }).end(JSON.stringify({ error: 'handle required' }));
    return;
  }
  try {
    const [status, body] = await closeStop(match[1], handle, lastTry);
    res.writeHead(status, { 'Content-Type': 'application/json' }).end(JSON.stringify(body));
  } catch (err) {
    console.error(match[1], handle, err.message);
    res.writeHead(502, { 'Content-Type': 'application/json' }).end(JSON.stringify({ action: 'retry', reason: 'Could not read the photo' }));
  }
}).listen(8788, () => console.log('proof of delivery service on :8788'));

stops stands in for your dispatch database, with one stop on today’s route. The app acts on the action in the response:

A driver with a bad photo sees the reason on screen and the camera opens again while they’re still at the door. Sometimes a better photo isn’t possible, and lastTry is how the app says so. Send it after two retakes, or when the driver taps a “Customer has gone” button. The stop is then marked needs_review with the reason, the photo and the stored page. A person looks at those few stops instead of every stop.

A second tap on “Delivered” for a closed stop returns the stored result without reading the photo again.

Keep a straightened copy of the note

This is where Document Detection comes in. The original photo is the evidence, so the service keeps its handle. A support agent or a customer disputing a delivery has an easier time reading a straightened copy.

doc_detection finds the page in the photo, crops it, straightens it and turns it into black text on white paper. On the photo at the top of this post, that takes the shadow off the page. The document detection parameters include preprocess:false, which keeps the color instead.

Chaining store after the task saves that page as a file of its own. The policy then needs the store call as well as read and convert. Without it, the request returns 403 ... call is not allowed: (call: store). Each request stores a new copy, which is why we run it only once, when the stop closes.

The crop can trim the outer margin of a note, as it does on the left edge of the page above. So the service keeps both handles, the page for reading and the photo for the whole scene.

Document Detection includes 1,000 images a month, then $0.20 an image, and OCR includes 5,000 units a month, then $0.03 a unit. One closed stop uses one of each, and a retake uses one OCR read.

When there’s no note in the photo

A driver who photographs the parcel on the doorstep instead of the note sends a photo with no text in it. ocr returns an empty text, and the service asks for a retake with “No delivery note in the photo”.

doc_detection isn’t the test for this. Given a photo with no paper in it, it still returns a crop of whatever rectangle it finds, here the parcel, and with coords:true it still returns a plausible box. So we let the OCR text decide. For receipts and invoices, the detection on receipts and invoices post covers the same task.

The other two failures play out like this. A thumb over the order number leaves every other line readable and no order number, so the driver sees “Order number not readable”. Yesterday’s note, photographed by mistake, reads cleanly and returns “This note is for ORD-2026-58197”.

Run it locally

You now have the three files the service needs. Start it:

node --env-file=.env server.mjs

try-stop.mjs plays the driver’s app. It stores a photo with the REST upload endpoint, then posts the handle to the service for a stop:

import { createHmac } from 'node:crypto';
import { readFile } from 'node:fs/promises';
import { basename } from 'node:path';

// Plays the driver app: stores a photo, then asks the service to close the stop
const [stopId, path, lastTry] = process.argv.slice(2);
const expiry = Math.floor(Date.now() / 1000) + 300;
const policy = Buffer.from(JSON.stringify({ call: ['pick', 'store'], expiry }))
  .toString('base64')
  .replace(/\+/g, '-')
  .replace(/\//g, '_');
const signature = createHmac('sha256', process.env.FILESTACK_APP_SECRET).update(policy).digest('hex');
const params = new URLSearchParams({ key: process.env.FILESTACK_API_KEY, policy, signature, filename: basename(path) });

const stored = await fetch(`https://www.filestackapi.com/api/store/S3?${params}`, {
  method: 'POST',
  headers: { 'Content-Type': 'image/jpeg' },
  body: await readFile(path),
});
if (!stored.ok) throw new Error(`store: ${stored.status} ${await stored.text()}`);
const { url } = await stored.json();
const handle = url.split('/').pop();

const res = await fetch(`http://127.0.0.1:8788/stops/${stopId}/proof`, {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ handle, lastTry: lastTry === 'last' }),
});
console.log(res.status, await res.json());

Pass the stop and a photo, and add last for the final try:

node --env-file=.env try-stop.mjs route-14-stop-7 note.jpg

For the photo of the right note, the script prints 200 and { action: 'closed', order: 'ORD-2026-58213', proof: '...' }. The service logs the stop with its photo, stored page and receivedBy. A photo of the parcel prints 422 with No delivery note in the photo. A stop that isn’t in stops returns 404, and a request without a handle returns 400.

The photos in this post are 4032 x 3024, the size a 12 MP phone camera saves. OCR reads the same text from this note at 2,000 pixels wide. So shrinking photos in the app before upload can save the driver’s mobile data. Check that on your own notes before you turn it on.

When the app uploads after the driver drives on

Some drivers work without signal, and the app queues photos until it reconnects. Nobody is at the door by then, so a check inside the request has nothing to offer. For those apps, build a workflow with one ocr task instead, and start it on each stored photo with the run_workflow task, signed with the runWorkflow and convert calls. The fs.workflow event carries the same OCR response under the task’s name, so findOrderNumber works unchanged, and the stops that fail go to review. The file processing workflows page covers building one.

The workflow result arrives only as a webhook notification, so the service needs a public URL for that route. For the capture step in the app itself, the phone camera capture post covers taking the photo on the phone and uploading it.

Next, replace stops with a lookup on today’s route in your dispatch database, and add the label each carrier you work with prints to ORDER_LABEL.

Frequently asked questions

Does OCR need the proof of delivery photo cropped or straightened first?

No. The OCR API reads a photographed delivery note as the driver took it, angle and shadow included, so there’s no crop or straighten step first. Point ocr at the uploaded photo.

How do you find the order number when a delivery note has more than one order-shaped number?

Find it by its pattern and by the label beside or above it. A second number with the same pattern, such as a back order, has a different label. When the label sits in a box above the value, the bounding boxes show which label the value belongs to.

What happens when the driver’s photo is unreadable or shows the wrong note?

The service checks the photo inside the driver’s request. An unreadable order number, a missing note or a note for a different order returns 422 with a reason, and the camera opens again while the driver is still at the stop. After two retakes, or when the customer has gone, the app sends lastTry and the stop is marked needs_review for a person.

Can Document Detection tell whether a photo has a delivery note in it?

No. Given a photo with no paper in it, doc_detection still returns a crop of whatever rectangle it finds, and with coords:true it still returns a plausible box. The OCR text decides whether a note is there. An empty text means no note, and the driver is asked for a retake.

How much do OCR and Document Detection cost per delivery?

OCR includes 5,000 units a month, then $0.03 a unit, and Document Detection includes 1,000 images a month, then $0.20 an image. One closed stop uses one of each, and a retake uses one OCR read.

Read More →