How to Benchmark OCR API Accuracy Before Choosing a Vendor

Most teams testing an OCR API usually do one of two things: trust the accuracy number provided by the vendor or test a few random documents and make a decision based on the results.

But neither approach gives you a reliable accuracy number. If certain types of documents start failing in production, it can be difficult to understand why.

In this guide, we’ll look at a simple and repeatable way to test OCR accuracy using your own documents before choosing an OCR provider. You’ll learn how to create a realistic test set, prepare the correct results to compare against, run the benchmark, and understand the results.

If you want to understand what affects OCR accuracy, such as image quality, lighting, and contrast, Improve Data Accuracy with OCR covers those factors in more detail.

Here, we’ll focus on how to measure and verify OCR accuracy. If you want to try this process with an OCR API, Filestack Capture, our OCR and data capture product, can be used to run a benchmark like this.

Key Takeaways

• Don’t rely on one overall accuracy percentage. Check OCR accuracy for each document type separately.
• Use a smaller set of real, varied documents instead of testing only clean and perfect files.
• Create a manually checked ground truth to compare with the OCR results.
• Use confidence scores as an extra signal, but check whether they actually match real OCR errors.
• Test OCR accuracy regularly because your documents and scanning methods can change over time.

Let’s take a quick look at why the accuracy number provided by a vendor may not match what you see with your own documents.

Why Vendor-Quoted Accuracy Numbers Don’t Transfer to Your Documents

A vendor might claim “99% accuracy,” but that number is usually based on its own test documents. Performance can be very different depending on the type and quality of the document. For example, an OCR tool might work very well on a clean, typed invoice but perform much worse on a blurry receipt photo or handwritten form.

The documents used for vendor testing may also be cleaner and easier to read than the files your users actually upload. Real users might take photos with their phones in poor lighting, upload low-quality scans, or submit documents with different layouts.

That’s why the best way to understand how an OCR API will perform for your use case is to test it with your own documents.

Once you know that a vendor’s accuracy number is only a starting point, the next step is building a test that gives you results you can actually use.

Building a Representative Test Set

Start by collecting real documents from each type your product handles. Don’t choose only clean, high-quality files. For example, if your product processes invoices, ID cards, and handwritten forms, include all three in your test set based roughly on how often you receive them.

You also don’t need thousands of documents to get useful results. Variety is more important than volume. For example, 100 documents that include tilted scans, poorly lit phone photos, low-contrast faxes, and clean files can be more useful than 1,000 documents that all look similar.

Make sure you include difficult and messy documents. These are often where OCR tools struggle the most, and they may not be well represented in a vendor’s own tests.

Once you have a representative set of documents, the next step is deciding what the correct OCR result should look like for each one.

Establishing Ground Truth

Ground truth is a manually checked and correct version of the text in your test documents. Think of it as the answer key that you’ll compare the OCR results against.

A person should carefully review and create the ground truth for at least a representative part of your test set. It takes more time, but it gives you a reliable way to know whether the OCR output is actually correct.

Before running the benchmark, you also need to decide what counts as an error. There are two common ways to measure this:

• Character-level accuracy: Checks whether every letter, number, and character was recognised correctly.
• Field-level accuracy: Checks whether important fields, such as an invoice total, were extracted correctly, even if there are small formatting differences.

These methods can give you very different accuracy results for the same document. Choose how you’ll measure accuracy before you start and use the same method throughout your benchmark.

This is similar to how speech recognition systems use word error rate to measure errors consistently.

Once you have your ground truth and know how you’ll measure accuracy, you’re ready to run the benchmark.

Running the Benchmark

When running your benchmark, don’t combine all the results into one overall accuracy score. Keep the results separate for each document type, such as invoices, receipts, forms, and handwritten notes.

This makes it easier to see which document types the OCR API handles well and where it struggles.

The testing process is simple: run each document through the OCR API and compare the returned text with your ground truth.

Here’s a beginner-friendly Python example using the standard library:

from difflib import SequenceMatcher

def word_accuracy(ocr_text, ground_truth):

    """Rough word-level accuracy: how many words match, in order."""

    ocr_words = ocr_text.split()

    truth_words = ground_truth.split()

    matcher = SequenceMatcher(None, ocr_words, truth_words)

    matching_words = sum(block.size for block in matcher.get_matching_blocks())

    return round((matching_words / len(truth_words)) * 100, 2)


# Ground truth: what the document actually says

ground_truth = "Invoice number 48213 dated March 3 2026 total 214.50"

# OCR output: what the API returned

ocr_output = "lnvoice number 48213 dated March 3 2026 total 214.5O"

print(f"Word accuracy: {word_accuracy(ocr_output, ground_truth)}%")

To keep results organised by document type, wrap this in a small loop that stores scores in a dictionary:

In this example, SequenceMatcher compares the OCR output with the ground truth and checks how many words match.

This gives you a simple word-level accuracy score. If you later need more detailed character-level scoring, you can use Levenshtein distance to measure the differences more precisely.

Next, you can organise the results by document type:

results_by_type = {}

def record_result(doc_type, ocr_text, ground_truth):

    score = word_accuracy(ocr_text, ground_truth)

    results_by_type.setdefault(doc_type, []).append(score)


# After running every document through this...

for doc_type, scores in results_by_type.items():

    average = round(sum(scores) / len(scores), 1)

    print(f"{doc_type}: {average}% average ({len(scores)} documents)")

This stores the scores for each document type separately and then calculates their average accuracy.

The example is intentionally simple. It gives you a useful starting point without requiring a specialised OCR testing library.

If you’re building the OCR pipeline that this benchmark will test, implementing scalable cloud-based OCR explains that process in more detail. If you want to learn more about character-level and word-level scoring, this walkthrough on CER and WER is a useful next step.

Run this same test against every document in your set, broken out by type, and you’ll end up with something like this:

This example shows why separating results by document type matters. An OCR API could perform very well on clean invoices but struggle with handwritten notes. A single overall accuracy score could hide that difference.

Once you have accuracy results for each document type, the next step is understanding what those numbers actually mean for your use case.

Interpreting the Results

Different accuracy scores across document types can tell you a lot about an OCR API.

For example, if an OCR API works well with scanned forms but struggles with phone-photo receipts, you’ve found an area where it performs poorly. Whether that’s a major problem depends on how often your users upload that type of document.

You should also look at confidence scores, but don’t rely on them alone. Check whether documents with low confidence scores are also the ones where your benchmark found real OCR errors.

If low confidence scores often match real errors, they can be useful for identifying documents that may need extra review. If they don’t match your benchmark results, don’t use them as a replacement for measuring actual OCR accuracy.

Filestack’s OCR feature is a useful reference if you want to see how confidence scoring can be used in a real product. For a more technical approach to comparing error rates across documents of different lengths, see the OCR-D quality assurance methodology.

Once you have accuracy results for each document type and understand how useful the confidence scores are, you can start comparing OCR providers based on your actual needs.

Turning This Into a Vendor Comparison

The main benefit of running this benchmark is that you can compare different OCR providers fairly.

Use the exact same test documents, ground truth, and scoring method for every OCR API you’re considering. If you use different documents for each provider, the results won’t give you a fair comparison.

Accuracy also shouldn’t be the only thing you compare. Consider factors such as processing speed, cost per document, and language support. For example, an OCR API that’s slightly less accurate but much faster or cheaper might still be a better choice for your use case.

If you want to compare these factors in more detail, how to choose the best OCR data extraction software for your business explains what to consider beyond OCR accuracy.

Once you’ve chosen a provider, there’s one more important thing to remember: OCR benchmarking shouldn’t be something you do only once.

Re-Running the Benchmark Over Time

The types and quality of documents you process can change over time. You might start receiving new form types, users may upload files from different devices, or your product may expand into markets with different document formats.

Because of this, a benchmark you ran six months ago may no longer show how well your OCR API performs today.

Mobile documents can also change differently from scanned documents. Phone cameras, lighting, and the way people take photos can all affect OCR accuracy. If many of your documents come from mobile devices, choosing the best OCR SDK for your Android project explains what to consider for mobile OCR.

Run the same benchmark regularly, such as every few months or whenever your document types change significantly. This helps you keep your OCR accuracy results up to date and make sure the API is still performing well on the documents your users actually submit.

With the full benchmarking process covered, here are a few best practices and common mistakes to keep in mind.

Best Practices and Common Pitfalls

Here are a few important things to remember when setting up your OCR benchmark:

• Do keep your ground truth separate from your testing process so it doesn’t accidentally get changed based on the OCR results.
• Do include difficult, low-quality documents that are similar to what your users might actually submit.
• Don’t combine different document types into one overall accuracy score. Keep the results separate for each type.
• Don’t use different test sets when comparing OCR providers. Use the same documents and scoring method for a fair comparison.
• Don’t rely on a benchmark forever. Run it again when your document types or quality change over time.

Keeping these points in mind will help you build a benchmark that gives you more reliable and useful results.

Conclusion

An OCR accuracy number is only useful if the test behind it is reliable. A vendor’s accuracy percentage can give you a starting point, but it may not show how well the OCR API will work with your documents.

The best way to know is to test the API using your own documents and ground truth, and measure the results separately for each document type. It takes more time at the beginning, but it gives you results you can trust when choosing an OCR provider.

If you’re ready to run your own OCR benchmark, Filestack Capture is a good place to start testing your document set.

If you still have questions about setting up and using an OCR accuracy benchmark, here are answers to some common ones.

FAQ

Why don’t vendor-quoted OCR accuracy numbers hold up on my own documents?

Vendor accuracy numbers are usually based on clean, carefully selected test documents. A single overall percentage can also hide large differences between document types. Your own documents may be very different, so the OCR accuracy you get can also be different.

How many documents do I need for a meaningful OCR accuracy benchmark?

Variety is more important than having a large number of documents. A smaller test set that includes your real document types, formats, and quality levels can give you more useful results than a large set of only clean documents.

What’s “ground truth” in an OCR accuracy benchmark, and how do I establish it?

Ground truth is a manually checked, correct version of the text in your test documents. You compare the OCR results against it to measure accuracy. Before testing, decide whether you’re measuring character-level or field-level accuracy, because each method counts errors differently.

Should I trust confidence scores instead of running a full accuracy benchmark?

Confidence scores can be helpful, but don’t rely on them without testing first. Check whether low-confidence results actually match real OCR errors in your documents. If they do, you can use confidence scores as an extra signal for finding possible errors.

Does an OCR accuracy benchmark need to be re-run periodically?

Yes. Your document types and quality can change over time. You might start receiving new forms, files from different scanning devices, or more images taken on mobile phones. Because of this, a benchmark you ran once can become outdated.

Read More →