Improving text quality with automatic majority editions

2017-02-10

Liam R E Quin (W3C)

This work is licensed under a Creative Commons Attribution 3.0 License, with attribution to W3C.

Copyright ^®2017 W3C® (MIT, ERCIM, Keio, Beihang)

Improving text quality with automatic majority editions

Making OCR More Awesome

Liam R E Quin (W3C)

2017-02-10

This work is licensed under a Creative Commons Attribution 3.0 License, with attribution to W3C
Copyright ® 2017 W3C^® (MIT, ERCIM, Keio, Beihang)

Context

Getting texts into XML;
Making advanced use of text content;
Having finite time and resources.

The Books

photo of 32book spines

Retyping

Over 16,000 pages of text

My OCR Attempt

My flat-bed scanner would destroy the binding...

leaving me with thousands of pieces of paper...

But I did get very high quality with Abby FineReader, using higher resolution images.

A chance discovery

I happened one day to hit on a copy of one of the texts online

Archive.Org

I found 157 files (there are more today)
Sorted them into scan sets and by volume (unreliable!)
Chose the best one by (very quick) inspection
Started a conversion project into XML.

Conversion Quality

Found lots of errors.
Conversion scripts do lots of checks...
Brainwave!

Majority Edition

Match up corresponding lines of text;
Choose the best.

Let’s look…

[script is shown]

Conversion strategy

Use rcs/git/whatever;
Use make or a script to run the conversion;
Lots of small stages...

Small stages

Each stage is a separate small script, e.g. mark page breaks, find footnotes, unhyphenate;
Each stage logs any problems;
Conversion continues where possible;
Use make or && to stop on a fatal error. On non-Unix systems. have each stage write an OK file at the end.

Results

Not as good as proofreading or rekeying;
Massively faster and cheaper;
Much better than using just one OCR run.

Future Work

Context-sensitive fixes;
Try to find Greek and improve it;
Go beyond the book (see talk on Saturday).

Links

http://words.fromoldbooks.org/Chalmers-Biography/
http://words.fromoldbooks.org/xml/
liam@w3.org