8 Ways to Overcome OCR Errors when Searching Newspapers

Everyone who has searched newspapers online will fail to find something. It happens incredibly often. The stakes are high for genealogy researchers, where finding an article about an ancestor can make a huge difference in filling out a family tree.

I have often heard researchers say "I can't find a single article about my ancestor, even though I have searched for hours!"

Laying competency aside as a factor, the biggest reason is that scanning of one and two hundred year old newspapers, either from paper or from microfilm, produces way less than optimal results.

More importantly, one must know that searching through an index created by humans who have read the source material and then typed the index is far superior to having a machine/software scan and process a dusty old newspaper. Yet the massive size of newspaper collections prevents the creation of the index manually. You must expect inferior results and set your expectations accordingly.

Please take a look at the following list, and hopefully some of these errors and anomalies will provide you with some hints to overcome them and actually find what you are looking for. There are many others - but these are ones that I have personally experienced:

1. Hyphenated words were often used because of fixed width type as well as the experience and capability of the typesetter. Hyphens are less utilized today but were a staple years ago. Take that into consideration if you are searching for a surname or other search criteria with many letters in one word. Try splitting the search into two words where the hyphen may have been normally used.
2. If there is an "h" in your search term, try exchanging a "b", since b's and h's are quite similar and can "confuse" the OCR process. As an example, searching the California Digital Newspaper Collection for one of my surnames - "Braunhart" yields 1,507 results. Replacing the "h" with a "b", hence searching for "Braunbart" yields 96 results - for the SAME person. That is approximately another 6%!
3. For a similar reason as "h" and "b" are confused - the same holds true of "c" and "e".  I have not had as many difficulties with this pair as with "h" and "b".
4. Likewise, lower case m's and n's are often confused. The m's are often converted to several combinations of letters.  Also r's and n's can be confused.
5. I's in lower as well as upper case can often be converted to slashes or exclamation points and the numeral 1. And vice versa.
6. if the original newspaper is "dirty," by that I mean there is excessive ink or the scan is dark - many times spaces will be scanned but not presented as spaces. There are a variety of strange characters that may be picked up.
7. If the newspaper was scanned and then processed directly with OCR, that is one pass. If the newspaper was scanned to microfilm and scanned again and then OCR'd that is two passes. Thus a two pass operation has the potential to have a decreased quality of results. There isn't much that you can do about it - but it is nice to know.
8. This one is not really about scanning, but is more of a cultural challenge. Until the last few decades, women were not referred to by their maiden or given name in newspaper publications. So in my mother's case, after she was married, any newspaper articles cited her as Mrs. Robert J Marks, or Mrs R. J. Marks, not Muriel Marks. So looking for "Muriel Marks" with the exception of her obituary - would have led to zero results.

So don't be discouraged by "lack of results" from doing online newspaper searches. You just need to "outsmart" OCR and try various combinations to get to those elusive ancestors. Be persistent.

An additional help would be more crowdsourcing to correct OCR errors and improve the text. An example is reCaptcha processing that is used for Google Books.

Another crowdsourcing example that I personally have used is that of correction on the actual online newspaper site, such as the aforementioned California Digital Newspaper Collection. In this example, registered users can provide edited text that is then incorporated into future searches. Kind of like a newspaper-related "pay it forward." This capability is provided on that site and many others from the fine folks at Elephind.com, who created the software used by the California collection as well as many other online newspaper sites.


For many more details about scanning, OCR and related subjects please read Scanning FAQ by Project Gutenberg.

Another excellent article is Analysing and Improving OCR Accuracy in Large Scale Historic Newspaper Digitisation Programs, from the March/April 2009 publication of D-Lib magazine.

Good luck - be persistent and have reasonable expectations.

---