More on the Google Books data

To the credit of the Google N-grams team, they made their dataset available. The files are available for download in chunks from storage.googleapis.com/books/ngrams/books/datasetsv3.html. Even the 1-grams are pretty hefty—maybe they’ll fit on your hard drive, but manipulating them for analysis is not trivial: one can’t simply load them all up in R (on a personal computer, anyway). In any case, as we said in class, despite the dataset’s massive size, what is really notable here is the slenderness of the metadata. The data format, the documentation tells us, is simply

ngram TAB year TAB match_count TAB volume_count NEWLINE

This format more or less prevents any analysis in which we wouldn’t lump all the scanned volumes from a single year together. “More or less,” because sometimes there are workarounds in which certain classes of n-grams can act as “controls” for comparing trends, or as a rough proxy for certain classes of books. But this is really an object lesson in the meaninglessness of size as a measure of dataset quality. Large size means certain estimates can be very precise—but what are they estimates of?

It also has to be said that the Ngram Viewer itself has a number of “advanced” query features, documented on the help page: books.google.com/ngrams/info. With these features in mind, the Ngram Viewer can be very suggestive for all kinds of initial explorations of its underlying data, as long as you remember that this data reflects word usages extracted algorithmically from an unbalanced collection of books in large research libraries. Yet all of this documentation lies atop the black box of the Google Books database.

For the study of language, there are other datasets that are more carefully put together and documented; creating and analyzing such data is the subject of the field of corpus linguistics. These corpora¹ are generally not freely available, though you can get a sense of what they are like by exploring the free parts of english-corpora.org, especially the Corpus of Historical American English.

For our purposes, however, the most notable alternatives are the text datasets offered by HathiTrust. HathiTrust, which we’ve discussed in class, is an organization created by the same libraries that participated in Google’s Library Project to administer the giant database of book scans that was being created. When the libraries agreed to let Google scan their books, they bargained for their own digital copies to keep. The database also includes books scanned under other auspices, so HathiTrust and Google Books are not totally identical, though there is a lot of overlap. The main way researchers use HathiTrust is as a giant metadata-and-full-text-searchable database of scans of individual books: this is what you can access at hathitrust.org.

But you can also get datasets of n-grams in books in HathiTrust. In fact you can get datasets of n-grams on each page of each book. The mechanisms are not as easy to use as Google’s—and the closer you come to accessing full texts under copyright, the more locked-down the computing environment you have to use—but they give access to much richer data and metadata. Indeed this was the way Underwood constructed much of the “Life Spans of Genres” dataset.² The techniques for accessing HathiTrust data go beyond the scope of the class, but the list of offerings from the HathiTrust Research Center indicates the starting points. There are big, already-constructed datasets of word frequencies called the Extracted Features, and there are tools to construct datasets from any subset of HathiTrust volumes you can specify. For the very bold, there are final project possibilities here, but the key would be to follow Underwood’s example and stick with small-batch, artisanal data. Alas, however, the data-wrangling challenges almost certainly require more time and programming practice than you’ve been given.

HathiTrust also hosts an exploratory tool like the Ngram Viewer called the HathiTrust Bookworm. This generates time trends of individual word frequencies (not 2-grams or any higher-grams), and, crucially, lets you restrict the counts according to book metadata categories like Library of Congress classification. Unfortunately it tends to be pretty slow, so working with it can get frustrating. The real engine for research here is the Extracted Features set or the more advanced HathiTrust access modes. Moral: doing this carefully is hard—technically, conceptually, and even legally. Not every organization has the same incentives to do things carefully.