The Whitman Archive

The online Whitman Archive lets you browse digital versions of all the editions of Leaves of Grass (and much else). Under the hood, the webpages with the poetry are generated from a digital edition of each volume in XML format; more precisely, the edition format is a particular XML schema, the Text Encoding Initiative’s P5 format. We will soon learn a lot more about XML and TEI, so I won’t anticipate that here. But I will point out for future reference that the XML files are readily available on the Whitman Archive website, and they constitute an interesting dataset of their own: download them from this Whitman Archive page and extract the zip archives to get the XML files. Then you can see the poems we were studying in their encoded glory:

<lg type="poem" xml:id="ppp.01750">
   <relations>
      <work ref="xxx.00112"/>
   </relations>
   <head type="main-authorial">CROSSING BROOKLYN FERRY.</head>
   <lg type="section" n="1">
      <head type="main-authorial">1</head>
      <lg type="linegroup">
         <l>F<hi rend="smallcaps">LOOD-TIDE</hi> below me! I see you face to face!</l>
         <l>Clouds of the west—sun there half an hour high—I see you <lb/>also face to face.</l>
      </lg>

XML is somewhat notorious for being cumbersome to manipulate programmatically.¹ We won’t cover XML-slinging in this course. But since my aim with these blog posts is to leave breadcrumbs for possible final project ideas, I’ll make a couple of remarks about the Whitman Archive data and data like it—i.e., hand-tagged XML of literary texts. If anyone is really attached to the idea of studying XML data for a final project, I might be able to help prepare some more digestible forms…if I hear far enough in advance.

What’s special about the XML format is that it documents some of the structure of the books and their contents: each file encoding an edition of Leaves of Grass explicitly marks where each poem begins and ends; where each section of each poem begins and ends; where each line begins and ends…and many other features of the printed book. Compare that to what you’d get by copying and pasting the texts of the poems from the Whitman Archive or any other website. The XML edition allows us to investigate the structure of the text and the book in ways that a “plain,” unstructured text version would make quite difficult.

Without pretending to reveal anything very meaningful, let me give some examples of data about structure. I’ll show the corresponding R code. R is not that well-suited for complex computations on XML data, but we can still do some basic stuff using the xml2 package. The universal maxim of data analysis, KNOW THY DATA, applies very strongly here. Whatever computations you’re doing, never forget to look at the data files with your own eyes. XML is human-readable by design, and TEI-XML in particular is something you can and should look at. The Oxygen XML Editor we will work with in our XML lab is a useful tool for opening and browsing an XML file (it has tons of features beyond this).

A visual examination of the Whitman Archive files shows that the poems in each volume are encoded as <lg> elements, whose children are further <lg> elements corresponding in some cases to sections and in other to stanzas, whose ultimate descendants are <l> elements whose contents are the individual lines of poetry. (Phew!) Thus the XML version of the poem is specially geared to finding information about the organization of volumes and poems. Let’s do a trivial example: how many poems did the editors identify in each version of Leaves of Grass?

# load libraries we need
library(tidyverse)
library(xml2)
library(knitr)

# I renamed the `ppp.*.xml` files with more meaningful names by hand
# and put them in a directory called whitmanarchive, so I can find
# all the files this way:
leaves <- tibble(filename=Sys.glob("whitmanarchive/leaves*.xml"))
# load XML from files
poems <- leaves |> mutate(xml=map(filename, read_xml)) |>
    # stripping namespaces is convenient when there's only one
    mutate(xml=map(xml, xml_ns_strip)) |>
    # use XPath search to get <lg type="poem"> elements
    mutate(poems=map(xml, xml_find_all, './/lg[@type="poem"]')) |>
    # get edition year from filename for display
    mutate(edition=str_extract(filename, r"(18\d{2}(-\d{2})?)"))

poems |>
    # tally poems by seeing how long each volume's list of poems is
    mutate(n_poems=map_int(poems, length)) |>
    select(edition, n_poems) |>
    kable()

edition	n_poems
1855	12
1856	44
1860-61	172
1867	237
1871-72	190
1881-82	293
1891-92	387

The hard work here is done by xml_find_all and the perhaps obscure expression './/lg[@type="poem"]', an XPath search string which specifies “all <lg> elements with a type attribute of poem.” For each XML tree, we get back a list of references to these elements. Counting them gives us the poem counts.

As long as we’re at it, we can count lines too. Does the spread of poem lengths change over time? The tricky part of exploring this is the bookkeeping necessitated by the hierarchical data structure.

poems |>
    # for each poem, get a list of <l> elements
    # the double `map` handles the list of lists 
    mutate(lines=map(poems, map, xml_find_all, './/l')) |>
    # find the length of each of those lists
    mutate(n_lines=map(lines, map_int, length)) |>
    select(edition, n_lines) |>
    unnest(n_lines) |>
    ggplot(aes(edition, n_lines)) +
        geom_boxplot() +
        scale_y_log10(name="poem length in lines")

This repeated boxplot shows the distribution of poem lengths in lines through the successive editions. Whitman’s massive “Song of Myself” is so long that it makes visualizing the distribution difficult! I have used a log scale on the y axis, which helps without eliminating the skew altogether.² The editions from 1860 on are characterized by many more short poems than the first two; but there are sharp differences between those two editions in the scale of poems as well. This is not in itself a significant finding, but it is a starting point. In an unstructured plain text format we would be hard put to it to write a program to find the boundaries of Whitman’s poems easily, and we’d be in constant danger of being tripped up by the distinction between line breaks and really long lines that are just “overflowed” on the page.

Obviously the data contains the actual words of the poems too. The XML markup puts us in a position to study the way words are fit into lines, which is where things might actually start to get interesting in terms of Whitman’s poetry. But this post is long enough already.