As we process our first large batch of data (thank you for the great progress made!) we’re interested in evaluating the accuracy of the work, and testing our workflows for aggregating multiple transcriptions on the same person into one. We began by looking at data for which we have verified entries—the name and the serial number.
Now we’re moving onto looking at how we aggregate text that takes on categorical values. A good rule of thumb in data processing is to start by looking at simple things first. In this case, simple is a question that doesn’t have a lot of possible answers. Hair color is an interesting one, and perhaps the results will be of interest. What color hair did New Zealand soldiers have?!
We were again encouraged by the apparent accuracy of the data. Of 3,216 transcriptions of hair color just 15 were not a hair color. Interestingly they were a religious profession which is beside it on the form. We can probably find the hair color in the entry for religion for these men. Transposition of entries from a known place is a problem, but a workable one. Of course, we should emphasize that an error rate of < 0.5% is really good.
The example here also shows another funny issue. A couple of people transcribed this as “light possum”, and we can see why! Context suggests that it is “light brown”, and the majority of transcribers recorded it that way. But since we use “flaxen” for blonde, why not use “possum” for grey hair.
To the data! We find that New Zealanders were a dark haired people, with 80% of the soldiers being described as having “brown,” “black,” or “dark” hair. The “fair” including flaxen were a small minority.
|Hair colo(u)r||Frequency||Percent||Cum. Pct.|
|Red or auburn||21||2.92||98.06|
Working through this data was a useful trial for our aggregation of more complicated fields. We hope you found it interesting, and look forward to sharing more results with you as we continue our research. Thanks for all your hard work, and lets keep transcribing!
Evan Roberts for the Measuring the ANZACs Research Team (@evanrobertsnz).
We reported last week that we were analyzing the 545,000 fields transcribed from our first year of collaborating with our citizen army to tell the stories of and measure the ANZACs. Among the first things we’re looking at is this question: how accurate are citizen transcriptions. If you’re reading this blog post you’re already among the most engaged volunteers, and so you will be thinking “I take a great deal of care with my transcriptions!” However by having our project open to the world we trade off the benefit of openness for the cost of having some less experienced volunteers who may not read the instructions, or take as much care.
So our question is to measure the measurers: how accurate are they, are you?! We need to show you’re doing a job that’s as good as we could achieve with methods traditionally accepted in the academic world for transcribing this form of material, such as employing undergraduate or graduate students.
We have an amazing opportunity to measure the accuracy of transcription in Measuring the ANZACs because our index to the records includes three variables or fields: first name, surname, and serial number that we see in the data as well and that are being transcribed. So we have a “gold standard” we can compare to. In a lot of citizen science projects the research team have to hand validate a sample of their data (we’ll do a bit of that too).
We measure “similarity” between your transcriptions and the gold standard with a metric called a Jaro-Winkler score that measures the similarity of the “string” (a string is a sequence of letters or numbers). 1 represents complete accuracy, and 0 total inaccuracy.
But to make the scores more concrete consider that Charles and Cjarles have a similarity of 0.91; Gerry Reid and Henry Reid have a similarity of 0.8, and B and Benjamin have a similarity of 0.74. We can still make a lot of use of transcriptions that score as low as 0.7. Note in particular that “B” is an initial and this may just reflect that the paper had “B” instead of “Benjamin”. The transcription is accurate, but the original person writing it down should have put the full name.
Here’s what we found and presented at last week’s meetings of the Social Science History Association: Citizen scientists are creating highly accurate transcriptions on complex forms with messy handwriting. We have a little work to do to compare this to our previous data collection by graduate students, but on first impressions this is of a very high standard.
Transcription in citizen science is relatively new, and it is important to show that it produces acceptably accurate data. We think it has, and we look forward to now proceeding with our substantive analyses with a high degree of confidence in what we’re analyzing. As we continue with our validation of the data we’ll keep you informed. Now, lets get back to Measuring the ANZACs! Transcribe!
|Given name||Surname||Serial number|
|Mean similarity score to truth (1 = absolute accuracy)||0.98||0.98||0.98|
|Proportion absolutely accurate||0.87||0.84||0.89|
|Proportion with similarity score > 0.9||0.95||0.95||0.95|
|Proportion with similarity score > 0.8||0.97||0.98||0.98|
|Proportion with > 3 words in string (more likely problematic transcriptions)||0.0029||0.003||0.003|
We got a batch of data out from the system recently, with more than half a million transcriptions of questions and answers.One of the automatically recorded pieces of data when you transcribe is the time and date.
Here is the raw data. We had a great opening week, and then in late 2015 we had some website problems. You can see how that affected our numbers, probably along with it being the holidays. What are the two huge jumps in January and February? Those are the days immediately following our interviews on TV One and TV Three. If we were on TV every night think how quickly we’d be done. Many hands do make light work!
As you can see from this graph there are some day-to-day fluctuations that make it harder to pick out the trend. It looks like the fluctuations are weekly, reflecting when people have more or less time (or faster internet at work?).
There are some formal statistical tests for the “periodicity” of data, but for simplicity lets just look at the average over a week. We take the rolling average which allows us to get away from worrying about the audience for our project which is in multiple timezones. In effect our weekends are nearly three days because the New Zealand weekend starts when it’s still Friday in the United States (we know our community comes largely from the UK, US, and New Zealand).
Here’s what that looks like. We got a bit of a bump from the news coverage early in the year, and then things drifted down.
Lets focus then on the more normal part of the year to see what’s happening over time. We got a bit of a bump from some presentations and publicity in April/May 2016, and things slowed down a bit after that. We’re really gratified to see that since our presentations in Wellington in July 2016 there’s been a sustained growth over the last couple of months.
Is this enough to build a memorial and research database by the end of the centenary of World War I? Not yet! Every day we’re now completing the equivalent of about 2-3 files, depending on how many lines there are on the statement of services and History Sheets. By completing we mean that every field has been done three times.
How does this compare to other Zooniverse projects? Very well indeed. We’re not as popular as Galaxy Zoo or Snapshot Serengeti in classifications per day, but we’re around where Penguin Watch is, which is great. But our challenge is that each image (a page in a file) contains so much information. On each image we’re trying to get 5-30 pieces of information, whereas other projects have a smaller number of tasks per image. It’s just the nature of the material. So we need to keep going, and we need you, our force of transcribers to help us recruit more people. If you think there’s an analogy to the war and getting men to the front, you’re right. We need volunteers (and we need conscripts).
The goal is both audacious and achievable. Audacious because a complete transcription of the main pages of 140,541 files is a big task. Achievable because there are more than 3 million adults in New Zealand. If every high school student and their family adopted two soldiers and marked and transcribed their file we’d be done. If every one with an ancestor who served marked and transcribed one file we’d be done. Help us do that. Reach out now to people you know and share the goal with them. Thank you for a great foundational year, let’s keep going further and faster!
In our name Measuring the ANZACs has a vision of measuring all the ANZACs. We’ve started with the “NZACs”. Today we’re going to talk about the “A” for Australia. Our collaboration with you to transcribe the New Zealand records has relied on
- The preservation and organization of the personnel files for 90 years by the NZ Defence Force
- The digitization of all 3.7 million page images by Archives NZ.
- Our collaboration with the Zooniverse
You are helping us with step 3, but if the literal century of work had not been done by others, we could not do anything. Records could easily have been destroyed, or remain hard to access as paper files. What are the prospects for measuring all of the ANZACs?
The National Archives of Australia has scanned and published as digital images the personnel records of the Australian Imperial Force (AIF), which acted as the country’s overseas armed forces during the First World War. The 376,000 AIF files survive intact and contain a more consistent set of documents than the British, and are remarkably similar to the New Zealand records.
There is a long history of Australian historians and scholars using the personnel files. In fact within the Commonwealth, Australian researchers pioneered the use of First World War service records as an historical source.  Access to the service files necessarily relies on metadata that act as index fields. The AIF metadata includes name, place of birth, place of enlistment, next of kin and service number. This metadata is slightly richer than the NZ records. But there remains much information of research interest additional to the metadata that must be transcribed on an image by image basis. What if we want to know which Australians served at Gallipoli, or were wounded? We need to transcribe all the records to find out!
Some research has already been done with the Australian records by Kris Inwood—a researcher on the Measuring the ANZACs team—and other colleagues. In Australia the attestation documents were transcribed for every soldier, nurse or officer whose name began with the letter B. This letter has the advantage of being easily discernible in most hand-writing. Moreover, B-surnames are common among relevant ethnicities. The resulting databases includes 35,000 Australian records, or about 10% of the AIF.
As with the New Zealand research we described in an earlier blog post Inwood has used the personnel data to study the Australian standard of living. This research, which follows an earlier literature using US Civil War personnel records and British army records extending back to the eighteenth century, begins with the premise that the patterns of adult stature for a large enough population reflect living conditions in childhood.
The basic idea in this literature is straightforward. If a child is exposed unduly to various diseases, or does not receive a sufficiently nutritious diet, or engages in child labour too early in life, then physical growth is stunted to some degree, and the individual will not realize her or his genetically-endowed potential stature. The growth-retarding impact of adverse childhood conditions may be ameliorated if conditions improve but the ‘catch-up’ is likely to be incomplete. If child growth is impaired for a sufficient number of individuals in a population, then average adult stature for the population will be diminished.
In a recent comparative paper Inwood and colleague John Cranfield show that Australia’s economic growth slowed in the 1890s but child health nevertheless improved relative to Canada, judging by the divergence in adult stature for labourers born in the 1890s (Figure 3). Opposing forces were at work in the two economics. An erosion of working class standards was the dark side of the biggest economic boom in Canadian history. For Australia, in contrast, the dramatic slowing of economic growth in the 1890s had many consequences but it clearly did not impede the advance of living standards. Indeed, the reduction in immigration and population growth probably reduced the intensity of some diseases and contributed to the improvement in child well-being.
As with the New Zealand research we described a few months ago, one of the limitations of the existing research is that we cannot make fine distinctions between cities or years. We would love as researchers to know whether Sydney was healthier than Melbourne, and whether cool Hobart or warm Brisbane was worse than both. Researchers beyond the Measuring the ANZACs team can use a complete transcription of the AIF records to understand who fought in the AIF and in what battles, and to uncover more of the hidden history of Aboriginal participation in the AIF.
The ANZAC name itself shows that while New Zealand and Australia were different countries they had a deeply shared history. Measuring the ANZACs can uncover not only the ANZAC campaign in World War I, but the intertwined personal connections of migration and ancestry that linked the two countries.
Onwards with transcribing, Measuring the ANZACs forces!
 http://www.naa.gov.au/collection/explore/defence/service-records/army-wwi.aspx January 24 2016. The archival reference is National Archives of Australia. First Australian Imperial Force Personnel Dossiers, 1914-1920. Series B2455. Another useful database was created by scanning the rolls of those who embarked for overseas service: https://www.awm.gov.au/research/people/nominal_rolls/first_world_war_embarkation/introduction. The printed embarkation rolls comprise a subset of all AIF and offer a subset of information recorded in the personnel records.
 L.L. Robson, “Origin and character of the first A. I. F., 1914-1918: Some Statistical Evidence,” Historical Studies 15 (1973): 737-749; G. Whitwell, C. de Souza and S. Nicholas, “Height, Health and Economic Growth in Australia, 1860-1940,” in Health and Welfare during Industrialization, eds. Roderick Floud and Richard H. Steckel (Chicago: University of Chicago Press, 1997), 379-422; G. Whitwell and S.Nicholas, “Weight and welfare of Australians 1890-1940,” Australian Economic History Review, 41 (2001): 159-75; Ralph Shlomowitz, “Did the mean height of Australian-born men decline in the late nineteenth century?” Economics and Human Biology 5 (2007): 484-488.
 Pat Thornton and Sherry Olson, “A Deadly discrimination among Montreal infants, 1860-1900,” Continuity and Change 18 (2001): 95-135.
 Only attestation papers are available for Britain and South Africa; we were able to consult a wider range of documents including reports of medical examinations for Australia, Canada and New Zealand.
 Robert W. Fogel, Stanley L. Engerman, Roderick Floud, Gerald Friedman, Robert A. Margo, Kenneth Sokoloff, Richard H. Steckel, T. James Trussell, Georgia Villaflor and Kenneth W. Wachter, “Secular Changes in American and British Stature and Nutrition,” Journal Interdisciplinary History 14 no. 2 (Autumn 1983): 445–81; Robert A. Margo and Richard H. Steckel, “Heights of native-born whites during the antebellum period,” Journal of Economic History 43 no. 1 (1983): 167–74; Roderick Floud, Kenneth Wachter, and Annabel Gregory. Height, health and history: Nutritional status in the United Kingdom, 1750-1980 (Cambridge: Cambridge University Press, 1990).
 Barry Bogin, Patterns of human growth, 2nd ed. (Cambridge: Cambridge University Press, 1999); Richard H. Steckel, “Stature and the standard of living,” Journal of Economic Literature 33 (1995): 1903-40; Richard H. Steckel, “Biological Measures of the Standard of Living” Journal of Economic Perspectives 22 (2008): 129–152.
James M. Tanner, Foetus into man: Physical growth from conception to maturity (Cambridge: Harvard University Press, 1978); James M. Tanner, A history of the study of human growth (Cambridge: Cambridge University Press, 1981).
 Terry Copp, The Anatomy of Poverty: The Condition of the Working Class in Montreal 1897-1929 (Toronto: McClelland & Stewart, 1974); David Gagan and Rosemary Gagan,”Working-Class Standards of Living in Late-Victorian Urban Ontario: A Review of the Miscellaneous Evidence on the Quality of Material Life,” Journal of the Canadian Historical Association 1 (Spring, 1991): 171-194; Martin Tétreault, “Les maladies de la misère. Aspects de la santé publique à Montréal, 1880-1914,” Revue d’histoire de l’Amérique française 36 no. 4 (1983): 507-526; Thornton and Olson, 95-135.
 R.V. Jackson, “Trends in Australian living standards since 1890,” Australian Economic History Review 32 (1992): 24–46; R.V. Jackson and Mark Thomas, Height, weight, and well-being: Sydney schoolchidren in the early twentieth century,” Australian Economic History Review 35, no. 2 (September 1995): 39-65; Ian W. McLean and Jonathon Pincus, “Did Australian living standards stagnate between 1890 and 1940?,” Journal of Economic History 43 (1983), 193–202.
 W. A. Sinclair, “Economic growth and well-being: Melbourne 1870–1914,” Economic Record 51 (1975): 153–73.
One of our motivations for working with you all on Measuring the ANZACs is to bring rare stories to light. We can do this together by indexing the files on more dimensions than are currently listed, so small groups of men with similar wartime experiences can be identified and studied together, or individually.
One example of how military experiences interact with our research on health and longevity is the question of prisoners. We were reminded of this when a member of the Measuring the ANZACs community shared this file of Thomas Michael Lynch (58995) with us on Talk (join the conversation and share your finds and questions!)
Being a prisoner was quite rare among New Zealand’s World War I soldiers in contrast to World War II, about much which more has been written. It seems that the experience of these men needs to be told. It’s a great topic for an Honours or MA thesis!
Being a prisoner and surviving to come home distinguishes these men in an interesting way. How did it affect their health later on? You might suppose that being a prisoner would shorten your life, owing to the stress and deprivation. On the other hand, sometimes being a prisoner meant a lower risk of further injury on the battlefield. A fascinating study of American Civil War prisoners by Dora Costa found that “Among severely stressed former Union Army prisoners of war (POWs), the effect that dominates 35 years after the end of the Civil War depends on age at imprisonment. Among survivors to 1900, those younger than 30 at imprisonment faced higher old-age mortality and morbidity and worse socioeconomic outcomes than non-POW and other POW controls, whereas those older than 30 at imprisonment faced a lower older-age death risk than the controls.” But studies of World War II POWs from Australia and the US have typically found that being a prisoner of the Japanese was bad for men’s health in the long-term.
World War I may have been different again, but we don’t know. There have not, as best we can tell, been any studies of what happened after the war to World War I prisoners. As with suicide in which you are helping us do important research, the files of New Zealand’s prisoners could tell us a lot. Help us tell their stories, and what happened to them after the war. Onwards with transcribing!
It’s our first birthday! This time a year ago we were incredibly excited to get the project publicly launched after several months of development time, and feedback on the beta.
Anniversaries and birthdays are a great time to take stock and share with our friends and partners what we’ve learned in the past year, and where to from here.
Our two biggest achievements are related. First, the system we built on Scribe has worked to enable the separation of marking and transcription, and having you help us with the transformation of structured but messy records on paper into structured machine-readable records.
Social science and humanities databases are hard enough to construct with teams of research assistants or classes working alongside you. One needs systems with some optimum combination of flexibility and rules. For example we shouldn’t see anything but numbers in answer to a question “How old in years is this man”. But in reality, we do see other things. Not often, but the times you see notes beside the man’s age we need some ability to capture that information, because it’s probably going to be interesting. Multiply these kinds of challenges across the 147 different variables we’ve identified in the personnel records, and taking it to the crowd is going to be challenging.
Our approach has been to collect nuggets of information. You mark a field, then it’s available for transcription. Repeat 147 fields x 3 iterations of each transcription x 140,000 soldiers (and that’s an undercount, because some fields repeat. People were wounded or transferred between units many, many times …)
Second, because the system is relatively easy to use we’ve built a great community of markers and transcribers. Thank you! We couldn’t do this all by ourselves, and the data we’re creating will be a shared resource for scholarship, a memorial to the men who are recorded herein, and a way to open up the stories of the men to a much wider audience. At the end of the project we’ll have the records indexed by much more than the name and serial number they’re currently indexed by. Want to find all the men born in Wellington? It’ll be easy (so get transcribing! Go now!)
No one gets through their first year without a few challenges. We’d like to thank you for bearing with us while the server was quite unstable in the first couple of months! It was a little trying.
The challenges have been smaller than the achievements, but we know we haven’t had a perfect year. In the first instance getting 3.5 million images across the Pacific from Archives New Zealand computers to ours has been time consuming. On this end they have to be processed into a new database before they can appear on the site. It’s not as easy as just pointing our interface at the complete collection you can find on Archway.
This means we still don’t have everyone up. The promise that you can come along and easily find your relatives or other people you’re interested in hasn’t quite been realized. We know this is important, and we’re working on it. One of the things that makes Measuring the ANZACs different from ecology or astronomy projects is that our data represents known entities. People! Actual people you’ve heard about. The architecture of the Zooniverse was built around a slightly different type of data. The good news is, again, we’re working on it. At the Zooniverse we know that transcription is important to bring citizen science to the social sciences and humanities, and these are part of the changes we have to make (go check out the other great transcription projects, then come back here).
We’ve also found that transcription is hard. There’s less immediate gratification in a little bit of transcription, than classifying an image with animals in it, or identifying stars. I know, because I’ve done the other thing. Finding the community of people who are interested and committed to the slightly harder work of transcription is an ongoing joy of reaching out through social media, giving talks, and mentioning what we’re doing whenever we can. Which leads us to
We’re incredibly grateful for the support of the Zooniverse, and the leadership team there of Lucy Fortson and Chris Lintott who are a joy to work with. Archives NZ and Auckland Museum have been incredibly helpful as well in spreading the word, and without Archives incredible efforts to scan all this material we wouldn’t have a project. Measuring the ANZACs is a Zooniverse at UMN project, and we thank UMN for their financial support of that intiative (check out the new UMN project Mapping Change about biodiversity). Our efforts to get all of the men’s stories transcribed is fundamentally about studying a population, and the support of the Minnesota Population Center is also important to us.
Beyond the team we work with, we’ve been thrilled to join the wider community of people interested in how we can transform handwritten text into useful machine-readable data in the social sciences and humanities. We’ve talked to classes at Carleton College, faculty and students at George Mason (Center for History and New Media) and Macalester, and to THATCamp Twin Cities. It was an incredible honor to give the ANZAC Lecture in April at the Center for Australian, New Zealand and Pacific Studies at Georgetown. In Wellington we were glad to give talks at the National Library, Wellington High School to the Wellington History Teachers Association, and Victoria University.
We started incorporating material uncovered by our citizen scientists into our research on post-war suicide; an important and sad topic where we can make a start with small numbers of men’s stories. Your help in uncovering these rare stories is helpful. We are still a little while from having enough completely transcribed men to fold them into our measurements of the ANZACs (so get transcribing!).
The road ahead
Millions of fields transcribed is an incredible achievement, and we thank you for it. It’s not yet enough because there are tens of thousands of men whose files haven’t been touched. But we do have enough data to start addressing a very fundamental methodological problem: how accurate is citizen transcription? This takes us back to the starting point of this blog post. Our alternative is enlisting (really, conscripting or employing) the help of undergraduate and graduate students. Looking at what you have transcribed we think the quality is pretty good, but we need to, you know, measure that and put a number on it.
Unusually for a transcription project we have two validated “gold standard” data elements in our data. The men’s names and serial numbers are part of the “metadata” from the NZ Defence Force and Archives New Zealand, and represent the truth about how names were spelled and the assigned serial number. What did you actually transcribe, how much do the errors matter, and where do they occur? These kinds of metrics will be important in the acceptance of citizen science data as a scholarly source, and we’ll be presenting first results at the Social Science History Association conference in Chicago in 6 weeks.
It’s been a great year, we’ve learned a lot, met a lot of new people, and collaborated to build a platform to remember and understand 140,000 men’s live. Here’s to just a few more years of transcribing, and many years of remembering, understanding and studying. Thank you!
We’re excited to announce that we’ll be holding a workshop for teachers in Wellington (New Zealand, just to minimise confusion!) on 22 July at 1pm at Wellington High School. Please email us to register your interest, or with questions.
The workshop will introduce you to citizen science, and the Measuring the ANZACs project specifically, and suggest ways in which you can use the project to develop classroom and out-of-classroom (do we still call it homework?!) learning experiences.
The workshop will be led by research team member, Evan Roberts, from the University of Minnesota. Evan grew up in Wellington, went to school at Wilton (now Otari) School and Onslow College, and Victoria University, and taught New Zealand history at Victoria from 2007-2011. The workshop will be interactive, so please bring a laptop if you can, so you can participate in the discussion and design of assignments and learning experiences. If you don’t have a laptop, you’ll still be able to participate by working with other participants.
We’re holding this workshop because we believe citizen science can be a powerful tool for learning. Measuring the ANZACs asks you to read old handwriting, so it’s most suitable for high school students. If you’re a university or polytech lecturer, you are also welcome to attend. The difficulty of the assignments we suggest can easily be adapted for students from age 13 to 23.
Our subject matter suggest that History and Social Studies teachers will find this the most valuable. But we’re a multi-disciplinary team and we’re interested in getting the data from our project into the classroom for use in maths and statistics classes as well. Working with Measuring the ANZACs can provide a great introduction to survey and database design and applied social statistics as well.
The workshop will also help us meet our audacious goal to have data entered on every NZEF soldier by November 2018. There are ~140,000 files in our data. Everything is entered by three people. There are ~280,000 high school students in New Zealand. If every high school student worked with their parents to study and transcribe the records of two soldiers we’d be done (and then some). Please join us at the workshop, and please enlist in the effort!