Caitlin Ostroff is the Miami Herald's data reporter and a graduate of the University of Florida. Mike Stucka is the data dork at the Palm Beach Post and is a graduate of Northeastern University, Loyola University Chicago, and a great IRE bootcamp a decade ago. Today, they contribute a guest post to INN Labs about the elections software they built.
Want to get live election results on a shoestring budget? We did. The result is a multi-component package of election reporting tools that ease the way for newsrooms to build their own scrapers, output them in a semi-standardized format and optionally use a frontend for display.
Python code parses through several pipe-delimited text files published by Florida’s Secretary of State to get statewide results, and also scrapes local results for several counties. The key was to adopt pseudo-standards from software created by The New York Times and National Public Radio, who worked together to process election results from The Associated Press.
By building scrapers against the CSV format of the Elex package, it became easy to combine multiple levels of results that could be handled and processed by a single system. You can combine your own scrapers and AP's data — or just your own scrapers, or just AP's data — to get results for your pages. The Miami-Dade County scraper was easily adapted for Kentucky, and Palm Beach County's scraper easily became West Virginia's.
The Palm Beach Post built a front end by baking out the pages using Flask. Three scrapers ran on Election Night and were beaten into more than 500 different widget embeds for 11 newspapers, with each complete scrape-parse-build run taking about 40 seconds total.
The Miami Herald used JavaScript to parse out the results from a JSON file and render them to the Herald’s static media server, using fixed-size iframes of different breakwidths to bring them into its proprietary content management system. The Herald enabled cross-origin sharing, which allowed it to control caching in an .htaccess file.
There's a lot of flexibility: one of us un-called a race from a cell phone while waiting to pick up a kid from school, because the open-sourced publishing tool uses a Google Sheet to allow edits of race names, candidate names, parties, winners and runoffs.
In addition to rendering live election results, this setup also allowed both papers’ newsrooms easy access to data, from the margins between candidates to live vote count changes from newly-tallied votes. The data structures and workflow helped on election night and through a recount process that stretched more than a week. The Miami Herald and The Palm Beach Post used the code to render the current tallies as Florida inched closer to a recount after the election, as well as to drive analysis for reporting.
Existing widgets ran with stories, and new widgets were fed in near-real-time from the data we'd pulled. The code for whole-election results gave both publications a framework for pulling precinct-level results.
How cheaply done was this? We think we might have gotten about five weeks total to work on this. A much more ambitious project is Politico's open-sourced Civic, on which a development team of five focused for about five weeks each to add improvements. Our effort is far more limited, but likely also a lighter lift to get started with. Weigh your options and see what fits your organization.
Would you like to use our code? The front-end is available now and you can check out the scrapers here and here, and all are released under the MIT License. You can use this code as-is or contribute improvements. We have a sample widget collection here, and implementations at the Herald and the Post. Want to improve the project? We'd welcome that; send a pull request, drop us an email, find us in News Nerdery, call us by phone.