Caitlin Ostroffis the Miami Herald's data reporter and a graduate of the University of Florida. Mike Stucka is the data dork at the Palm Beach Post and is a graduate of Northeastern University, Loyola University Chicago, and a great IRE bootcamp a decade ago. Today, they contribute a guest post to INN Labs about the elections software they built.
Want to get live election results on a shoestring budget? We did. The result is a multi-component package of election reporting tools that ease the way for newsrooms to build their own scrapers, output them in a semi-standardized format and optionally use a frontend for display.
Python code parses through several pipe-delimited text files published by Florida’s Secretary of State to get statewide results, and also scrapes local results for several counties. The key was to adopt pseudo-standards from software created by The New York Times and National Public Radio, who worked together to process election results from The Associated Press.
By building scrapers against the CSV format of the Elex package, it became easy to combine multiple levels of results that could be handled and processed by a single system. You can combine your own scrapers and AP's data — or just your own scrapers, or just AP's data — to get results for your pages. The Miami-Dade County scraper was easily adapted for Kentucky, and Palm Beach County's scraper easily became West Virginia's.
The Miami Herald used JavaScript to parse out the results from a JSON file and render them to the Herald’s static media server, using fixed-size iframes of different breakwidths to bring them into its proprietary content management system. The Herald enabled cross-origin sharing, which allowed it to control caching in an .htaccess file.
There's a lot of flexibility: one of us un-called a race from a cell phone while waiting to pick up a kid from school, because the open-sourced publishing tool uses a Google Sheet to allow edits of race names, candidate names, parties, winners and runoffs.
In addition to rendering live election results, this setup also allowed both papers’ newsrooms easy access to data, from the margins between candidates to live vote count changes from newly-tallied votes. The data structures and workflow helped on election night and through a recount process that stretched more than a week. The Miami Herald and The Palm Beach Post used the code to render the current tallies as Florida inched closer to a recount after the election, as well as to drive analysis for reporting.
Chris Persaud and Mike Stucka built a Datawrapper chart of Republican candidates' lead in vote counts, using data from their election handling tool.
Existing widgets ran with stories, and new widgets were fed in near-real-time from the data we'd pulled. The code for whole-election results gave both publications a framework for pulling precinct-level results.
How cheaply done was this? We think we might have gotten about five weeks total to work on this. A much more ambitious project is Politico's open-sourced Civic, on which a development team of five focused for about five weeks each to add improvements. Our effort is far more limited, but likely also a lighter lift to get started with. Weigh your options and see what fits your organization.
Would you like to use our code? The front-end is available now and you can check out the scrapers here and here, and all are released under the MIT License. You can use this code as-is or contribute improvements. We have a sample widget collection here, and implementations at the Herald and the Post. Want to improve the project? We'd welcome that; send a pull request, drop us an email, find us in News Nerdery, call us by phone.
What we learned building an election app for INN members with the NPR visuals app template
Last week, just in time for the election, our team at the Investigative News Network (INN) launched Power Players — a state-by-state exploration of campaign finance and top political donors across the country. The project is a collaboration between thirteen INN member organizations who did their own reporting and analysis of the data we provided to them. To support this reporting, our team built a national app with easy to embed components.
As this was one of the first editorial projects we built as a team, we decided to start things off on a solid foundation by creating an app template — something that contains a library of components we might want to reuse for future projects and allows us to create those projects more easily.
Fortunately for us, NPR’s Visuals team has generously open sourced their app template, which we used as the foundation for our own. We shamelessly stole NPR’s code and set up an INN-specific template by following the steps outlined in Tyler Fisher’s excellent post “How to Setup the NPR App Template for You and Your News Org.”
It was a (mostly) painless experience but we learned some things along the way and wanted to share our experience to help others who might tackle this process in the future.
Why use NPR’s app template?
For one, we didn’t want to build our own toolkit for deploying news apps from the ground up.
When someone (or some team) builds and open sources a tool that addresses a problem you’re facing, using it will almost certainly save you time, money, blood, sweat, tears and heartbreak. Do yourself a favor and try really hard to seek out and use stuff that other smart people have already built.
The other motivating factor was that we’ve never had the chance to use NPR’s app template and a number of us have been curious. Being that this was INN’s first news app and we had a short amount of time to get something up and running, we thought this might make the perfect opportunity to take it for a test drive.
Setting up the app template for INN
Tyler acknowledges in his post that the process “seems like a lot.” But we discovered it’s just a matter of knowing where to find all the NPR bits so you can make the template your own.
One of the goals for us was to create a resource that any of our member organizations could use to bolster their coverage of the elections — either by hosting information for an entire state, or including individual power player cards in articles covering campaign finance and the election.
Thus the embeddable versions of the state and power player pages. These are essentially the same as the normal templates, with a simple full-width layout and simplified INN branding (a credit link at the bottom of each embed).
The Kentucky Center for Investigative Reporting is one INN member making great use of the app. Here is a complete list of all the members organizations participating in the project and the stories they've published (so far).
The entire project is responsive (thanks to pym.js, yet another NPR project), made to look great no matter what size container the embeddable elements get placed in.
Snags and snafus
In working with the NPR app template we encountered some things that aren’t well documented (yet).
CSS and JS pseudo-tags
Tyler covers this briefly in another blog post on the app template.
What wasn’t clear at first was how the JS and CSS pseudo-tags interact across templates.
We ran into a problem where, in separate templates, we “pushed” or queued different sets of javascript files to be rendered. In both templates, we were passing the same file name to the render function, which resulted in the second file overwriting the first when deploying.
Once you realize that JS.render outputs a file, the contents of which are determined by preceding calls to JS.push, you realize that having different calls to JS.push before rendering to the same file just won’t work.
In this case, if template 2 is rendered after template 1, “js/app-footer.min.js” will be missing “underscore.js”, potentially breaking functionality in template 1.
By making the filename passed to JS.render unique to each template, we can be sure we’re not clobbering any javascript files.
Flask’s url_for function and your project’s path prefix
Another issue we encountered was that the app template, using Flask’s default url_for function, doesn’t take into consideration your project’s path. That is, when you deploy your app to S3, it is meant to live at something like http://apps.yourdomainname.org/project-slug/ whereas the development server uses something like http://localhost:8000/ without the project slug.
For example:
<a href="{{ url_for('some_view') }}">Hey, a link to a page</a>
Renders as:
<a href="/some-view/">Hey, a link to a page</a>
What we want when deploying to staging or production is an URL that includes the project slug:
<a href="/project-slug/some-view/">Hey, a link to a page</a>
To remedy this, we created an app_template_url_for function to replace Flask’s standard url_for. The app_template_url_for figures out the current target environment (i.e. development, staging or production) and inserts the project slug as necessary.
This provides the flexibility to include assets outside of the CSS and JS pseudo-tag framework if you find yourself with the need or desire to do so.
Working with Member Organizations
The Power Players project presented other challenges, too, because of the number and diversity of INN member newsrooms participating.
First, the data needed to be analyzed and populated in the app template for about a dozen states. Campaign finance data is notoriously dirty (especially when trying to look at individual donors, as we were), plus we were combining data from different sources, so the cleaning and analyzing stage took longer than expected. Some member organizations also wanted more custom analysis, such as looking at local data only rather than statewide data as most members were. This forced us to make some compromises in the design of the app like using the state outline for California but labelling it “San Diego.”
A project such as this also takes a great deal of communication. INN’s director of data services, Denise Malan, held a kick-off conference call with members and stayed in contact with them as she was analyzing the data and the tech team was designing and building the app. We also provided members with instructions on how to add photos and customized content to the app while it was still a work in progress.
As is the case with all our editorial collaborations, INN members can publish stories according to their own timetables, and several ran their stories before the app was finished because we were working right up to the deadline on Election Day. Others have yet to publish because they choose to wait for updated data with post-election numbers.
Ideally, we would love for all members participating in a project to use our work, and we learned some valuable lessons in the process of building our first editorial app.
In future projects, we would like to have mockups of the news app ready as early as possible so our members are able to visualize what we will be providing and how this will fit with their reporting. We also want to provide a firmer deadline for launching an app so members can plan their publishing dates accordingly and leave time to implement the stuff we build. We’ll also do a better job providing the partners with documentation to make it as easy as possible for them to adopt our work.
Conclusion
We learned a lot in the process of building our first app, both about the NPR app template and also what it takes to manage a complex project working with so many partner organizations.
Will we use our fork of the NPR app template for everything? Probably not. We’ll continue to experiment and try out different things before settling on our default set of tools and templates. For projects where it’s a good fit or where we need to deploy something quick and easy, we definitely plan to use it as a solid starting point in building future apps.
Since this is our first app and we’re still learning, we’d love to hear your thoughts and feedback. You can find our team on Twitter @INNnerds or send us email at nerds@inn.org.
Last week, just in time for the election,