What You Don’t Know Can’t Hurt You…Unless You Don’t Ask

We were talking with a respected INN member during the Nerds’ open office hours last week. While asking a question about how to do something on his site, he said a couple of times that he doesn’t know much about website coding. But it struck me that he clearly does know a lot, he just didn’t know the answer to this particular question.

I have seen this behavior in many other people, and also in myself. When talking with people we believe know much more than us about a given topic, we sometimes minimize our knowledge up front.

I suspect we do this because we have learned from past experience that people sometimes use their status as experts to belittle us. This kind of behavior is common, especially in technical fields. Saying “I don’t know much” is a smart strategy if we suspect the expert will act like a jerk in response to our question. For many of us it's a defense reflex.

I can safely say that none of the INN Nerds will ever treat you this way. We welcome questions from all members and constituents from any level of technical knowledge, and it’s in our DNA to not act like jerks.

Not acting like a jerk is also hard-coded in the INN technology team manifesto, which outlines how and why we work. We hold ourselves accountable to this, and you should, too. Here are a few excerpts:

  • We’ll invest in education: creating curriculum and training for members, investing in our apprentices/students, and pursuing continuing education opportunities for ourselves.
  • We will be open to new tools and processes, resisting the stale comfort of “this is how we’ve always done it.”
  • We won't use snark or pedantry to exclude people from conversations.
  • We’ll never judge you or shame you for not knowing something.
  • We won’t feign surprise or jump into conversations with Well, actually...
  • In emergencies, we will send pie.

Because news technology is changing so rapidly, there are many reasons for each of us to feel we don’t know as much as we should. The pace of change is also precisely why we should ask many questions, even at the risk of exposing what we don’t know. Our guest during office hours did exactly that, and deserves to have his question (and his many other contributions as a professional) treated with respect. We will always do that.

When it comes to the web and digital technology, each of us is somewhere on the learning curve. The value of a community like the one we’ve got is that we can help each other gain the knowledge we need to improve and sustain our work. At a time like this, we should make extra efforts to communicate and collaborate.

So please use the Largo Help Desk for any site problems or requests, email us at nerds@inn.org for anything general, and sign up any time for open office hours. We’ll never shame you for not knowing something, and might even have some dumb questions ourselves.

INN Heads to Portland for OpenNews Code Convening

We're thrilled to attend the OpenNews Code Convening at Write the Docs in Portland, on May 17, 2015.

In anticipation of the event, teammate Ben Keith and I spent some time thinking about and writing down what we want to accomplish this Sunday.

Here's a quick rundown of our plan.

Our audience:

Our primary audience for the work we aim to complete this weekend is the group of developers using Largo as the base/parent theme for their news site. We want to pay special attention to INN's members that are short on time, money and technical experience.

Our goals:

  • We're aiming to start filling out the API docs/function reference, which means LOTS of cleanup of Largo source code.
  • We need technical docs for those people looking to use Largo as a framework to customize and/or enhance their site.
    • Homepage layout system.
    • Child themes (partially covered in Largo Sample Child Theme).
    • Custom Largo functionality that can be used in widgets and child theme.
    • How to setup a development environment to work on your child theme.
    • Teach people how to work with us more effectively.

Assumptions:

  • Our audience has some background in programming. They are familiar with the high-level components of computer programming (e.g., variables, functions, conditions, etc.).
  • Our audience may not be familiar with PHP, WordPress or Largo. We should use plain language and be as explicit as possible in order to help them reach their goals.

Other thoughts:

It's important to set realistic expectations for this sort of event. With 140+ open issues in the Write the Docs milestone for Largo, I'm sure there will be work left to do after the event.

Of course, even if we manage to close every last ticket in the queue, there will always be more work to do and other ways to contribute.

If you're interested in helping improve Largo, checkout our docs on how to work with us and our contributing guidelines.

And if you'd like more information on the work we'll be doing at the code convening this weekend, we also have a doc for that!

75 Days to Learn and Build With the INN Nerds

Phoenix, AZ
The view from my home office when I sign-in for scrum.

My first week with INN has me reflecting on how job circumstances influence our approach to work and how we think about it.

Working remotely

My colleagues Kaeti Hinck and Meredith Melragon have some great thoughts about making remote work work and starting a job with the INN Nerds team.

Remote work has a unique set of challenges compared to office environments, but it's made easier by terrific documentation, and a friendly and welcoming team. I'm using a co-working space in my neighborhood, but I get plenty of face time with my colleagues. I've even got my own emoticon!

Working with an end date

Every job I've ever worked has come with a finite end date either at the end of a summer or the end of a semester. It changes how you approach a job.

Whenever I start a job, I count the days I have to work. This isn't an act of counting down to escape, but instead adding up opportunities. With an end date, job security is irrelevant. I'm focused on making meaningful contributions and leaving things better than I found them.

I have 75 work days with the INN Nerds this summer (70 after today). Knowing that number helps me immensely. It dictates the kind of projects I’ll pitch to my team and budget time for meetings and bugs.

Working on iterative products instead of one-off products

Prior to joining INN, all of my design and development has been for one-off products used by a single organization. It’s much simpler. One brand. One group of users. Some corners can be cut and user experience decisions are simpler.

But iterative products are much more exciting and a better investment of development energies. Each enhancement made to an iterative project like Largo helps dozens of newsrooms, and has to be documented . Building tools intended for a wide range of skill levels and organizations forces design problems to be solved more completely. The risks and challenges are greater, but so are the rewards.

Fun stuff

I pushed an initial version of NewsPub Cookbook earlier this week, a roundup of data visualization and publishing-centric development tools. It's not quite complete -- pull requests welcome!

https://twitter.com/dryanmedia/status/598212416485216256

I’ve always liked good organization-branded desktop backgrounds (the name of the team is the INN Nerds). On my personal account I use photos, but particularly doing remote work I appreciate the reminder of “where” I am (at work, even if it is on my couch) and who I’m doing it for.

INN_BG
David's INN desktop background (h/t Kaeti Hinck). Download

Contributing to the INN Nerds docs repo using Github.com

This guide is for those of us who are not regular Terminal/shell users and those who do not want to or can't use the Gitbub client to make changes to a Github repository.

While this is written specifically for folks that want to contribute to our docs repo, the steps are the same for any repo.

Keep in mind that our docs repo is just a collection of Markdown files, making it an ideal candidate for this approach to contributing.

Since you can't run your code on Github.com to test changes, the potential to introduce bugs by editing code directly on Github.com seems extraordinarily high, in my opinion. If you have a repo with actual code, I'd suggest learning command line git or dive into the Github client (for Windows, for Mac). The learning curve is steep, but the tools are more powerful.

Getting started

First of all, you'll need a Github.com account if you don't already have one. It only takes a minute. We'll wait for you.

After you have your account set up, go to the INN docs repository.

From there, you can use the "Fork" button in the top right to create a full copy of the repository under your Github account.

Fork this repo

If you are a member of multiple organizations on Github, you will be asked where you'd like to fork the repository. In this case, I want my fork of the docs repository to live under my personal Github account:

Choose account for fork

After you choose an account, Github creates your new fork and drops you on its homepage. Notice the URL and the message just below the title of the fork indicate that this is a derivative of another repository.

Fork homepage

Step one: complete. Not so bad, right?

Editing a file

Now let's make a simple change to one of the files in our fork of the repository. In this case, we're making a superficial change to the README.md file as an example.

From the homepage of the fork, find the README.md link and click:

README.md

Once the README.md has loaded, look for the edit icon near the top on the right side of the page:

Edit README.md

Once you click the edit icon, you are presented with a text editor where you can make changes to README.md:

Editing README.md

Towards the bottom of the file, I notice that the spacing between the heading Version 0.1 and the list that follows is not consistent with the style of the rest of the document. So, I add a new line after Version 0.1.

Before and after:

Before and after

Next, we must add a message to go along with our change. At the bottom of the page, you'll see an area labeled "Commit changes". In the first input, you should provide a brief but sufficiently descriptive message regarding your change, so that anyone browsing the history of changes made to the repository will have an idea of what changed and when. For example:

Commit message complete

The secondary input is optional. I'm going to skip it here.

Once your message is in place, click "Commit changes".

Submitting a pull request

Great! So far you have your own fork of the repository and you're making changes left and right. But how do we let the maintainers of the original repository know that we have lots of good stuff for them to incorporate in the original?

The answer: submit a pull request.

To do so, go to the homepage of your fork. In my case, https://github.com/rnagle/docs. Towards the top, right side of the page find the "Pull request" link and click.

Pull request

On the page that follows, you'll see something like:

Pull request details

If you've followed these instructions, the pull request page should default to settings that essentially say, "Submit a request to the maintainers of the original repository, asking them to pull in the changes made to my fork of this repository."

We'll gloss over the details of what each of the dropdowns at the top of the page can do for the pull request's settings. Let's stick with Github's sensible defaults for now.

Towards the bottom of the page, you can see the "diff" (i.e., difference) between the original version of the README.md file (left) and the changes made to the README.md file in your fork.

Now, click the big green "Create pull request" button at the top.

You'll be presented with another dialog, asking you to include any relevant details about your changes, consequences it might have for other areas of the repository, why your change is the "right way" to do it, etc.

Pull request details

Once you add a sufficient message, click "Create pull request" (this time, with feeling!).

Hooray! We now have an open pull request on the original repository's page:

Open pull request

Note that the URL and the repository title no longer reflect our Github account or fork of the repository.

What happens next?

When you submit a pull request, a notification is sent to the maintainer(s) of the repository. They can then review your proposed changes and either merge (e.g., pull) your changes back into the original repo or reject them.

In this case, since I am a maintainer for the original repository, the pull request page in the screenshot above displays a "Merge pull request" button. This will not appear if you do not have permission to make changes directly to the original repository.

Congratulations, you're now well on your way to contributing to our docs repo (or any other repo, really). Drop us a line in the comments or @INNnerds with your questions and pro tips for contributing to projects using Github.com.

How We Redesigned INN in 60 Days

inn_logo_blue_finalToday, INN announced that we have changed our name to the Institute for Nonprofit News (formerly Investigative News Network). This change reflects the organization’s refined mission and commitment to nonprofit journalism. You can read more about that in the official press release.

When we found out about the name change, it seemed like an ideal opportunity to refresh our visual identity and relaunch inn.org to reflect our updated mission and direction moving forward. The catch? We had a little less than two months to make it happen.

Here's a behind-the-scenes look at how we redesigned both inn.org and our visual identity during the last eight weeks.

Where we started

INN's focus has always been on providing services and support to our members. As such, our own web presence and branding hadn't really been a top priority.

Old INN home page

 

INN logo

When I joined INN as design director a few months ago, I wanted to take that opportunity to establish a stronger and more consistent visual identity — something that better reflected our commitment to forward-thinking journalism.

Planning and sketching

Our team met at INN headquarters in California to start hashing out goals for the project. We discussed design direction and everything we wanted INN's brand to represent. From these conversations, we put together the initial creative brief for the identity redesign (you can read the full creative brief on Google Docs).

Together, we also finalized the project summary for inn.org — a short guiding document that we write for all of our web projects. The summary identifies goals, features, stakeholders, timeline, and user profiles.

We decided that we wanted a bold and simplified visual identity, with a website to match. The former INN homepage was trying to do too much and made it hard for users to navigate. The INN brand was being applied inconsistently across our various properties and didn't evoke any sort of organizational personality.

With initial discovery and planning done, I jumped right into sketching.

sketches_tri

I  filled up dozens of pages in my sketchbook, working through potential themes and shapes, killing my darlings left and right.

image_3

I started playing with blocks and geometric elements (representing pieces of a whole, like INN's members). Around this time, I also reached out to a Minneapolis-based designer, Anthony Lane, to work with me on developing the logo and to help us put together all of the collateral pieces that go with rebranding (think business cards and swag). Tony's clean, bold style was a natural fit with the design direction I envisioned.

Wireframes and building

image

While we worked on getting closer to our new logotype and symbol, we also needed to start building out the architecture of the new site. Working with Adam, I put together wireframes and presented a prototype to our project stakeholders (we used InVision to do this presentation remotely, which worked quite well).

wireframe_home

(Feel free to browse all the wireframes.)

After incorporating excellent feedback from our CEO, Kevin Davis, we set to work on building the site. Adam used Largo to create the general framework and began developing the custom layouts.

As Adam dove into the code, I created some mockups and style tiles to point us in the right visual direction. Here's a glimpse of the mess I lived in as I worked on brand elements and web styles concurrently (one of many Sketch pages):

sketch_doc

Visual identity

Meanwhile, Tony and I had homed in on a direction for our logo and presented it to the team. Starting from the building blocks concept, Tony suggested pursuing a flexible system — a living logo and identity that would reflect our mission.

LogoCycle_Single_0309

Starting from a simple, three-block mark, we developed a full nine-mark system.

inn_logo_preso

We got the go-ahead to pursue this direction and started to finalize typography, symbols and the color system. If you're a designer, you know that this part of the process involved a lot of back and forth that would look like nonsensical minutiae to anyone with a bird's eye view.

For the logotype, we were down to the wire choosing between GT Walsheim and FF Mark. Both are highly geometric and modern without being kitschy. We wanted something that would have personality while also standing the test of time. FF Mark won out over Walsheim — it felt like the stronger typeface overall, was less cartoonish, and had great readability at all sizes.

fonts_compareThe color system was also close to final at this point, inspired by the bold simplicity of the logo mark. We chose bright, saturated colors that would read well in print materials and on the web.

Screen Shot 2015-03-09 at 7.20.21 PM

We encountered some issues with the vertical space of the original logo in horizontal situations (specifically the website's navigation) and adjusted the lockups. The resulting system feels more balanced and will end up being more flexible in the long run.

alllogos

 

With the colors and logo locked down, Tony and I started pulling together the guidelines for use (download the full PDF).

 

logotype

 

clearspace

Launch and what comes next

Throughout this, Adam, Ryan and I were working on refining the functionality and styles of the new inn.org. Laura took the lead on content strategy and revised almost every piece of copy throughout the site.

Screen Shot 2015-03-09 at 7.29.50 PM

The entire INN staff helped us edit, revise copy, test features, and QA the site to get it ready to launch today. It wouldn't have been possible without them.

Did we get to everything on our to-do lists? Not even close. But we're tackling this project like everything we do — iteratively, learning and refining as we go. We're looking forward to seeing how people respond and will use what we learn to adjust strategy over time.

Just for fun, you know I love a good before and after shot:

beforeafter

 

Many thanks to Tony Lane, the tech team and the entire INN staff for making this possible. I'd love to hear your thoughts or suggestions for our new site and visual identity. In the end, I think our new look exemplifies who we are aiming to be — bold, forward thinking, practical, and, above all, a community.

INN-Symbol-Reversed-6

Updates to INN’s Deploy Tools

We've made a lot of changes and added a bunch of new commands to our deploy tools since the last time we checked in.

Most notably, we've added commands to help scaffold and run unit tests for your plugins AND your themes. Behind the scenes, our scaffolding and tests works as described by Will Haynes in this post.

Other changes include migration to Fabric's new-style task declaration and updated documentation.

Unit testing tools

To scaffold/set up tests for your theme:

$ fab dev wp.tests.setup:largo-dev

This command:

  • Drops and recreates your testing database (which should always be throw-away, ephemeral)
  • Copies essential test files (e.g. phpunit.xml and a tests directory with test-sample.php and bootstrap.php files) to your theme or plugin directory if they are not already present
  • Installs the WordPress test framework to /tmp/wordpress-tests-lib/includes
  • Configure a wp-tests-config.php file and place it in /tmp/wordpress-tests-lib

With those files in place, you can run tests for your theme or plugin:

$ fab dev wp.tests.run:largo-dev

Example output:

Screen Shot 2014-11-25 at 9.45.54 AM

Namespaces and aliases

The other big change we made was moving all commands to Fabric's new-style task declaration. Using new-style tasks gives us all the benefits described in Fabric's documentation on the subject.

While the deploy tools are not taking advantage of every feature of new-style tasks, they are better organized thanks to namespacing and aliases.

Here's some example output from fab -l:

Screen Shot 2014-11-25 at 10.00.07 AM

Here you can see each module (and submodules) along with defined tasks. Tasks use dot-syntax, which should feel more explicit and intuitive if you're familiar with Python (which is what Fabric is built on).

Also note that the command wp.migrations.s2m is an alias for the much-longer, sometimes-harder-to-remember command wp.migrations.single_to_multisite.

I'm very lazy, so anytime I can save some mental energy and type a bit less, I consider it a win. This is as true for commands I use very frequently as it is for commands I use once in a great while.

Documentation

We also expanded the deploy tools' documentation to include all commands. You can find the documentation on Github.

Notes

Remember, the deploy tools were built with our situation in mind. We deploy to WP Engine, so some stuff is very WP Engine specific.

For example, the wp.fetch_sql_dump command only knows where WP Engine stores recent SQL dumps. The wp.maintenance.start and stop commands assume you are deploying to a host that uses Apache and that you have a .htaccess file in the root of your FTP server.

With that said, much of toolkit will work with any host provided you have (S)FTP access, including the push-button deployment rig I wrote about previously.

If you are using or thinking about using the deploy tools, have questions or suggestions, let us know.

Point of View: Reflecting on My First Week at INN

The view from my remote office window.
The view from my remote office window.

Sitting down at a desk in your own home is not the traditional way to begin the first day of a new job.

As a new member of a remote team, the first few days can feel disorienting and humbling, but frequent access to my co-workers and the team's established practices enabled me to feel a part of the team immediately.

Posts, found here on the team blog, offered me guidance about remote work, insight into how the INN team works, and a template for reflection. I read through all the posts and appreciated the public record of INN's work and the team's history.

Orienting myself, with guidance and support, included slipping into the stream of both projects in process and our longer range plans.

The team's daily Scrum has been a huge help in getting up to speed, serving as a quick check-in on the previous day's work, a prompt to plan for the next and a barometer of all my teammates' projects. A few short minutes sheds light on the team's priorities and connects us together -- united by our preference for hoodies on chilly days.

A new working situation offers a fresh start, a bit like the first day of a new school year, an opportunity to consider two big ideas and themes – organization and documentation.

Decisions about how to organize communication, resources and time impact work flow and efficiency. Moving onto a new team in a role focused on supporting and documenting digital work, I am inspired to consider organization with design and functionality in mind.

I have started by thinking about existing structures and tools such as the team docs repository on GitHub.

Documentation is both a private and public endeavor. Many years as a student trained me to take copious notes. The challenge is to make them useful to both myself and my team.

As I reviewed the INN member sites using Largo, the WordPress platform that our team builds and maintains, I created a chart that identified the Largo features each member site incorporated. My intention is that the chart can be used a reference for my team and not solely a personal exercise in documenting my own exploration.

I am looking for relevant examples, templates and advice. Here are a couple that I've found helpful so far:

If you have any other favorites, please send them my way.

Stealing the NPR App Template for Fun and (Non-)Profit

What we learned building an election app for INN members with the NPR visuals app template

Screen Shot 2014-11-13 at 5.40.59 PMLast week, just in time for the election, our team at the Investigative News Network (INN) launched Power Players — a state-by-state exploration of campaign finance and top political donors across the country. The project is a collaboration between thirteen INN member organizations who did their own reporting and analysis of the data we provided to them. To support this reporting, our team built a national app with easy to embed components.

As this was one of the first editorial projects we built as a team, we decided to start things off on a solid foundation by creating an app template — something that contains a library of components we might want to reuse for future projects and allows us to create those projects more easily.

Fortunately for us, NPR’s Visuals team has generously open sourced their app template, which we used as the foundation for our own. We shamelessly stole NPR’s code and set up an INN-specific template by following the steps outlined in Tyler Fisher’s excellent post “How to Setup the NPR App Template for You and Your News Org.”

It was a (mostly) painless experience but we learned some things along the way and wanted to share our experience to help others who might tackle this process in the future.

Why use NPR’s app template?

For one, we didn’t want to build our own toolkit for deploying news apps from the ground up.

When someone (or some team) builds and open sources a tool that addresses a problem you’re facing, using it will almost certainly save you time, money, blood, sweat, tears and heartbreak. Do yourself a favor and try really hard to seek out and use stuff that other smart people have already built.

The other motivating factor was that we’ve never had the chance to use NPR’s app template and a number of us have been curious. Being that this was INN’s first news app and we had a short amount of time to get something up and running, we thought this might make the perfect opportunity to take it for a test drive.

Setting up the app template for INN

Tyler acknowledges in his post that the process “seems like a lot.” But we discovered it’s just a matter of knowing where to find all the NPR bits so you can make the template your own.

In fact, it took just seven commits to completely scrub NPR specific stuff from the app template.

For your reference, here is our fork of the NPR app template and the code for the Power Players app itself.

Building with the app template

The structure of the Power Players app

Our project concept was relatively simple. There are a total of five views and their corresponding templates.

An example of a individual power player embed.

They consist of:

One of the goals for us was to create a resource that any of our member organizations could use to bolster their coverage of the elections — either by hosting information for an entire state, or including individual power player cards in articles covering campaign finance and the election.

Thus the embeddable versions of the state and power player pages. These are essentially the same as the normal templates, with a simple full-width layout and simplified INN branding (a credit link at the bottom of each embed).

The Kentucky Center for Investigative Reporting is one INN member making great use of the app. Here is a complete list of all the members organizations participating in the project and the stories they've published (so far).

The entire project is responsive (thanks to pym.js, yet another NPR project), made to look great no matter what size container the embeddable elements get placed in.

Snags and snafus

In working with the NPR app template we encountered some things that aren’t well documented (yet).

CSS and JS pseudo-tags

Tyler covers this briefly in another blog post on the app template.

What wasn’t clear at first was how the JS and CSS pseudo-tags interact across templates.

We ran into a problem where, in separate templates, we “pushed” or queued different sets of javascript files to be rendered. In both templates, we were passing the same file name to the render function, which resulted in the second file overwriting the first when deploying.

Here’s what NOT to do:

{# Template file no. 1 #}
{{ JS.push('js/lib/jquery.js') }}
{{ JS.push('js/lib/underscore.js') }}
{{ JS.push('js/lib/bootstrap.js') }}
{{ JS.push('js/lib/template_1.js') }}
{{ JS.render('js/app-footer.min.js') }}
{# Template file no. 2 #}
{{ JS.push('js/lib/jquery.js') }}
{{ JS.push('js/lib/bootstrap.js') }}
{{ JS.push('js/lib/template_2.js') }}
{{ JS.render('js/app-footer.min.js') }}

Once you realize that JS.render outputs a file, the contents of which are determined by preceding calls to JS.push, you realize that having different calls to JS.push before rendering to the same file just won’t work.

In this case, if template 2 is rendered after template 1, “js/app-footer.min.js” will be missing “underscore.js”, potentially breaking functionality in template 1.

Do this instead:

{# Template file no. 1 #}
{{ JS.push('js/lib/jquery.js') }}
{{ JS.push('js/lib/underscore.js') }}
{{ JS.push('js/lib/bootstrap.js') }}
{{ JS.push('js/lib/template_1.js') }}
{{ JS.render('js/app-footer-1.min.js') }}
{# Template file no. 2 #}
{{ JS.push('js/lib/jquery.js') }}
{{ JS.push('js/lib/bootstrap.js') }}
{{ JS.push('js/lib/template_2.js') }}
{{ JS.render('js/app-footer-2.min.js') }}

By making the filename passed to JS.render unique to each template, we can be sure we’re not clobbering any javascript files.

Flask’s url_for function and your project’s path prefix

Another issue we encountered was that the app template, using Flask’s default url_for function, doesn’t take into consideration your project’s path. That is, when you deploy your app to S3, it is meant to live at something like http://apps.yourdomainname.org/project-slug/ whereas the development server uses something like http://localhost:8000/ without the project slug.

For example:

<a href="{{ url_for('some_view') }}">Hey, a link to a page</a>

Renders as:

<a href="/some-view/">Hey, a link to a page</a>

What we want when deploying to staging or production is an URL that includes the project slug:

<a href="/project-slug/some-view/">Hey, a link to a page</a>

To remedy this, we created an app_template_url_for function to replace Flask’s standard url_for. The app_template_url_for figures out the current target environment (i.e. development, staging or production) and inserts the project slug as necessary.

View the source code here and here.

Another change we made to INN’s version of the app template is modifying the Flask app’s static_folder:

app = Flask(__name__, static_folder='www/assets')

View this change in context here.

What this does is allow you to use url_for to build urls for static assets kept in www/assets.

<link rel="shortcut icon" href="{{ url_for("static", filename="icons/favicon.ico") }}" />

This provides the flexibility to include assets outside of the CSS and JS pseudo-tag framework if you find yourself with the need or desire to do so.

Working with Member Organizations

The Power Players project presented other challenges, too, because of the number and diversity of INN member newsrooms participating.

First, the data needed to be analyzed and populated in the app template for about a dozen states. Campaign finance data is notoriously dirty (especially when trying to look at individual donors, as we were), plus we were combining data from different sources, so the cleaning and analyzing stage took longer than expected. Some member organizations also wanted more custom analysis, such as looking at local data only rather than statewide data as most members were. This forced us to make some compromises in the design of the app like using the state outline for California but labelling it “San Diego.”

A project such as this also takes a great deal of communication. INN’s director of data services, Denise Malan, held a kick-off conference call with members and stayed in contact with them as she was analyzing the data and the tech team was designing and building the app. We also provided members with instructions on how to add photos and customized content to the app while it was still a work in progress.

As is the case with all our editorial collaborations, INN members can publish stories according to their own timetables, and several ran their stories before the app was finished because we were working right up to the deadline on Election Day. Others have yet to publish because they choose to wait for updated data with post-election numbers.

Ideally, we would love for all members participating in a project to use our work, and we learned some valuable lessons in the process of building our first editorial app.

In future projects, we would like to have mockups of the news app ready as early as possible so our members are able to visualize what we will be providing and how this will fit with their reporting. We also want to provide a firmer deadline for launching an app so members can plan their publishing dates accordingly and leave time to implement the stuff we build. We’ll also do a better job providing the partners with documentation to make it as easy as possible for them to adopt our work.

Conclusion

We learned a lot in the process of building our first app, both about the NPR app template and also what it takes to manage a complex project working with so many partner organizations.

Will we use our fork of the NPR app template for everything? Probably not. We’ll continue to experiment and try out different things before settling on our default set of tools and templates. For projects where it’s a good fit or where we need to deploy something quick and easy, we definitely plan to use it as a solid starting point in building future apps.

Since this is our first app and we’re still learning, we’d love to hear your thoughts and feedback. You can find our team on Twitter @INNnerds or send us email at nerds@inn.org.

Batch Processing Data With WordPress via HTTP

For a recent project, we found ourselves in need of a way to verify the integrity of an entire site’s worth of posts after a mistake in the initial migration meant some of the posts in the database were truncated/cut-off.

Our options were limited. Since our host (WP Engine) doesn’t offer shell access and we can’t connect directly to our database, we would have to write a script to accomplish via HTTP requests.

We wanted something that was at least semi-unattended. We’d also need to be careful to avoid exhausting PHP’s memory limit or running up against any PHP or Apache timeout settings.

Given the constraints, processing of posts would have to happen over several HTTP requests.

Enter batchProcessHelper a class that provides ways to accomplish all of the above.

How it works

You can find all the details on usage at the project’s Github page.

batchProcessHelper is meant to be extended. There are two methods you must override: load_data and process_item.

The first method, load_data, is responsible for accessing the data you plan to process and returning an array of serializable data that will be the data queue. The array can consist of objects or associative arrays or be something as simple as a list of IDs or range of numbers. This is entirely up to you.

The only constraints is that load_data must return a data structure that can be serialized for storage in the database as WordPress transient. This is will be your data queue.

The second method, process_item, is where you put the business code to process individual items in the queue.

This method should return true if it succeeds in processing an item, false if it fails.

How to use it

Here’s a super simple example of a class extending batchProcessHelper. You can see and learn how to run the full example on the project Github page.

include_once('batchProcessHelper.php');

class userImportProcess extends batchProcessHelper {

    function load_data() {
        return csv_to_array(ABSPATH . 'example_data.csv');
    }

    function process_item($item) {
        $this->log('Processing item: ' . var_export($item, true));
        return true;
    }
}

This example doesn’t actually do anything. In load_data, we simply load the example data from csv, converting it to an array of associative arrays.

The process_item method logs each of the associative arrays to the batchProcessHelper debug/error log.

However, this code could easily modified to create a new user for each row in example_data.csv.

function process_item($item) {
    $result = wp_insert_user($item);
    if (!is_wp_error($result)) {
        $this->log(‘Successfully created new user: ‘ . $item[‘user_email’]);
        return true;
    }
    return false;
}

Instantiating and running

So, we’ve defined our userImportProcess class, perhaps in a file named userImportProcess.php. How do we run it?

$process = new userImportProcess(array(
    'blog_id' => 99,
    'batch_size' => 10,
    'batch_identifier' => 'User Import’,
    'log_file' => '/path/to/your/file.log'
));

$process->process();

The only requirement is a batch_identifier — a name for your process.

Optionally specify the batch_size. If you don’t, batch_size defaults to 10.

Also, optionally specify a log_file — a path where debug and error messages will be sent. If you don’t specify log_file, batchProcessHelper will try to write one in /tmp.

If you want your process to run in the context of a specific blog, specify a blog_id as well.

Finally, call the process method to start.

Where do I put this stuff?

You’ll need to create a new directory in the root of your WordPress install. It doesn’t matter what it’s called, it just has to be in the root of your WordPress install.

Let’s say you create a directory named wp-scripts. You’ll want to put batchProcessHelper.php and userImportProcess.php in that directory.

Also, if you’re loading data from the filesystem (as in the example above and on the project’s Github page), you’ll want to place the data file in the appropriate location.

Finally

Visit userImportProcess.php in your browser.

If all goes well, you should see a message along the lines:

“Finished processing 10 items. There are 390 items remaining. Processing next batch momentarily…"

At this point, the page will refresh automatically, kicking off work for the next batch.

Once all batches are done, you’ll see the message:

“Finished processing all items.”

It should all go something like this:

Notes

If you have a multisite install, you'll need a Super Admin account to run code that extends batchProcessHelper. If you have a standalone install, you'll need an account with Administrator privileges.

Also, if you plan on keeping your wp-scripts directory around for any length of time, you should consider allowing only a select few IP addresses to access the directory. Refer to the project's README for more information.

Showing How We Work

We're excited to announce the release of a new collection of documents that show how our team works.

In putting this collection together, we wanted to go beyond a style guide (also important, and we're working on that, too) to try to explain as much as possible about the process and values that inform our work: What makes our team unique, how we recruit and hire new team members, our internal process, and how outside members or clients interface with our process.

Opening up our process has a number of benefits for us as a team and, we hope, for others, as well.

Codifying existing processes in one place makes them easier to reference and helps keep the team on the same page. It allows new hires to get up to speed faster and gives prospective employees insight into how we work, our mission and values, and whether working with us would be a good fit.

It also helps external partners, like INN members and our consulting clients, learn how to work with us most effectively.

Above all, we hope that collecting this information in one place will be useful to other organizations who are building and managing similar teams.

This is especially important for us because INN's mission includes a strong educational component and we want to do everything we can to help our members make smart technology decisions.

By showing not only our process, mission and values, but also expanding the scope of this collection to include things like job descriptions, how we run meetings and the tools we use to work effectively as a remote team, we are attempting to create a sort of "missing manual" for running a news apps and technology team.

We hope, over time, to make this manual even more comprehensive as we refine our process and our thinking. And we hope that providing this model will make it a little easier on organizations and managers traveling down this road in the future.

We're grateful to the teams that have come before us who have written and released documentation that served as a source of inspiration for various parts of our team docs, particularly:

- ProPublica's News App and Data Style Guides
- The NPR Visuals Team's app template, coding best practices and manifesto
- Guides and process docs from The Chicago Tribune's News Apps Team
- MinnPost's style guide

This is a work in progress and we plan on updating frequently so we'd really value your feedback and contributions. Feel free to contribute to the project on GitHub or send us suggestions to help us improve our existing docs (or to propose new sections you'd like to see us add).