Measuring Impact

This is a blog post summarizing the "Measuring Impact" session of the Berkman Center Teaching Data Storytelling for Civic Impact Study Group. This blog post is based on the collaborative live-notes we took together on Thursday April 2nd.

Measuring civic impact is not a novel idea however with the growth of the use of data in storytelling we are seeing more attempts to measure the impact of data and the stories on civics. Do new methods of storytelling require new metrics for measuring impact? What of the impact on individual and how do we engage people with the data and the story?

Introduction

Guest and PhD student at MIT’s Center for Civic Media Erhardt Graeff started the session with a brief introduction into his work on creating media and technology to encourage effective citizens. He starts to talk about measuring impact amongst youth organizations and funders. Bringing an example from a workshop Erhardt organized in the UK on Youth Empowerment and Digital Literacy, in which he states that Funders are concerned about impact and so are the organizations that run civic media project, but neither are sure of how to measure impact. Most projects conducted by these organizations start with the assumption that youth are empowered by creating stories and media objects, however these organizations receive funding for literacy. As a result, the empowerment framing is lost. Funders can measure code learned, or jobs earned, but they can’t measure empowerment. So the questions is how do you measure empowerment?

This brings up two questions:

1. How do we set goals that appeal to this question about civic impact and what we are trying to get at, ie efficacy, social cohesion

2. How do we educate funders and others that this is a valid way to measure impact. How to measure self efficacy?

Erhardt is currently conducting a study with a major company around youth perceptions of race, specifically looking at the impact this company had. If people looks at the numbers, they aren’t amazing (10,000 tweets are hardly anything). But there’s a way in which how the conversation unfolded with people using their own stories. Erhardt and team read through the individual tweets and they realized that people started using the hashtag for other things after the campaign. This is telling a story of impact, but not simply through numbers. It impacted the way people are thinking -- this is particularly important if you are talking about impact of individual story. Does it become something people refer to online? How do we talk about people talking about what we've created and are doing?

Case Studies

Laura Amico (Boston Globe)

Laura comes from a traditional and entrepreneurial news background and is currently thinking of how stories at the Boston Globe do and don’t meet community needs. She is currently trying to start a conversation in the newsroom on how to change the stories to create more engagement from the audience. She asks: how do we start conversations around making data-driven stories in the newsroom? what do we need to ask our readers to get higher levels of engagement?

For her case study Laura presents a number of online stories while explaining the issues with their format and structure, calling them missed opportunities. On examples is the Thanksgiving story: a story ran by the Boston Globe on travel in Boston during Thanksgiving, Laura explains that the charts and text are not necessarily appealing to the audience.

[IMAGE]

Laura then shows a story from the Tsarnaev trial and wonders what the story is asking the reader to DO with that information? The article doesn’t ask the reader to engage with the Globe or the community. However, one successful example according to Laura is the article asking people whether they are eligible for jury duty in Tsarnaev jury. The reason to its success is that the article asks readers to engage in a specific way.  

Ultimately, the challenge is that reporters and editors don't tend to think about the story after it leaves their desk. What do we want from readers? Who do we want them to become?

The participants suggest that there are two issues that are of concern in this situation, it’s getting the newsroom to care and the people to care. It is best to meet people where they are and figure out what the newsroom wants from the people.

Pedro Reynolds-Cuéllar (MIT D-LAB)

Pedro’s presentation is a continuation of his prior case study with communities in Africa, where they produce technology and the community shares that technology to potentially become part of the marketplace. The lab’s research works on discovering bad products while working with the communities, then work on providing better products and informing the decisions the communities make with regards of the products.

Pedro’s example is of a project about solar lanterns. The lab ran a study of more than 20 solar lanterns, they looked at suitability, sustainability and scalability. After completing the research they release the results, now they are exploring what they would do with the information released. How to help communities and foundations connect to the data and how to build a compelling story through the data. Pedro want to also ensure that the communities are able to make the right decisions based on the research. Keeping in mind that the audiences are communities, multilateral organizations and manufacturers that need the information on scalability, sustainability and suitability.

One suggestion from the participants is the use of parody to explain what the wrong choice would be. Another suggestion was to look at the definition of impact and the issues that lie outside the project that might work against the goals the lab is trying to achieve. Co-design is also an important aspect that will help the community coming on board, having a shared project and a shared stay in the outcome can help.

Ben Fry (Fathom Information Design)

Ben runs a small design firm called Fathom that uses code to build stories.  All their projects are centered around how to explain and understand information. Ben’s case study focuses on the storytelling project done for the Clinton and Gates Foundation on the progress of women and girls worldwide over the past 20 years in 200 countries with a 1000 different indicators. The project can be found on http://noceilings.org/

Their project goal was to tell interesting and shareable stories through the data they had. They aimed to have stories that involved the audience, specifically millennial women and girls, in addition to a policy audience. The challenge was to work with sophisticated information and convert it to tweeting/ sharing material. Ben presents the project by showing  the “peel off” stories with a data board that has pinterest like stories. Have it be interactive and with video. At the bottom of the page there is a map for users to see the change happening over time. He also shows a host of exploratory data analysis that helps users find interesting stories about voting, marriage and political representation.

Ben noted that it is difficult to measure the attention given to the stories that have a meaningful and lasting impact. He then noted that: we collect data about the things that we care about. But there are times when there is no data and this is a problem, even around talking about the gaps of the data?

Ben’s questions to the group:

• How do you phrase the things to make the policy audience happy. Is it something straightforward - shareable cards with single stats with the ability to dig in deeper?

• The stories showed that women have unequal internet access. What is the likelihood of somebody sharing these stories?

Ben’s biggest challenge now is to get more traffic to the site. When asked a question about giving context to the data in a way that is not misleading, Ben responded with their attempt to layer the data in a way that would provide more in-depth context. There are different versions of stories based on size of story, etc. And there are appropriate levels of context based on it.

Dalia wraps up the session with some questions:

1. What are we trying to achieve?

2. How do we define impact?

3. How do we get people to engage with data?

She then introduced the breakout groups with themes and questions that emerged during the first session.

• What do we mean by successful campaign, how can we engage people around data?

• How can we build metrics around impact?

Breakout groups:

1. Engagement (Facilitated by Catherine) - What do we mean by a successful campaign? What is an engaging data story?

The group continues talking about Ben's project, focusing on:

• Relating the data to personal experience and creating sharable pieces?

• The project map

• Creating a query for the data and generating reports

The conversation revolved around explaining the project for a client’s perspective and making a case for the stories. Then moved on to audience engagement through ownership. For Ben, it’s about how to engage people's curiosity. Get them to start thinking about the data. Personal metrics - for people who didn't know, then they take that on. The Majority of people weren't thinking about child marriage - now they are. That's a win. What does that lead to? Participation leads to further initiatives.

Thinking of the data and stories, one question focused on how much is prescribed and how much of it is undefined. Whereas the other question focused on trying to answer what the data can do, which helps understand how if could help awareness through the data. Who can deal with child marriage? What can you do? Here's how you can act? What are the levers for change? For the rest of the world it’s great to know about it. Another suggestion is to have multichannel storytelling, lead traffic back to your website.

Ivan thinks of impact as a right measure for some kinds of things - policy change, etc. Impact is a wrong thing to talk about for everything. However, you never stop being part of a conversation in civics. We are setting a ground, there is an opportunity to participate. Making sure there are no biases. It is of concern when we import social science concepts into this space. Concerned to emphasize the quantitative.

Useful - how do conversations travel and bridge conversations and norms? Day to day analytics. A couple slices of data analysis of our community - map of 10000 people that link to or share global voices content.

Very useful - shows that what they do is bridging - global voices people played key role in translating content back and forth. To understand things across languages.  

Ben talks about the things he would like to do moving forward - identify misrepresentation and lack of representation in media context. Change the frame - that's the measure of success. They don't have the code or data team to make that happen. There is also the question of dichotomy of awareness raising vs. impact. Is there a dichotomy?

By impact - we can also mean AFFECT. People sometimes think simplistically about impact and awareness raising. Quantifying --- think differently vs ‘success’ meaning changing your mind. Another pathway to awareness - Would be educator guides,  engagement from school communities - you can’t just give them data and say “good luck”. You need accompanying materials - curriculas etc. People in software think that it's 90% building the tool and then people will come when the reality is the inverse.

2. Measurement (Facilitated by Erhardt) - What metrics and frameworks that can be created for measuring impact?

There is always something that comes before measurement and that is what your goals are. What do we want? What is the thing that we want to see in the world? How it is expressed?

Mariko on her work: One of the goals is to have a digital platform for data and for people to access it. Agencies are currently using the site in a closed-beta group. Agencies contributing to it also get access to it. But then the questions that arise are: How is it used? Who is using it? How is it being used for economic development? Who really cares or is it some niche?

Rahul: Open data has become a proxy for open thinking, but it’s useless unless you have engagement and open engagement. Somerville is doing data-driven decisions with none of it open. External and internal frameworks of thinking - how you actual change behavior within your organization.

Mariko: That’s what we’re calling data management. One big thing will be getting political cover, political buying from the governor. That’s the one big thing to do, but we’re scared, even though the governor self-identified as the open- data leader.

Erhardt: At the end of the day the decision maker will actually be the governor’s advisor. Anything that falls within an administration is an expression of the people within it. The more headway that you make with open data you will ask explicitly for someone who can adapt the system for one of those key policy pillars from the gov. One is you sit down and see how they do it. Secondly you see how they do it themselves.

Rahul: I’m a fan of setting short term and long term metrics for measuring impact. So setting up goals for the short term and long term to see what was achieved over time. Writing the long term outcomes is hard, but if you write those you can write the short term outcomes.  

Erhardt Community organizing aspect: there is a way to work with middle managers that you can’t work with anyone else. There is a sweet spot where they will listen to you if you understand their motivations. He spent two weeks in Kuwait recently talking to people who wanted to work with the government (that has a lot of bureaucracy). But there was one guy who had a lot of success with the pothole app. Kuwait is a place with zero apis, but this guy went to the middle manager that fill the potholes and told them “I will help you by automating the process of the requests and then will send the jobs done to the government for accountability”. Since Instagram is huge in Kuwait the guy used the platform - tagged managers and it presented a great portfolio piece for them. There is value in this because the manager has nothing to lose, everything to gain. Win win systems with a positive feedback loop, in this case its posting the pothole fixes on instagram.

Questions about measurement: Stories are most compelling aspects, not just as what we do but also as we measure what we do.

Rahul talked about data murals as a way to start the conversation, but then how can we figure out of people change the way they think about a certain issue based on these murals? One option is to conduct interviews with people who saw it. It is a lot of groundwork and learn what they are thinking about. Who are they referring to when they talk about the issues.