The Usable Blog

Some Program Evaluation Reading and Listening

2010-01-11T08:27:00.000-08:00

Two things of note we've come across recently: A book called Developing a Learning Culture in Nonprofit Organizations by Stephen Gill and a story by Emily Hanford on the impacts of pre-school recently aired by American Radio Works. We'll be blogging about these two pieces in the future but wanted to give them each a plug right away.

How are they related? The American Radio Works piece in fascinating to the extent that it discusses the unintended outcomes of a social program- pre-k schools for disadvantaged kids. How often do we think about assessing unintended outcomes (both good and bad)? How often do we build evaluation designs that allow us to capture them? The Hanford piece shows the importance of being able to look beyond what we are looking for. The Gill book talks about how, by promoting a learning culture, organizations can facilitate this.

What do both books have to do with program evaluation? They both lead us to the conclusion that evaluators ought to move away from the business of evaluating programs (which they have certainly begun to do over the last decade) and move towards supporting organizations in their efforts to learn from their work.

Jott was Nice but Reqall is Nicer

2009-12-23T09:14:00.000-08:00

A couple of posts ago we introduced Jott. Jott is now owned by Nuance, which gave us Dragon Naturally Speaking, the speech recognition tool. Unfortunately, Jott has significantly increased its price so we've turned to Reqall which does much the same thing for less. Both Jott and Reqall are speech-to-text (or regular text-to-text) reminder systems. Reqall is available for the iPhone but I drink a different brand of kool-aide so I'll describe how it works for the BlackBerry.

You can send a to-do to Reqall from the Reqall for BlackBerry application as a text message, dictation, or via a phone call. Sending as a text message is pretty straightforward. To send by phone, select the 'Send By Phone' option, again, from the Reqall for BlackBerry application. Reqall dials and when the call connects it recognizes your phone number. You speak your to-do and, if you like, you give it a due date:

"Hello this is Reqall. Do you want to add, share or recall?"

"Add"

The system beeps

"Remind George about the Pensky file. Tomorrow, 2 PM."

Tomorrow at 2PM, you get a text message reading 'Remind George about the Pensky file.'

You can also choose to record the message on your BlackBerry rather as well (useful, if you're somewhere that doesn't have cell phone service). As soon as service is available, the application will connect and upload the audio for processing.

We use Reqall primarily as a way to capture to-do items as they come up. Reqall has a number of other features that are worth exploring. For example, if you smartphone has GPS capability you can associate particular locations with particular tasks. Let's say you make regular site visits as part of your job. You could create to-dos for eachsite and when the GPS sensed your proximity (you’d need to designate the site), Reqall would send you the appropriate reminders. You can also share to-dos. Instead of leaving yourself a to-do, you could send it directly to any of the contacts you've entered into Reqall.

In addition to getting SMS reminders of your to-dos you can also view them from your browser. That's what we tend to do around here. We use the BlackBerry to create them and the browser to manage them. As with many such applications, Reqall plays well others. You can use it with Evernote, Google Calendar and Outlook. In a mash-up contest however, Jott is the clear winner. Let us know what you think. You can find out more by viewing David Pogue's video tour which you'll find here.

We'll be posting about other tools in the coming weeks so stay tuned for more.

Internal Versus Enternal Evaluation

2009-12-21T19:22:00.000-08:00

Should your organization outsource evaluation? Again, we favor a blended solution as discussed in the last post. Consider working with a consultant to set up an evaluation system for your organization but perform routine evaluation work on your own. If you decide to conduct a more comprehensive evaluation such as the one described in our previous post, call in that consultant to manage the project. The Kellogg Foundation has a worthwhile discussion on the topic which you'll find here. You'll find an even better one from Bernice Taylor Associates here.

There are two caveats. The first is that while a simple internal evaluation is entirely doable, few organizations are usually able to actually allocate the staff time to get one done. In most cases therefore it may pay to outsource the work. Most organizations want to spend on time with their participants not entering survey data or running analyses using Excel.

That said, distinguishing internal versus external evaluations may not really be the right way to think about the question of ‘who does the evaluation’. A good evaluator will work closely with program staff to make an evaluation a success so in this way there are both internal and external staff on the evaluation team. We have a lot to say on this subject and take it very seriously. Evaluation utilization is often founded on a collaborative approach that at times blurs the lines between an internal versus external study. To our thinking it’s more a question of accountability… who is responsible for project deliverables and this is usually a question of budget and organizational capacity. You may want to take a look at a white paper we have on this subject. You’ll find it here.

How often should a program be evaluated?

2009-12-08T19:59:00.000-08:00

We heard from an organization recently that wanted to know how often they should conduct an evaluation of their program. Of course it depends on a number of factors including budget but most importantly, what kinds of questions they have about their program. The group runs a number of summer camps around the country for inner city youth, an area we've worked in. They serve about 1500, mostly pre-teen kids. Here's what we said:

In terms of how often a program should be evaluated I would make the following recommendation. Once a solid evaluation design has been developed-- including instrumentation (surveys, interview guides, etc), and a data collection and analysis plan-- a program like yours might consider undertaking an annual evaluation. If the same survey can be repurposed each year, and the survey can be done online, the cost of such an approach is relatively low and will yield information useful to program planers and those within your organization who need to beat the bushes for funding. Going through the process every year will help you focus on results and ensure that basic information about program participants is complied and archived for future use (such as longer term follow-up with participants). Just as important this annual pulse-taking will help you generate ideas for less frequent but more in-depth evaluation work. Depending on budgets and the kinds of questions that arise in the routine annual evaluations, you may decide to do a more comprehensive study of your program say every five years or perhaps at an interval that corresponds to your strategic planning cycle.

Maybe a simpler way of putting this is the following, do an annual assessment to ensure that the program is staying on track-- that the staff and participants are happy and benefiting, at least in the short term, from the services they receive. Then do a more wide ranging evaluation when you have meaningful questions about how the program is working and about the kinds of longer term outcomes you are having. Think of the second evaluation as more strategic in nature.

What do you think?

We Love Jott

2008-07-22T09:21:00.000-07:00

Too many gadgets, too little time. Nonetheless we spend a good deal of time checking out new productivity tools that have zero learning curve, zero (or next to zero) setup curve and that are actually useful. Jott gets the nomination on all three counts. Their slogan "Talk to Jott, get simple back" is for real. So what is Jott. Jott is something for your cell phone.

You're walking down the street and you remember that you need to call one of your managers to discuss the Pensky grant. You're sitting in traffic, hear some interesting new music on the radio and want to check out the artist when you get back home. You're laying on the beach and suddenly remember that you realize you've forgotten to reschedule an important meeting. You call Jott and a friendly female voice says voice says:

"Who do you want to Jott?"
You say "Myself."
Jott says "Jott self."
You say "Remember to call Janice about the Pensky grant."
Jott says "Got it. Want a reminder?"

Jott recognizes your speech and converts it into text which it: 1) Emails to you, and 2) places in your online Jott to-do list. If you requested a reminder, Jott asks you the day and time and then sends a text message to your phone at the appointed time.

It really works, is really useful, and, as of this writing is dollar free and ad free. There are more features, for example you can send Jotts to others (either email to text messages) or to groups and you can categorize your Jotts as well. For example, you could create a folder for "The Pensky Project" (who knows maybe George Castanza could have kept his job if he'd had Jott) and then tell Jott to put a particular to-do in that folder for future reference.

You can get Jott here. It is probably the best productivity tool I've found all year. If you try it, let us know what you think.

Informed Consent

2007-11-06T05:44:00.000-08:00

We just discovered a cure to the informed consent blues. You'll find it here. It's an informed consent builder from the University of Wisconsin. University culture, and the fact that such forms are usually developed with lots of input from the legal department, means that the form that gets created my be overkill but it certainly does cover all the bases and its easy to modify the form in a word processing program to suit your project's particular needs. Rather than struggle with wording a consent, we edited one the system created based on answers to a few simple questions. Give it a try if you need to obtain informed consent.

Play to Win: The Nonprofit Guide to Competitive Strategy

2007-02-28T09:02:00.000-08:00

The title says it all in David La Piana and Michaela Hayes' Play to Win: The Nonprofit Guide to Competitive Strategy. Whether they recognize it or not, these authors write, nonprofit organizations compete with one another for clients, staff, funds and media attention despite the collaborative and often anti-competitive ethos that pervades nonprofit culture. The sooner they recognize and address this dimension of their work the better off they and the communities they serve will be.

The book begins with background on why nonprofits often fail to embrace the competitive dimensions of their work and instead choose to focus attention on forming collaborative relationships. The authors roll out the usual suspects here including an overall orientation towards inclusiveness and sharing and an antithesis to values normally associated with the marketplace whose spillover 'bads'― inequality, poverty, lack of opportunity― they seek to mitigate. La Piana and Hayes point out as well that collaboration is often something foisted on nonprofit organizations by the foundations and governmental entities that fund them.

Want to read the full review? The folks over at Nonprofit Central are now hosting our reviews on their site. Point your browser here to view it.

Logic Model Training

2006-07-10T10:22:00.000-07:00

We've just created an interactive training module on logic models. It distills everything you need to know to create a logic model into one 15 minute audio-visual program. You can find it here. Please take a look at it and let us know what you think.

Foundationspeak... A book review

2006-06-25T09:07:00.000-07:00

If you've had just about all the foundation speak you can handle, point your browser to three PDFs available here. The first, When words Fail: How the public interest becomes neither public nor interesting is actually the last in the series and we think the best. We particularly enjoyed author Tony Proscio's comparison of the prose created by right leaning versus left leaning foundations. The former was straight and to the point, the later obfuscating. To make this point, Proscio extracts some from a document created by a conservative think tank and does a jargon audit. In place of buzz words like 'teacher work-force', 'career advancement structures', 'competencies' and 'human capital' he finds straight forward language such as 'hoops and hurdles', 'tests', and 'get rid of'. With this analysis, it's easy to see why the right is winning the war of words.

Proscio goes on to offer a theory for why left leaning discourse is so obtuse. It's centered around the idea that as the foundation community's ideals about altruism, sacrifice and the common good loose force in a culture dominated by the materialist ideology of the marketplace, left leaning organizations retreat and come to develop a culture of isolation complete with a secret and inbred language all their own.

We disagree. In our view it's quite the opposite. Foundationspeak is what it is because foundations seek to align their language with their primary reference group, academics, policy makers and other experts who require a (seemingly) value neutral language in order to sound non-partisan, dispassionate and ultimately, scientific. As Proscio himself points out, much foundationspeak parrots the latest language of business school-- 'metrics', 'value-proposition', etc. Straight forward talk about beliefs and values are nowhere to be found.
This theoretical disagreement aside, there is much in this essay to value. We particularly liked his deconstruction of some leading foundation buzzwords and his presentation of the pro-jargon position-- yes there is such a thing and it is more compelling than you may think.

Take a look. Whatever position in the jargon wars you take, it's nice to know the arguments on both sides.

By the way, you can find a complete dictionary of foundation jargon, based largely on this essay buy clicking here.

More Logic Models

2006-05-15T10:04:00.000-07:00

We've blogged a bit about logic models and talked about how they can help non-profits formulate evaluation questions. But we see some problems with using them as well. The first is that non-profits often get bogged down in lengthy discussions about whether something is an output or a short term outcome or a long term outcome or whatever. We don't view this sort of thing as a good use of time. For us, as evaluators, logic models are heuristic devices. That is to say, they are tools for generating questions and hypotheses about whether a program is doing what it has said it will do, and if it is, whether it is producing results. Often, a list of these results, short term, medium term and long term, is all one really needs. When we work with a client to help formulate evaluation questions, we usually start this way and don't spend a great deal of time on working up a full blown logic model.

The other issue we see with logic models relates to the fact that they usually neglect the evaluation work an organization already does. Take a look at the Kellogg Foundation's Logic Model Guide or the United Way's Outcome Measurement Resource Network. Both view logic models as the first step in the process of creating an evaluation strategy. But as a practical matter, few organizations in fact start evaluating within such a grand scheme. Instead they start in little ways, by taking attendance at training classes, collecting demographic and referral information from clients, tracking community satisfaction with their work, or discussing case notes at weekly staff meetings. An approach to evaluation that doesn't foreground what an organization already does is doomed to step on toes, re-invent the wheel, and duplicate effort. Yet there is no place for this in the logic modeling processes we've observed. In these approaches one simply specifies outcomes, develops indicators for these outcomes, and collects data on these indicators in order to learn if, and under what conditions, they are present.

We advocate greater practicality. Generate evaluation questions using a logic model, but figure out how you can leverage existing data collection systems to answer them. If your organization is like most, you won't have to start your evaluation effort from scratch if you follow this approach. Determine out what you collect now, what you can easily collect in the future, and what you'll want to plan for down the road, and drive your plan based on what you can realistically accomplish.

Logic Models

2006-04-19T04:33:00.000-07:00

We did an evaluation training two weeks ago in New York at the Support Center for Non-Profit Management. Like many introduction to program evaluation trainings, this one included a section on logic models. Logic models are basically graphical representations of how a program uses resources --> to create activities --> which have tangible results --> that lead to desired outcomes. If you like flowcharts with lots of arrows you'll like logic models, if you like Microsoft Visio, you'll love them. Here are some places to find examples: The Two Professors, UW Extension Service (extension services seem to love them), or our favorite The Kellogg Foundation Logic Model Guide.

Logic models have lots of uses. Funders like to see them and many require them. They're great for getting a program's stakeholders to sit down and specify their goals and how the work being done everyday will bring about those goals. Underlying any good logic model is a theory about how the world works-- for example, a theory that acquiring knowledge about how to write a resume will lead students to write better resumes and then to get better jobs. Logic models are also useful in designing an evaluation.

Why? Because logic models force those who design them to specify what a program's activities are and what outcomes-- short term, medium term and long term-- will come about once those activities take place. Here's an example: offer resume workshop and job search networking coaching to 20 recently unemployed workers--> workers learn resume writing and networking skills --> workers send out improved resumes --> workers make 10 networking contacts in first two weeks --> workers obtain interviews --> workers get jobs.

It shouldn't be too hard to see how even such a simple logic model like this one could help an evaluator development an assessment plan. Here are some of the questions: How many resume writing workshops took place? How many people attended? Did they take away the required knowledge? How many changed their resumes? How effective were the new resumes (did they implement the knowledge correctly)? How many used the coaching service? Did they view the session positively? Did they understand what was said? Were they able to apply it? How many interviews did they get? How many of the interviews were appropriate? How many started new jobs within a given time frame? The list goes on.

Logic models are useful in evaluation because they require programs to make very specific lists of exactly what they are going to do and what they expect to see after they do it. This is crucial information for an evaluation since it specifies exactly what needs to be measured. Logic models get everyone on the same page and tell evaluators, in concrete terms, what they need to look for. That's what makes them so valuable. So it shouldn't be surprising that nearly every "Intro to Evaluation" training we've ever observed has devoted a lot of time to logic models.

But we believe that their strengths have led evaluators to adopt logic modeling uncritically. Check back in a couple of days to see what we have to say on the subject.

Gifts of the Muse

2006-04-13T10:24:00.000-07:00

Gifts of the Muse: Reframing the debate about the benefits of the arts

Gifts of the Muse argues that beyond the instrumental benefits of the arts—on individual cognitive development, on attitudes or behaviors that improve school performance, on physical health, on community economic development— people are drawn to the arts for their intrinsic benefits, the sense of meaningfulness, pleasure and satisfaction they provide. The four authors, all associated with the Rand Corporation, make the case that the research which attempts to document instrumental affects is often weak and inclusive. They call for widening the discussion of the value of public support for the arts so that it includes intrinsic value considerations and offer a language and a rationale with which to begin that work.

While there is much in this volume that will be interest to arts organizations seeking public (and private) support, any organization which offers a program that provides intrinsic benefits should take a look. Here's an example.

We did an evaluation some time ago for an organization that teaches chess to children in the public schools. The program wanted to be able to document that chess helped kids concentrate, improved their spatial reasoning skills, and their ability to persist at solving problems. Despite a strong evaluation design, many thousands of dollars later all we could document were very modest effects. Does this mean chess should be eliminated. Of course not. Teaching chess to kids is an intrinsically valuable thing to do. It makes a small, if not entirely measurable, impact on many kids as the study showed. And it will probably make a significant impact on at least a few (say those who go on to play at the tournament level), but that's not the point. Exposure to chess gives kids a chance to try something new, something other than reading, math and, increasingly, test prep. It gives them the chance to try something different, exercise new intellectual muscles, and connect with a pastime that is centuries old.

There are two issues to think about here. The first relates to finding ways to persuade funders that a program's worth isn't always reducible to its instrumental benefits. As we said before, Gifts of the Muse provides a language for doing that. The second relates to figuring out how measure its intrinsic benefits. That's usually not easy and sometimes impossible to do. We'll work on developing some suggested strategies and post them as they percolate up.

Why this?

2006-02-27T16:07:00.000-08:00

Why would anyone want to read about program evaluation? To own it, is the best answer we can come up with. Program evaluation is an inherently political activity. It competes for and uses resources, it articulates a program's goals, it defines criteria for success, and it can be used to justify continuing a program or eliminating it. As a practical matter, few programs are eliminated based the results of a program evaluation, mostly, they are changed, so we don't believe that most of the politics resides there. In our view, the political nature of evaluation is manifest most when it articulates goals and defines what success will look like. Yes, everyone in a well run organization knows what its goals are-- but often only in a very general and vague sense. Evaluation forces a program's stakeholders to sit down and specify the connections between what they want, what they do, and what they actually see. To be sure, we've evaluated programs that make a point of not having clearly articulated goals. But even then, the process of creating instrumentation, collecting data, and then searching through it for interpretations and meanings filters some things in and some things out.

So this blog is going to about these kinds of choices.