Feeds:
Posts
Comments

Archive for the ‘Program Evaluation’ Category

MP900313964

A retrospective of my journey into strategic performance management (my last five blog posts). There were two issues that I encountered in my exploration of strategic performance management.  One was terminology and the other was wondering where strategic planning may fit in.

Terminology There were some key terms that were commonly used: strategy, outcome, and objective. In some cases author(s) did not define key terms, which contributed to complexity in comparing comparing articles. For example, in one document the author refers to policy objectives whereas another author may refer to policy outcomes in a similar situation.

After looking back at my posts, I acknowledge that I failed to define terms in some cases. In case you desire to look at my previous posts, these are the definitions for key terms.

Activity — actions taken (e.g., training workshop, meeting)

Assumption — principles, beliefs, or ideas that are the basis for how and why the program will work

Desired Results — vision of the future, including short- and long-term outcomes

External Factors — conditions that may affect the program

Outcome — specific changes expected in individuals or the community based on the organization’s activities

Strategy — broad, coordinated approach to achieve an outcome or a goal

As you have read, I used the term outcome in the development of strategic performance management framework for my fictitious not-for-profit. I felt that outcome was the more appropriate concept in this context, that is, at the strategic level. I consider objective as a tactical or operational term, and I believe it’s more suited for use in strategic planning.

Strategic Plan? Where does strategic planning fit with strategic performance management? I suggest that the strategic plan follows developing the strategic performance management plan. The strategy and outcome map (step 2) provides the framework for the strategic plan.

The strategic plan identifies the time-specific goals and objectives to implement the strategies and perform the activities towards achieving the outcomes. Briefly, goals are statements related to outcomes and objectives are SMART intermediate steps toward achieving goals.

  • S pecific
  • M easurable
  • A ttainable
  • R elevant
  • T ime-framed

While I was looking for an image for this post, I was trying to find a way of describing the strategic performance management and strategic plan processes in everyday language. The architectural plan seemed the most appropriate. Architectural plans correspond to strategic performance management. Whereas, the strategic plan is the construction project management plan. How do you think they fit together?

This journey has been a brief excursion into strategic performance management, more or less to chart one possible route. It has given me one systematic method of linking the vision, mission, strategies and outcomes as well as identifying potential indicators to continually evaluate organizational performance. Development of the logic model articulates the assumptions and external factors that may need to be addressed when reviewing performance indicators. Development of key questions regarding outcomes leads to developing measures that provide information toward answering those questions, thus linking back to outcomes. Lots of charts. I wonder if I should try to put them all together. Would that be a tangled web or an interesting infographic?

Read Full Post »

I have finally reached the step of developing measures! Indicators are means of measuring progress and direction towards an outcome, not whether the outcome has been achieved. The indicators should also provide information towards answering key questions.

The Guide to Measuring Advocacy and Policy states,

These outcomes can provide concrete direction to your selection of What to measure that will allow you to make the case about success and will also deepen your understanding and learning about your progress.

Bernard Marr in How to Design Key Performance Indicators, states,

Analytics is most powerful when the analyst (or whoever is conducting the analysis) has both qualitative and quantitative metrics to work with. Most organizations have a preference for choosing quantitative metrics (those collected in numbers). This is not surprising as quantitative data is easier to collect and to translate into meaningful metrics. However, it is important to balance numeric data with ‘qualitative’ (non numeric) assessment of performance, as this can be a powerful way to highlight issues that are important to customers and stakeholders.

The last step in my journey — the indicators for my hypothetical not-for-profit.

metrics_page1

metrics_page2

Read Full Post »

compassHow will we measure success? Both Foster and Louis as well as Bernard Marr suggest developing key questions prior to identifying measures.

Each audience has a different perspective and will have different definitions of success.  For example, the board of directors would be interested in the outcomes relating to customer and financial perspectives. The staff would be interested in the outcomes relating to internal and learning and growth perspectives in addition to the customer and financial perspectives.

While evaluative questions help us to create better and more meaningful performance indicators, they do much more. Questions direct us to develop management information and performance data in a context, that is, related to the outcomes.

Bernard Marr provides guidance in preparing key performance questions in his Advanced Performance Institute Management White Paper, What are Key Performance Questions? He suggests that the questions should be open, that is, resulting in answers other than yes or no. Such questions would start with what, why, how. Open questions lead us to further analyze the results of the associated metric and link back to outcomes.

I have developed the following questions for my hypothetical not-for-profit.

questions

Tomorrow, the last step 4, identifying measures, metrics, indicators.

Read Full Post »

Links of ChainThe Guide to Measuring Advocacy and Policy indicates that the next step is to identify outcomes. Outcomes are the specific changes expected in individuals or the community based on the organization’s activities.

The selection of outcomes is a crucial step in the evaluation process. The outcome needs to be a direct reflection of the type of change that it is intended and realistic in the context of advocacy and policy work. The selection of outcomes relates to the theory of change and provides specificity regarding the success measures that are most relevant to a given effort.

The identification of outcomes is a critical step for the organization, even if you don’t follow the journey to determine measures or metrics. Identification of outcomes is clarification for all stakeholders of what is anticipated as a result of the organization’s execution of its mission. It helps to ensure focus.

These outcomes can provide concrete direction to your selection of What to measure that will allow you to make the case about success and will also deepen your understanding and learning about your progress.

I found the outcome categories in the Guide useful, so I used them in my outcome map. After outcomes, the next step is to specify the activities, which are generally based on the strategies, to achieve the outcomes.

I created the following outcome map for my hypothetical not-for-profit.

outcome_map

What these approaches do not recognize are the additional organizational elements needed to ensure that the entire organization is working towards common ends.  These additional elements are presented in the balanced scorecard. Management of the organization and the capability to engage in program activities require financial, internal process, and learning and growth perspectives.

  • Customer Perspective – adding value to the customer, client, member, community
  • Financial Perspective – increasing revenue and improving productivity
  • Internal Perspective – organization processes
  • Learning and Growth Perspective – employee skills, technology, and organizational climate

To include the four perspectives, Bernard Marr suggests starting with a strategy map. Marr points out in his Management White Paper, How to Design a Strategy Map,

The aim is to create a picture of the strategic objectives that captures everything that matters but at the same time keep it simple, compact and meaningful.

[…]

[O]ne of the great strengths of a Strategy Map is that it enables the visualization of causality — the value creation chain is clear and understandable.

I also looked at Ascendant Strategy Management Group’s guidance on strategy maps in The Mission Drive Strategy Map and Balanced Scorecard Strategy Maps Nonprofit and Charity Examples.

strategy mapBoth Marr and Ascendant Strategy Management Group advise writing objectives for each of the four perspectives of the strategy map. However, the definition of objective to which I’m accustomed doesn’t seem to match their application of objectives. Since I have most recently been involved in strategic planning, I’ve been operating with the definition of an objective as a measurable step taken to achieve a goal.

Ultimately these guides on strategy mapping are based on Kaplan and Norton’s strategy map, Having Trouble with You Strategy? Then Map It, published in the Harvard Business Review, September-October 2000. Kaplan and Norton advise to enter strategies in each perspective. The example in the Kaplan and Norton article seems to align with the definition of strategy that I’m familiar with, which is that a strategy is an approach to achieve a goal. Therefore, I entered strategies in the strategy map for my hypothetical nonprofit as follows.

strategy_map2

I feel that I need to combine the strategy and outcome maps. This results in a visualization of the vision, mission, assumptions, external factors, strategies and activities with the balanced scorecard perspectives, and short- and long-term outcomes based on community-organizing perspectives.

strategy_outcome

Tomorrow, step 3, ask questions.

Read Full Post »

Key in LockBefore defining success, management should begin with understanding the nature of the organization. In this case, let’s look at community organizing. Foster and Louis (Grassroots Action and Learning for Social Change: Evaluating Community Organizing) write,

Community organizing is a democratically-governed, values-driven process that catalyzes the power of individuals to work collectively to make the changes they want to see in their communities. Community organizers honor and develop the leadership potential in everyday people by helping them identify problems and solutions, and then by supporting them as they take action to make those solutions a reality. In so doing, organizing challenges the existing power structure.

Relationships lie at the heart of organizing, and the “one-to-one” relational conversation between an organizer and a community member is the building block of organizing. As those community members participate in social change work, build skills, and take on responsibilities, they become “leaders” within the organizing group. Developing these leaders and building the “base” of leaders and other community members is an ongoing focus of community organizing.

[…]

Like policy advocacy, community organizing efforts frequently require long timeframes to come to fruition. The process of growing and mobilizing a base, building power, shifting attitudes, and changing policies or practices does not happen overnight. Holding those in power accountable following change also takes time. The work of organizing, like advocacy, is complex and iterative. Strategies shift, often rapidly and multiple times, based on the external environment and other factors. Ultimate outcomes may not be readily quantifiable. The work is frequently collaborative, which brings with it benefits and challenges. And organizing campaigns, like advocacy campaigns, often spring from a sense of crisis and cause the participants to approach the work with a high level of emotional intensity.

I also found the Authentic Demand authentic demand honeycombHoneycomb model of interest and applicable to the hypothetical not-for-profit I’m planning to use in this journey.

I found this model in Sustaining Neighborhood Change (Annie E. Casey Foundation). Authentic Demand is the set of strategies for Making Connections, an initiative of the Casey Foundation.

In this framework,

Each element of the Authentic Demand honeycomb represents a different way for residents and other community members to engage in achieving results, sustaining progress, and building community capacity for change.

Foster and Louis, as well as A Guide to Measuring Advocacy and Policy (Annie E. Casey Foundation), advise to begin with the theory of change. The theory of change links strategies, outcomes, and goals with the vision, mission, and assumptions. The W. K. Kellogg Foundation Logic Model Development Guide and Theory of Change: A Practical Tool for Action, Results, and Learning (Annie E. Casey Foundation) provide processes for a visual representation of the theory of change.

I modified the W. K. Kellogg logic model for my theory of change for my hypothetical not-for-profit.

theory_of_change

Tomorrow, step 2, identify outcomes and activities.

Read Full Post »

In my next four blog posts I’ll experiment with strategic performance management for a hypothetical community organizing not-for-profit. This journey has been one that has taken me to several documents and sites over the last several weeks. I’ve found common ground and a few differences in the sequence of steps that are recommended.

I have mapped the steps that seem most logical to me based on my research and my prior experience in evaluation in the public sector. The steps are:

  1. Understand the nature of the organization and the theory of change
  2. Identify outcomes and activities
  3. Ask key questions
  4. Determine measures

First, what is strategic performance management?

According to Bernard Marr in Strategic Performance Management in Government and Public Sector Organizations,

This goes far beyond the narrow definition of Performance Management, which is often used simply to describe the act of performance measurement and reporting, or simply people management. Strategic Performance Management is about identifying what matters, measuring this and then managing it to improve the effectiveness, efficiency and overall performance of an organisation.

Strategic performance management endeavors to systematically link performance indicators and measures of success with mission, strategy, objectives, and outcomes. Metrics or indicators without context lack meaning. Strategic performance management involves prospective evaluation; that is, measuring progress towards goals and using the information improve the program and organization from the onset.

In Transactions – Transformations – Translations: Metrics that Matter for Building, Scaling and Funding Social Movements by Manual Pastor, Jennifer Ito and Rachel Rosner,

Metrics can help organizations articulate where they are going, what road they are taking, and what they expect to find along the way. They can help groups strike the right balance in the trade-offs above, allocating time and energy to be maximally effective. They can serve as a guide and tool for lifting up lessons for the field and for funders. But they should never be the tail wagging the dog; as one convening participant warned, it is the “mission that determines the path— not the metrics.

Catherine Crystal Foster and Justin Louis state in Grassroots Action and Learning for Social Change: Evaluating Community Organizing,

Defining and measuring success plays a critical role in evaluation, whether that means a policy win, the achievement of learning goals, or achieving growth in organizational capacity. Through our work in a variety of social change contexts and with a range of funders, it has become clear how important it is to surface assumptions about what success looks like from the perspective of an organizer, versus an advocate, versus a funder (and even among different types of funders).

As we have noted, success for organizers encompasses leadership development and positive shifts in power dynamics as well as desired policy change.

What does success look like? This is the critical question for a not-for-profit, for it is the functional definition of the mission — what are the results or the impact of the organization’s work? Because of the work we do, this is the difference that we are making in the community.

My next blog post will describe my journey through step 1.

Read Full Post »

MP900321197What’s a balanced logic scorecard model? It’s my proposition that you don’t have to adhere to one model or process to evaluate your organization or to develop metrics. The balanced scorecard and the program logic model are two rigorous processes that are suited to nonprofits. Perhaps a custom blend of the two may apply to your situation.

Balanced Scorecard

The balanced scorecard helps you to develop measures from four perspectives: customer, financial, internal (operations), and learning and growth (people and systems). The balanced scorecard was originally developed for the for-profit sector to measure strategic performance other than strictly financial performance. The balanced scorecard has been adapted to the nonprofit sector and can be used for strategic management to

  1. Translate vision and strategy
  2. Link strategic objectives and measures
  3. Align strategic initiatives
  4. Provide strategic feedback

Robert S. Kaplan in Strategic Performance Measurement and Management in Nonprofit Organizations (Management and Leadership, Spring 2001) describes how to adapt the balanced scorecard approach to nonprofits.  Kaplan writes,

The Balanced Scorecard has enabled the nonprofit organizations to bridge the gap between vague mission and strategy statements and day-to-day operational actions. It has facilitated a process by which an organization can achieve strategic focus, avoiding the pathology of attempting to be everything to everyone. The measurement system has shifted the organization’s focus from programs and initiatives to the outcomes the programs and initiatives are supposed to accomplish. It has helped organizations avoid the illusion that they have a strategy because they are managing a diverse and noncumulative set of programs and initiatives. It has enabled them to align initiatives, departments, and individuals to work in ways that reinforce each other so that dramatic performance improvements can be achieved. Used in this way, all organizational resources—the senior leadership team, technology resources, initiatives, change programs, financial resources, and human resources—become aligned to accomplishing organizational objectives.

Logic Model

There are several logic model formats, probably as many as there are nonprofits. They generally fall in one of three categories: program design, program implementation, and program evaluation. At a minimum, a basic logic model helps you to identify program activities, outputs, outcomes, and impacts. More complex models may include a needs statement, assumptions, strategies, strategic questions, and evaluation information.

The W.K. Kellogg Logic Model Development Guide defines the program logic model as

a picture of how your organization does its work – the theory and assumptions underlying the program. A program logic model links outcomes (both short- and long-term) with program activities/processes and the theoretical assumptions/principles of the program.

[…]

Basically, a logic model is a systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan, and the changes or results you hope to achieve.

Balanced Scorecard or Logic Model?

Of these two processes, which one is right for you? Perhaps both, with perhaps features from other models not discussed here. My ideal is a schema that provides a visual depiction of the overall strategic management of the organization. One that links and aligns the needs statement, assumptions, vision, mission, goals, strategies for program delivery and fundraising, strategic objectives, resources, outputs, outcomes, impacts, and metrics.

In searching for any documentation on the internet about both the balanced scorecard and logic model, I found one reference, Creating Program Logic Models: A Toolkit for State Flex Programs, which contains an interesting analysis of the balanced scorecard and program logic model and how they can complement each other.

[The] sequence of developing the strategy map [for the balanced scorecard] based on an already developed strategic plan assumes that the organization’s or program’s strategic plan and related strategies and activities are clearly aligned with the achievement of its rated goals and objectives. [T]his is not always the case, particularly when an organization has failed to make explicit the underlying program theory upon which its strategic plan and activities are based.

The creation of an organization- or program-specific logic model can help to address this problem and can serve as an important developmental step in the implementation of a balanced scorecard by clearly laying out the organization’s or program’s goals; mapping the theory, assumptions, activities and strategies necessary to achieve those goals; and identifying appropriate outcomes and indicators to map progress towards the achievement of those goals.

Read Full Post »

Older Posts »

%d bloggers like this: