Welcome to Geo Education::The mission of Geo Education is to apply knowledge in service to the Earth and its people in evaluation, education & environmental work with community development and empowerment methods

United Nations Evaluation and Performance Measurement Methods for Results-Based Budgeting and Management

Bill Leon has been a consultant for the United Nations on reform of the cycle of planning, budgeting, monitoring and evaluation of UN programs as part of its transition to results-based budgeting and management (RBM). Now, every two years, when each program requests funding, it must also specify outputs, specific performance results it will achieve, and the indicators and performance measures it will use to determine the extent to which it has achieved its results or outcomes. This was a significant reform with the potential to be confusing and contentious for many staff and programs. It also had potential repercussions for many programs within the UN and around the world. We were able to ease the transition and to accommodate significant multi-national and multi-cultural perspectives and perspectives from different parts of the UN bureaucracy. Recently, Bill helped all UN Programmes review their expected accomplishments and indicators for relevance and evaluation practicality.
In 2007, Bill led an effort to interview UN managers, staff and representatives of Member States on the implementation of RBM. The report outlined concerns and suggestions for improving the way the UN uses data to measure its results. Bill then designed and led a workshop in Geneva for 63 representatives of 31 UN organizations to identify best practices in the use of RBM and ways they are best applied in the contexts that UN organizations work. The workshop report identified numerous ways that agencies are working to use meaningful data to guide their work and report on their results, and the community building efforts during the workshop led to the establishment of an ongoing network for sharing ideas, techniques and materials that all can use in their ongoing reform efforts.
In 2002, with Jane Reisman of Organizational Research Services, Bill worked with the staff of the UN Budget Office to operationalize the requirements in ways that helped staff in the various UN departments understand and comply with the new directives. We also developed a training manual in how to collect data to measure outcome-based performance measures. Bill trained UN staff from almost all departments over seven weeks at the UN’s New York headquarters. He explained the new processes, trained them in the basics of program evaluation, showed them how to develop viable indicators, and explained how to measure them using standard evaluation techniques. We also developed a separate training for key Budget staff and worked closely with then in a ‘train-the-trainer” model to empower the organization to maintain and use the expertise in-house and to share the training with colleagues around the world.

International Criminal Court Evaluation System Design and Reporting Assistance

In 2007 and 2008, Bill and colleagues have been helping the Outreach Unit of the International Criminal Court develop a formal evaluation process. The work involves helping staff in The Hague and in four African countries develop a descriptive theory of change and logic model to identify key outcomes; developing an evaluation plan with specific indicators methods; developing data collection tools and protocols; and designing and implementing a web-based database to track relevant information. Geo Education & Research also helped the Outreach Unit analyze and report on its 2007 accomplishments in a written report and multi-media DVD.

National Safe Place

Geo Education & Research is designing another comprehensive evaluation system for National Safe Place, a national nonprofit that supports youth homeless shelters in over 140 U.S. cities. Through collaborative work with local shelter staff, national advisory and governance board members and partners like the National Runaway and Homeless Youth Switchboard, we are developing appropriate ways to measure outcomes for shelters, their staff, business site partners, communities and youth. The work will integrate local data into a national database, making it easier to report results and analyze trends across sites and regions to better understand the needs and impacts of programs serving our most vulnerable youth.

Healthy Child Care Washington Evaluation for the Washington State Department

Bill worked as a principle project manager on this multi-faceted evaluation of programs to improve the quality of care among center-based and family child care providers across the State. The work involved research on best practices in child care and in promoting improvements; developing consensus among various staff levels and consultants on criteria and methods; developing data collection, management, analysis and reporting tools; training staff in local health jurisdictions; providing ongoing technical assistance; tracking outputs and outcomes; and reporting results. The work has allowed the project to report its significant impacts clearly and effectively and has allowed it to improve and better integrate its work by tracking outcomes for child care health consultants, child care providers, parents and children.

Washington Dental Service Foundation

Geo Education & Research completed a needs assessment for the Washington State Dental Service Foundation (funded by the largest dental insurer in Washington) focused on the needs of seniors in licensed care facilities. The work entailed surveys of and interviews with nursing homes throughout the state, dentists, dental hygienists, and for profit and nonprofit service providers in other states. A separate analysis of the acute needs of tribal members on reservations throughout the state was also performed. Based on our research, we developed pro-forma financial statements to describe different approaches to meeting needs through such methods as increased screenings and preventative care, mobile clinics or improved insurance programs.

Washington States’ Children’s Administration Outcome-Based Contracting

Bill worked with the State of Washington’s Children’s Administration (CA) to develop outcomes for their contracts with providers of services to children and families in crisis or great need and who are facing potential out-of-home placement. The work was necessary to clarify and measure various types of positive change that the providers are attempting to facilitate in people experience a great deal of stress from a wide variety of causes. The work entails soliciting input and active participation of CA staff, a statewide network of service providers, and other experts in the field of child abuse, neglect, and treatment. Bill assisted with the management of this cooperative process as well as the creation of the proposed outcomes that, when achieved, will help CA and its contractors facilitate more substantial success in helping children and families throughout Washington.

Shorecrest High School Small Class Size Evaluation

Bill Leon led a team of junior evaluators in a multi-modal evaluation of an effort by this school in Shoreline, Washington to increase personalization, engagement and student performance with small class sizes. It reduced the student teacher ratio for freshmen and sophomore English classes from 31:1 to 20:1. We investigated experiences, changes in attitudes and behaviors, changes in teaching behavior, and objective measures of performance to verify that the educational strategy had consistent and significant, positive impacts on teaching and learning. Data collected and reviewed came from grades, test scores, other school data, surveys, interviews and focus groups. This is the first comprehensive analysis to verify the educational benefits of smaller class sizes in U.S. public high schools.

Reinvesting in Youth (RIY) & Elements of Successful Programs (ESP)

Bill Leon has co-directed efforts to evaluate a major systems change effort in Seattle, Washington and surrounding King County. The change initiative focused on ways to reform the juvenile justice system to place greater reliance on early intervention and treatment as opposed to incarceration and then use potential savings to invest in prevention programs that can effectively prevent other youth from committing crimes and entering the judicial system. In this process, the team interviewed all key stakeholders; worked with other evaluation consultants investigating different, specific interventions; and distilled lessons learned that will assist with ongoing progress and inform others who might want to replicate the process.

One major component of RIY was the ESP program in which we identified (through a meta analysis of over 400 studies of juvenile intervention programs) 24 elements of success commonly found in programs that have been shown to be effective in reducing juvenile crime and violence. For each element we defined specific indicators that can be used to determine if the element exists in a program or organization. We then created and implemented a Program Assessment and Implementation Planning Process with 15 agencies. In it we helped them use the Guidebook to Elements of Successful Programs to take detailed looks at one of their programs to see how it compared to the “best practices” identified nationally. From this process-type evaluation, we helped them develop action plans to address deficiencies identified and then provided some of the technical assistance they needed to improve. Finally, we re-assessed their progress following the TA process. A separate, outside evaluator evaluated this process and found it to be very beneficial and well-run.

Copies of the Guidebook to Elements of Successful Programs and the assessment process tool can be found on our website (www.geoeducation.org).

Evaluation of Communities in Schools Programs in Alaska

Bill Leon worked with several sites around Alaska to evaluate the implementation of different programs to help at-risk youth enhance basic learning, learn job skills, serve their communities and receive other assistance to help them make a smoother transition into adulthood. He helped all of these programs develop appropriate evaluation methods for their interventions and taught staff the basic evaluation skills they needed to keep the evaluation processes active. A final summary report documented the various sites’ achievements in their programs and in their implementations of evaluation training.

Wilburforce Foundation Yellowstone to Yukon Program Evaluation

Bill Leon supported Wilburforce in an evaluation of its Yellowstone to Yukon (Y2Y) Program. The Y2Y program evaluation measured outcomes and impacts from different points of view of a Wilburforce initiative to engage scientists and environmental nonprofit organizations in research and advocacy collaborations in order to preserve wilderness to enhance habitat connectivity in priority ecological niches. Prior to the interviews and surveys, Bill helped them develop a theory of change and logic model to guide the evaluation and future program planning.

The Brainerd Foundation Strategic Plan Evaluation

Bill and his colleagues worked with the Brainerd Foundation to evaluate its last five-year strategic plan. After developing logic models for its operations, its communication and capacity building activities, and its environmental protection efforts, Bill and others developed three data collection and analysis strategies. Through reviews of all modest to major grants, they catalogued the grantees achievements in different priority realms. A web-based survey of all of their grantees measured the Foundation’s success from the points of view of their grantees, and collected suggestions for improving grantee relations and effectiveness. Interviews with all staff, selected grantees, trustees, advisors, consultants, other local environmental foundations, and unfunded applicants gathered detailed insights and suggestions in priority issue areas, and they produced a long list of creative ideas for the Foundation to explore. The results were presented to the trustees and staff, and with it the Foundation revised its strategic plan in order to enable it to enhance its effective grantmaking.

Zoo and Aquarium Teen Programs Assessment Consortium (ZATPAC)

Bill Leon has been one of the principal facilitators for the development of a generic and widely applicable rationale, strategy, and set of tools for evaluating a wide variety of teen programs commonly found at zoos and aquariums across the U.S. and in other countries. He have helped six institutions (The Brookfield Zoo, The Monterey Bay Aquarium, The New England Aquarium, The Philadelphia Zoo, The San Francisco Zoo, and The Woodland Park Zoo) develop 18 logic models that describe their different programs. Analyzing these documents and facilitating a workshop with representatives of the programs, he helped the team develop a generic theory of change model to illustrate their logical processes and guide their evaluation efforts. He has worked with the consortium on the development of survey and observational instruments to collect data on priority outcomes from youth participants. The first set of pilot data has shown some modest program impacts and serves as the starting point for a more expansive effort to evaluate program impacts.