Pre-conference workshops
30 August to 31 August 2010
A series of high-quality workshops were held on the Monday and Tuesday preceding the conference. Workshops focused on developing evaluation knowledge, techniques and practices and are aimed a range of experience levels.
Registration
Costs
| AES members | Non-members | |
|---|---|---|
| Full-day workshops | $410 | $553 |
| Half-day workshops | $220 | $363 |
Prices are in New Zealand dollars and include GST
Registration included workshop materials, morning/afternoon tea and lunch.
Workshop information
Summary of workshops
| Date | Time | Topic | Presenters |
|---|---|---|---|
| Monday 30 August | 8.30am to 4.30pm | Monitoring, Evaluation, Reporting and Improvement (MERI) | Jess Dart, Clear Horizon |
| Monday 30 August | 8.30am to 4.30pm | Introduction to Realist Evaluation Design | Gillian Westhorp, Community Matters |
| Monday 30 August | 8.30am to 4.30pm | Linking Evaluation and Social Justice | Donna Mertens, Gallaudet University |
| Monday 30 August | 8.30am to 4.30pm | Introducing Developmental Evaluation | Kate McKegg, The Knowledge Institute Ltd; Nan Wehipeihana |
| Monday 30 August | 8.30am to 12.30pm | Using Evaluation for Policy: Rapid Evidence Assessments (REAs) - FULL | Rowena Cave, Ministry of Justice |
| Monday 30 August | 1.30pm to 5.30pm | Using Programme Logic in Multi-Level Multi-Site Programme Evaluations - FULL | John Owen, Centre for Program Evaluation |
| Tuesday 31 August | 8.30am to 4.30pm | Developing Monitoring and Evaluation Frameworks | Anne Markiewicz, Anne Markiewicz and Associates, and Dr Ian Patrick |
| Tuesday 31 August | 8.30am to 4.30pm | Evaluation design and implementation - FULL | Elliot Stern |
| Tuesday 31 August | 8.30am to 4.30pm |
Actionable answers for real-world decision makers: Real evaluation nuts and bolts that deliver - FULL |
Jane Davidson, Real Evaluation Ltd. |
| Tuesday 31 August | 8.30am to 4.30pm | The power of the visual image: Using photo based data collection techniques to enhance participation and dissemination | Rosalind Hurworth, Pamela St Leger and Diane McDonald from The Centre for Program Evaluation, The University of Melbourne |
| Tuesday 31 August | 8.30am to 12.30pm | An Introduction to the Most Significant Change - uses and applications of the technique | Tracey Delaney, Consultant |
| Tuesday 31 August | 8.30am to 12.30pm | Jerome Winston, CeDRE | |
| Tuesday 31 August | 1.30pm to 5.30pm | Thomas Schwandt, University of Illinois | |
| Tuesday 31 August | 1.30pm to 5.30pm | Designing improved performance measurement systems | Graham Smith, Numerical Advantage Pty. Ltd. |
Workshop details
Monday 30 August
8.30am to 4.30pm
Monitoring, Evaluation, Reporting and Improvement (MERI)
Jess Dart, Clear Horizon
This workshop will outline a practical approach to embed flexible monitoring, evaluation and reporting and improvement (MERI) processes into your programme. This learning-based approach to monitoring and evaluation is designed to assist programme teams to collect meaningful data and assess whether they are on track to achieve their intended results, and if not, to guide them towards programme improvement.
MERI offers a solid evaluation planning process using programme logic mapping and some innovative tools to help you capture expected and unexpected outcomes. This approach aims to foster continuous learning and adaptation throughout the programme cycle and will provide a structure for you to tell the story of your programme. MERI is an adaptive management tool used extensively across a wide range of sectors.
By the end of the workshop participants will:
- Understand the steps used to create a MERI plan
- Understand the role of programme logic in MERI
- Understand how this logic can be used to structure a comprehensive monitoring, evaluation, reporting and learning framework plan for their project or programme.
The workshop will use a mix of small group work, hands on exercises and plenary discussion. Participants will create a draft monitoring and evaluation plan for a case study project. This workshop is pitched at the beginner to intermediate level, and is aimed at those looking for practical, learning-based monitoring and evaluation techniques.
Monday 30 August
8.30am to 4.30pm
Introduction to Realist Evaluation Design
Gillian Westhorp, Community Matters
Realist approaches to evaluation are relatively well established in some sectors and have provoked significant interest internationally. However, realist approaches are still new to some evaluators. This workshop will introduce evaluators to its distinctive features and explore the implications of a realist approach for evaluation design. The workshop will:
- Explain basic concepts of realist philosophy
- Introduce the implications of that philosophical position for understanding ‘what programmes are’ and ‘how programmes work’
- Explore the implications of those understandings for the role and tasks of evaluation and evaluators
- Introduce a framework for evaluation design that builds on those understandings.
The workshop will include presentation, whole group exercises working through a particular example, and small group exercises where participants can consider their own evaluation project. A reference list and some reading materials will be provided.
This workshop is intended for new evaluators and graduate students, or for more experienced evaluators who are new to realist approaches. It will also be of value to policy and programme staff who commission evaluations and want a working understanding of a realist approach for that purpose. There are no pre-requisites for attendance.
Gillian Westhorp is Director of a small consultancy company specialising in realist evaluation. Over the past eight years, she has completed a number of realist evaluation projects in education, health and community services. Gillian undertook her PhD in realist evaluation methodology with Professor Nick Tilley, co-author of the original text, Realistic Evaluation.
Monday 30 August
8.30am to 4.30pm
Linking Evaluation and Social Justice
Donna Mertens, Gallaudet University
This workshop explores the basic assumptions that encompass human rights as a driving force to define the evaluator’s world view and how it impacts decisions in the evaluation process. The transformative paradigm will provide the philosophical framework for determining contextual considerations in designing and conducting transformative mixed methods evaluation.
The workshop is geared to meet the needs of evaluators who have responsibility for evaluation in communities that reflect diversity in terms of culture, race/ethnicity, religion, language, gender, and disability. Participants will be able to deepen their understanding of basic concepts and principles of evaluation by extending these to include dimensions of cultural diversity and transformative mixed methods approaches.
Specific examples will be used to demonstrate how a transformative mixed methods approach enhances the evaluator’s ability to accurately represent how deficit perspectives that are taken as common wisdom can have a deleterious effect on both the design of a programme and the outcomes of that programme. Alternative strategies based on transformative mixed methods are illustrated through reference to the presenter’s own work, the work of others, and the challenges that participants bring to the workshop.
A formal presentation integrated with large group discussion will address the implications of the transformative paradigm for the use of mixed methods directly focused on the furtherance of social justice. Following a mini lecture, small group brainstorming with subsequent large group sharing will address the following questions:
- What are the knowledge, skills and attitudes required for an evaluator who works from a transformative stance?
- How does one acquire such skills?
- What approaches are useful for an evaluator to undertake a transformative mixed methods evaluation in diverse contexts?
Specific case study based, interactive activities will give participants an opportunity to apply the theoretical guidance provided in the first part of the workshop to methodological questions, such as:
- How can transformative mixed methods be applied to increase the probability of social justice goals being achieved?
- What sampling strategies are appropriate?
- What does it mean to address the myth of homogeneity?
- What data collection strategies are appropriate for diverse groups in transformative mixed methods evaluation?
- What are the implications for the evaluator’s role in terms of the use of the findings for transformative purposes?
The workshop will end with an interactive session in which participants can discuss the application of transformative mixed methods strategies to their own evaluation work.
This is an intermediate course designed for participants with existing knowledge and experience with case study approaches.
Donna M. Mertens, Professor, Department of Educational Foundations and Research at Gallaudet University in Washington, DC, teaches research methods and programmeme evaluation to graduate-level deaf and hearing students in multiple programmemes. The major focus of her work is the blending of issues of social justice and human rights with research and evaluation frameworks and methods.
Monday 30 August
8.30am to 4.30pm
Introducing Developmental Evaluation
Kate McKegg, The Knowledge Institute Ltd; Nan Wehipeihana
In a time when solutions to the world’s most pressing social problems are neither predictable nor known, developmental evaluation provides a way to apply evaluative thinking and practice to the complex realities of today’s organizations and communities.
This workshop will introduce evaluation practitioners to developmental evaluation with a focus on its practical application in the real world. The workshop will be interactive and organized around case examples drawn from the presenters’ experience in a range of contexts including policy, programmeme and community settings.
The workshop will cover:
- The fundamentals of developmental evaluation – some theory and practical strategies
- When to use developmental evaluation, and when not to
- The benefits and challenges of developmental evaluation
Participants will take away detailed workshop notes, templates and examples as well as a CD pack with a range of the latest literature on developmental evaluation. This workshop will be useful for evaluation practitioners, funders and commissioners of evaluation, particularly those working in innovative and complex situations and contexts.
Kate McKegg works as an independent evaluator and has nearly 20 years experience in policy and programmeme evaluation, evaluation capacity building, research, training and facilitation in government, non-government and community contexts. She is an experienced trainer, providing evaluation courses and workshops in a wide range of settings.
Nan Wehipeihana has more than 15 years experience in designing, leading and managing evaluation and research projects. She has a strong track record in policy and programmeme evaluation, using mixed methods in evaluation, organisational capacity and capability development and extensive evaluation training experience. The development of culturally-based evaluation outcome frameworks is a particular area of expertise.
Monday 30 August
8.30am to 12.30pm
Using Evaluation for Policy: Rapid Evidence Assessments (REAs) - This workshop is now FULL
Rowena Cave, Ministry of Justice
This workshop will show how Rapid Evidence Assessments (REAs) identify and synthesise ‘best evidence’ for policy, and how they can inform evaluation practice.
Government agencies internationally are increasingly moving towards more evidence-based approaches to policy development. An REA is a systematic review with particularly tight limits, thereby making it ‘rapid’. It is a useful tool when policy needs to be developed quickly, but nonetheless must have a robust evidence base. The aims of the workshop will be to:
- Increase the awareness of those working in evaluation about what systematic review methods (and policy-makers) require of evaluation research design
- Provide some guidance about how to report evaluations so that they are useful to policy-makers
- Provide an introduction to developing an REA.
This workshop will be suitable for people working in evaluation and/or other research; and also for policy analysts and advisors working with government agencies and non-government organisations, particularly those involved in strategic policy or service delivery.
The content of the workshop will include:
- An overview of the development of REA methodology
- An interactive ‘how to’ session, clarifying the guiding principles
- Examples of methods and tools, how they are applied and how to choose the best for your purpose
- Explanation of roles required.
By the end of the workshop, the participants will have learned about REAs in social science, how they are developed and when they are useful. In addition, they should have a good understanding of:
- Evaluation from the perspective of policy-makers
- How to structure evaluations so that they receive attention from policy-makers
- Where they might look for resources and further training
- Standardised methods of critical appraisal
- Ways in which they can apply this information to improve their own evaluation work.
The workshop will be conducted in a mixed format, introduced with a PowerPoint presentation. This will be followed by a discussion or question-and-answer session, combined with a chance to read some existing REAs produced in the broader social sector. This will lead into a hands-on session where participants will work through some of the process involved in REA development for themselves. This will give them the opportunity to try out some of the tools and have any resulting questions answered.
Monday 30 August
1.30pm to 5.30pm
Using Programme Logic in Multi-Level Multi-Site Programme Evaluations - This workshop is now FULL
John Owen, Centre for Program Evaluation
Early studies of the impact of interventions (policies and programmes) tended to treat these interventions as ‘black boxes’. The assumption was that the ‘treatment’ was a given. This assumption is sometimes valid, but in the majority of evaluative situations involving social interventions, it is inadequate. For example, in impact evaluations of mature programmes we often wish to know what aspects of programme implementation affects outcomes.
Also, in programmes that are not mature, evaluative work needs to examine implementation characteristics as a means of programme improvement. Evaluative work based on programme logic (theory-driven evaluation) provides a way of undertaking evaluations that address issues such as these. Over the past 20 years theory-driven evaluation has become a major development in thinking about the practice of evaluation.
This workshop is devoted to application of programme logic to the planning and conduct of evaluations. In particular, the workshop will consider this application to multi-level multi-site interventions. These are interventions that involve the development of policy in response to a perceived or real need across a social system. Delivery of the response also requires delivery at multiple sites across the system. Many interventions developed and delivered by government and non-government agencies can be classified as multi-level multi-site. There is a challenge for programme logic to capture complexities associated with this type of intervention. While there are well-developed suggestions for the use of programme logic as a basis for evaluating individual programmes, there are far fewer guidelines that can be applied to more complex interventions.
This workshop is designed to contribute to participant understanding of the issues outlined above, and will introduce some techniques for using programme logic in these situations. The workshop will be based on adult learning principles. This will involve participants in activities based on a case-study. The workshop will also include discussions designed to allow participants to transfer their understandings to their own contexts.
John Owen has interests in the design of evaluations that meet the information needs of policy and programme decision makers. He is also committed to the education and training of evaluators, and was instrumental in the development of a graduate evaluation programme at Melbourne University, where he was Director of the Centre for Programme Evaluation from 1992-2002. He is the author of a major evaluation text Programme Evaluation: Forms and Approaches.
Tuesday 31 August
8.30am to 4.30pm
Developing Monitoring and Evaluation Frameworks
Anne Markiewicz, Anne markiewicz and associates, and Dr Ian Patrick
The development of monitoring and evaluation frameworks that integrate the two functions in a planned fashion have become an increasingly prevalent requirement of the programme design and development process.
This workshop aims to equip participants with an understanding of the process of developing monitoring and evaluation frameworks. The workshop will present participants with a conceptual model for development of an monitoring and evaluation framework, discuss design and implementation issues and consider any barriers or impediments with strategies for addressing these.
The workshop will work progressively through a document titled “Guide to Developing Monitoring and Evaluation Frameworks” developed by the trainer. In this process participants will develop knowledge of the format and approach required for the development of an monitoring and evaluation framework, the techniques and skills involved in the design and implementation of the framework, and develop an appreciation of the parameters of the tasks involved and how to approach them. Participants can take this guide and apply it as a prototype in their workplaces or in their evaluation practices.
The workshop is pitched at beginner to intermediate level.
Tuesday 31 August
8.30am to 4.30pm
Evaluation design and implementation - This workshop is now FULL
Elliot Stern
Evaluations need to be designed and planned. This not only involved the choice of methods but also clarity about evaluation questions; the requisite skills that must be employed; and the capacities of client systems to use evaluation results and processes in their own context. This workshop will explore these issues through a mix of presentations; exercises; and question and answer sessions. It is intended for experienced, practicing evaluators or commissioners of evaluation.
Elliot Stern is Professor of Evaluation Research at Lancaster University and the current editor of Evaluation: the international journal of theory, research and practice.
Tuesday 31 August
8.30am to 4.30pm - This workshop is now FULL
Actionable answers for real-world decision makers: Real evaluation nuts and bolts that deliver
Jane Davidson, Real Evaluation Ltd.
Ever read an evaluation report and been none the wiser about whether the whole programme was a waste of money or not? What if evaluations actually asked evaluative questions and gave clear, direct, evaluative answers? This hands-on workshop covers the most important ‘nuts and bolts’ concepts and tools for delivering actionable answers:
- Big picture utilisation-focused thinking about key stakeholders, their information needs, and the evaluative questions they need answered
- A macro-level framework for evaluation design
- A reporting structure that gets to the point like no other
- A powerful, practical, meaningful evaluation tool for getting real, genuine, direct, evaluative, concise answers to important questions – the mixed method evaluative rubric.
Jane Davidson was formerly the Associate Director of the Evaluation Center at Western Michigan University and now heads an evaluation consulting firm in Auckland, New Zealand.
Tuesday 31 August
8.30am to 4.30pm
The power of the visual image: Using photo based data collection techniques to enhance participation and dissemination
Rosalind Hurworth, Pamela St Leger and Diane McDonald from The Centre for Program Evaluation, The University of Melbourne
Research shows that photo-interviewing in its various forms can be a particularly powerful tool for the researcher. Furthermore the photo-voice can be used as one component of multi-methods triangulation to check research credibility. While some potential issues have been raised in association with photo-interviewing; such as difficulties concerning appropriate photo selection; failure to present negative images; photos only portraying partial reality; the potential to be confrontational; and issues about recruitment, these seem to be outweighed by the positive attributes of the technique.
Efforts to capture the photo-voice can be particularly useful when working with certain stakeholders: helping to build empathy and trust; ensuring that the views of those in less influential positions are heard; and even revealing some unexpected findings. The visual image is an influential and rigorous technique that can be included in either a qualitative or quantitative evaluation design.
Evaluators, particularly those who want to incorporate participatory, interactive and empowerment approaches would do well to add photographic and other visual methods to their ‘toolkits’.
Specific objectives of the workshop are to:
- Introduce participants to various types of visual techniques that can be used in evaluation
- Consider specific evaluation contexts in which visual techniques can be most effective
- Explore how a visual component can enhance evaluation findings and reporting.
Rosalind Hurworth is the Director of the Centre for Programme Evaluation, Melbourne Graduate School of Education at the University of Melbourne where she teaches on-line and face-to-face within the Master of Assessment and Evaluation. She has particular expertise in qualitative methods, including the use of the visual medium, and published 'Teaching Qualitative Research' in 2008.
Pamela St Leger is a senior lecturer in the Centre for Programme Evaluation, Melbourne Graduate School of Education at The University of Melbourne. She has taught courses in programme evaluation and supervised postgraduate students who are undertaking research in aspects of evaluation. She is currently involved in an ARC research project that has used photo interviewing with young people.
Diane McDonald is a Programme Evaluator at the Centre for Programme Evaluation, Melbourne Graduate School of Education at The University of Melbourne. She has extensive experience in the management of a wide range of qualitative evaluation studies, including in cross cultural and other challenging contexts. Her recent work has involved the use of participatory visual techniques to evaluate an education project in remote villages in Lao PDR.
Tuesday 31 August
8.30am to 12.30pm
An Introduction to the Most Significant Change - uses and applications of the technique
Tracey Delaney, Consultant
Most Significant Chang (MSC) is a powerful tool for monitoring, evaluation and organisational learning. In many monitoring and evaluation systems, the indicators or things we measure are defined by people distant from where the events happen; they are often set by senior executive staff or specialist monitoring and evaluation units and as such are defined by looking out from the programme. MSC on the other hand gives those closest to the events (e.g. the field staff and beneficiaries) the right to identify a variety of stories that they think are relevant. With MSC, the type of data collected is therefore potentially far more dynamic and can include unexpected outcomes.
Allowing beneficiaries to choose what to report also means that MSC stories can reflect real changes in the world as well as changing views about what is important within a particular target group. Moreover, MSC goes beyond merely capturing and documenting stories of impact, to offering a means of engaging stakeholders in effective and meaningful dialogue. Each story represents the storyteller’s interpretation of impact, which is then reviewed and discussed. The process offers an opportunity for a diverse range of stakeholders to enter into a dialogue about the programme intention, impact and ultimately future direction.
The workshop provides a basic overview of MSC and will include group discussions, examples from the field and experiential exercises during which participants will be invited to interpret some stories of significant change. The different uses of MSC for both evaluation studies and on-going monitoring will be explained, together with the strengths and weaknesses of the approach.
This workshop is suitable for beginners as well as for those more experienced in monitoring and evaluation. Anyone involved in monitoring and evaluation who is interested in how stories can be used to report on outcomes, and particularly the learning and improvement aspect of monitoring and evaluation would benefit from learning more about this innovative technique.
Tracey has extensive experience in program design, monitoring, evaluation. In particular, Tracey has an extensive knowledge of the ‘Most Significant Change technique' (MSC) and currently moderates the MSC egroup. Her experience with MSC has included designing MSC systems, assisting with the implementation of MSC in a range of settings and providing training and mentoring on the technique. Tracey is a skilled facilitator and trainer with strong communication skills.
Tuesday 31 August
8.30am to 12.30pm
(Re)Discovering How Applied Systems Theory Improves Evaluation Planning and Performance Reporting when Working with Multi-Level and Multi-Programme Interventions
Jerome Winston, CeDRE - This workshop is now FULL
This workshop will suit all participants and does not require previous knowledge of systems theory or programme evaluation. It will be of particular interest to participants who have been given multiple, and often conflicting, advice about how to use performance indictors and performance measures.
The terms 'systems theory' and 'systems thinking' are used to describe many different methodologies which draw upon a wide range of fields, including operations research, project management, systems analysis and systems dynamics. These methodologies are used to describe both ‘soft’ and ‘hard’ systems. The 'jargon' or terminology that was originally associated with applied systems theory (AST) in the 1950s has been adopted and revised by a wide range of professionals, each of whom brought their own expertise in particular aspects of programme planning, monitoring, evaluation, budgeting, audit, financial management and performance reporting.
These professionals developed their own personal use of systems concepts in line with their own preferred approaches to planning, monitoring and evaluation. Not surprisingly, there is now a proliferation of definitions and expert advice about how to apply systems terminology to programme planning, monitoring evaluation and performance reporting. Multiple and conflicting advice is readily available, as demonstrated throughout the workshop.
During the workshop, participants will engage in exercises and discussions that develop insights into how AST can be used in different contexts (eg, agriculture, education, defence, international development, financial management, health, disability services, etc) to improve programme planning, monitoring, evaluation and performance reporting. Workshop examples will be based on field studies of the use of AST in Australia, Malaysia and a number of countries in Africa. Attention will be given to how systems models are used to develop ‘integrated, result-based management’ systems that are intended to link programme planning, budgeting, monitoring, evaluation and performance reporting, within and across departments, ministries and sectors.
Exercises and examples will highlight pitfalls that distort performance reporting, including the misuse of aggregated data (the ‘ecologic fallacy’) and the way that particular definitions of performance indicators and measures can encourage inappropriate goal displacement and data gaming. Special attention is given to the need to distinguish clearly between programme effectiveness and (allocative) efficiency and the programme’s operational effectiveness and efficiency.
Jerome Winston started teaching and learning to apply needs analysis and systems theory to evaluation in 1973. He has been leading evaluation workshops and seminars since 1975. In 1979, he developed an innovative graduate programme in public sector monitoring and evaluation in Melbourne. His particular interests are the planning and management of monitoring and evaluation, use of systems theory and data analysis in monitoring and evaluation, evaluation of programmes by and for indigenous peoples, and the use and misuse of performance measurement systems.
Tuesday 31 August
1.30pm to 5.30pm - This workshop is now FULL
Evaluative Reasoning, Evidence & Use
Thomas Schwandt, University of Illinois
This workshop focuses on evaluation as a cognitive activity. It will explore the requirements of evaluative arguments; the nature of evidence and the role that empirical evidence plays in reaching an evaluative judgment about the value (merit, worth, significance) of a policy or programme; and, issues related to making sense of the concept of 'using' evaluative knowledge in decision making.
Audiences for this workshop include experienced evaluators who wish to sharpen their understanding of evaluation as critical thinking, as well as those just starting up.
Thomas A. Schwandt is Professor and Chair, Department of Educational Psychology, University of Illinois at Urbana-Champaign, USA and Editor of the American Journal of Evaluation. His scholarship is focused on the theory of evaluation with particular interests in evaluative reasoning and the use of evidence in forming evaluative judgments.
Tuesday 31 August
1.30pm to 5.30pm
Designing improved performance measurement systems
Graham Smith, Numerical Advantage Pty. Ltd.
The purpose of the workshop is to enable participants to either design a new set of performance measures for an organisation or programme, or to improve the current set. The workshop will focus on discussion and development of systems for performance measurement. This recognises that even though each organisation or project will have different requirements for performance measurement, a systems focus makes it easier for participants to consider jointly common factors in developing performance measures that are tailored for their environment. Nevertheless, the workshop will also include practical examples, most from the author's experience, of what works and what doesn't in performance measurement.
The specific objectives of the workshop are to:
- Understand the difference between establishing a system of performance measures and developing individual performance measures
- Discuss the criteria for good performance measurement systems; to discuss the criteria for good performance measures
- Work in groups to develop performance measurement systems appropriate to a particular case study (which can be provided by the group)
The workshop will consist of:
- An initial assessment of the group's level of prior knowledge and agreement on the objectives for the session
- An introduction that is in lecture format with interaction and questions from the participants; small group work on developing performance management systems
- Discussion and review of each group's results; conclusion, including interim results where possible on the performance measurement issues brought to the group by participants; and evaluation of the workshop
An intermediate level of knowledge will be assumed. Some knowledge of performance measures will be of assistance, and if participants bring specific performance measurement issues with them for discussion it will aid the relevance and focus of the workshop.