We undertook research for and research of policy. With the LMPG Group of the Department of Labour we developed a case-study based research methodology that provided rich data and insight for analysts and industry/firms/individuals.
We also designed, tested and refined a developmental work research (DWR) methodology that enables policy and regulatory agencies to engage with the people, organisations or industries of interest to them in a way that enables the two parties to co-design the policy or regulatory artefact or action. This research-based innovative instrument is termed the Co-design Laboratory.
We also undertook evaluations of policy and policy or operational programmes. We evaluated the effectiveness of agency engagement and activity.
- Research for policy and operations
- Evaluation of policy and programmes
- Connect policy and practice
- Case-study based methods for policy and operations
Research for public policy development and evaluation
NB: Changes in the Public Sector and shift to cloud-based servers meant that many links to research reports are broken. We are hunting out the links using the Wayback Machine.
Employment Contracts Act 2000. Case Study Research in the Accommodation, Wine Making and Brewing Industries (1999). Evaluation of the impacts of the Employment Contracts Act 2000 carried out with the Labour Market Policy Group of the Department of Labour (DoL), and Richard Bird. http://www.dol.govt.nz/PDFs/op1999-4.pdf
Accident Insurance Act, 1998 (2000-2002). The ACC case study research project (1999-2001). A three-year evaluation, led by DoL, which examined how firms responded to the 2000 Injury and Accident Prevention Act, and the reasons for that response. Carried out with Bob Williams.
Health and Safety in Employment Act, 1992 (2001-2002). The Department of Labour (OSH) commissioned WEB Research to work with OSH and the Ministry of Economic Development to conduct qualitative research on the topic, involving 21 case studies of manufacturers. With Richard Bird. A copy of the report, The Costs and Benefits of Complying with the HSE Act 1992 2001-2002, can be found at:
Social costs of occupational injury and illness (2000-2001).
Aftermath: How the Lives of Fifteen Ordinary New Zealanders Have Been Affected by Workplace Injury and Illness 2000-2001. WEB Research was involved in this OSH-led research project investigating the social and economic consequences of workplace injury and illness. The case histories featured in the report can be found at: http://www.dol.govt.nz/PDFs/Aftermath.pdf
Employment Relations Act, 2000 (2002). WEB Research participated in an LMPG-led Evaluation of the Short-Term Impacts of the Employment Relations Act 2000. WEB Research undertook 20 of the 51 cases collaboratively with DOL. The full report of the lead study, The Effects of Employment Regulation case study research (1999-2000), can be found at: http://www.dol.govt.nz/PDFs/ERA%20Evaluation.pdf
Precarious Employment (2002-2004). Exploratory Case Study Research into Precarious Employment 2002-2004. LMPG commissioned WEB Research to collaborate on qualitative research that aimed to better understand the practices, behaviours and values of employees in non-standard forms of work. The 17 case studies focused on individuals working in large firms in the cleaning, financial/call center, fish processing and labour hire/construction industries. The full report can be found at: http://www.dol.govt.nz/pdfs/precarious-employment.pdf
The Pure Business Project (2003-2006). A highly experimental approach designed to transform relations between government and business by addressing their habitual, but contradictory, urges to regulate and to resist regulation, respectively. A Cross-Departmental Research Pool (CDRP) project, funded through the Ministry for Research, Science and Technology (MoRST) that has been very successful and drawn international attention to the possibility of cross-departmental co-ordination and co-operation with business. See Hill, R., Capper, P., Whatman, R. Wilson, K, & Wong, K. (2007). Workplace learning in the New Zealand apple industry network: a new co-design method for government ‘practice making’. Journal of Workplace Learning. 19(6), 359-376. http://www.dol.govt.nz/initiatives/workplace/sme/index.asp
Economic Restructuring and Skills Formation (ERSF) – FRST. A case based study of the impact of economic restructuring on firms, people and learning needs. See: Hill, R., Bullard, T., Capper, P., Hawes, K., & Wilson, K. (1997). Learning about learning organisations: Case studies of skill formation in five New Zealand organisations. Learning Organization 5(4), 184-192. Learning Organisation article.doc
Learning and expertise in teams and networks (LETN) – FRST. The companion study to the ERSF study above. In this case-based study, WEB Research worked with firms to create practical responses, tools and strategies for use in workplaces to enhance the capacity of the firm and staff for learning, innovation and technology uptake – and to increase productivity. For an example, see: R. Hill, P. Capper, K. Hawes and K. Wilson (1998). Using Activity Theory and Developmental Work Research as Tools to Analyse Contradictions in Information and Operational Systems: a Case Study of DHL. LETN – Wollongong chapter.DOC
Research into the design and effectiveness of public funding schemes
Performance Based Research Fund – Tertiary Education Commisison (TEC)
WEB Research (2004) Phase 1 Evaluation of the implementation of the PBRF and the conduct of the 2003 Quality Evaluation Wellington. WEB designed and undertook a substantial evaluation of the initial impact of the Performance Based Research Fund and the conduct of the 2003 Quality Evaluation across the tertiary sector. The evaluation required high levels of analysis, reporting and sensitivity. A copy of this report is held at http://www.tec.govt.nz/downloads/a2z_publications/pbrf_report_overview.html
Marsden Fund – Ministry of Research, Science and Technology (MoRST) WEB Research was commissioned to carry out an evaluation of the Marsden Fund (2004). A project undertaken with Technopolis Ltd, a leading UK-based consultancy specialising in the design, operation and evaluation of science and technology funding by governments and EU agencies. The report can be found at: http://www.morst.govt.nz/Documents/publications/evaluations/Marsden-Fund-Evaluation-Report.pdf
Policy development and research & evaluation for the public sector
WEB Research uses case study-based evaluations to help government agencies bridge policy and practice. The approach is a form of developmental evaluation that allows the agency to understand the systemic links between:
- Its policies and processes (for example, labour market policy); and
- The activities and practices of those who are affected by these policies and processes (for example, the behaviour of firms, managers and employees).
WEB and our research associates developed and trialled the case study-based approach with the Department of Labour (DoL) in a series of projects carried out between 1996-2002. The initial impetus for DoL to undertake this form of evaluation stemmed from their need to develop ‘evidence-aware’ policy in understanding firms, workplaces and the impact of departmental policy and regulation. DoL wanted to augment its standard research practices with a more bottom-up, practice-oriented approach. Case study-based evaluations allowed the department to investigate what was actually happening in workplaces and allowed for that information to flow back to policy in a meaningful way. Our evaluations are developmental for both policy groups and firms and workplaces.
In the past 13 years, WEB Research has undertaken a substantial number of policy evaluation projects in New Zealand to inform or evaluate public policy and its operational effectiveness. This work includes the Phase One Process Evaluation of the PBRF for the Tertiary Education Commission and the Evaluation of the Marsden Fund (in partnership with Technopolis) for the Ministry of Research, Science and Technology. Other clients include the Ministry of Education, the Department of Labour, the Ministry of Social Development, the New Zealand Qualifications Authority and Inland Revenue.
NGO: Developmental evaluation
What is it?
- Developmental evaluation is a facilitated, high involvement process to help organisations and groups get robust evidence of the effectiveness and value of their policy, programme, or funding initiative.
- Research tools are used to document and share lessons learned, using a user-friendly approach that has high ‘face validity’ (Davidson, 2007).
- Our approach is designed to build an ongoing evaluation capability within those organisations that use our services.
Where did it come from?
- WEB Research developed the method by drawing on:
- Insights and tools from our research experience on how adults learn and change their practices at work; The work of international colleagues, particularly at the University of Helsinki in Finland (Engestrom, 1996);
- Our case study-based approach to evaluation developed jointly with the Department of Labour between 1996-2002 (see Sankar and Hill, 2002);
- Our work as researchers with Charitable Trusts and NGOs in New Zealand; and
- The expertise of evaluation experts such as New Zealander Jane Davidson (2007).
- Integral to the method is a close partnership between the organisation commissioning the evaluation, the research agency (WEB), and those whose projects and activities are being evaluated.
Who uses it?
NZ Charitable Trusts and NGOs in funding relationships
- Developmental Evaluation enables a Charitable Trust or NGO to develop a partnership relationship with funded service providers in a short, medium or long-term relationship. The method allows the funder to:
- Evaluate the process and outcomes of the service provided;
- Reflect upon and re-design (with the active participation of the service provider) monitoring and reporting tools, and processes for engagement between the funder and provider; and
- Change the behaviour or services of each other to better secure the agreed outcomes.
- Traditionally funders let contracts to organisations that carry out research or intervention programmes for people in a particular community sector. In these cases, the connection between the funder and the people it seeks to assist is mediated by the contracting organisation.
- This arrangement no longer suits some funders. They want to create the possibility of all parties – the commissioning and commissioned organisations and the people they intend to benefit – collectively monitoring, reviewing and re-designing interventions on a rolling basis.
- This is a sophisticated approach and requires a medium to long-term high trust relationship between the parties. A pure contracting model is not possible; the precise form, tools and behaviours of the new relationship have to be created by the parties as they undertake their respective roles and tasks.
Evaluation research with charitable trusts and NGOs
A five-year evaluation project for the Wayne Francis Charitable Trust (2007-2012). Carried out in collaboration with Dr Jane Higgins. The project evaluates how well the Trust’s process of funding projects works, and how funded organisations view and experience the Trust. The work also includes outcome evaluations of funded organisations.
An Evaluation of the Salvation Army Employment Plus (2008). This was an evaluation of that part of the Salvation Army that operates as a Private Training Establishment.