These webinars form the two-part series on our Performance Refresh. The AWP Performance System Refresh is central to a number of significant reforms AWP is currently undertaking to ensure that it remains as relevant and as effective as possible.
Why is AWP undertaking a Performance Refresh?
AWP’s system for managing and measuring performance and linking this to coherent and convincing performance narrative has been identified as a weakness during the mid-term review. Consequently, AWP has been leading on a process to overhaul the AWP Theory of Change, the AWP Operations System, and the AWP M&E System – a process collectively referred to as the AWP Performance System Refresh. In undertaking this Refresh, AWP aims to achieve greater coherence between AWP’s purpose and its performance systems, delivering improved evidence-based reporting of AWP development impact, and a more realistic and proportionate approach to reporting on each AWP-funded activity. Ultimately, this will allow AWP to demonstrate our development and diplomatic impact and help to tell the story of our unique contribution and model.
How will this affect our current activity reporting?
AWP will be introducing a revised set of reporting templates for the Performance Refresh which will include new Progress and Completion Report templates for current and future activities.
What is the level of disaggregation when you measure indicators? For example, at the adopt level, will this be aggregated to country, nature of organisation?
It is very much a case-by-case basis. We want to understand more about the beneficiaries of the program. This can be applied to existing portfolio activities, as well as future programming. Disaggregation will be applied to the way you are currently measuring or monitoring performance.
How are we capturing results of the trials at Tier B activities?
The system can be applied to existing portfolio and reporting currently underway on activities. The focus is already on what is being trialled and new approaches in-country. With each activity we need to ask why is this change important, what success there has been or changes that need to be made?
We want to avoid activities in the future where the definition of tools, policy or practice is not explicit. Going forward, each activity would need to clearly identify the benefit of the activity.
For any new approach we need to understand why it is important or how it is different from what is already going on, and outline what the key benefits are to the new approach. We’re interested in understanding how communities will benefit directly from the new approach.
As several of the result statements rely on qualitative evidence, what is the framework to assess sufficiency of evidence from qualitative data?
Here we need to differentiate between Tier B and C:
We are looking at what the baseline is for the communities who might benefit from the reform, how we measure the baseline and how to assess whether the intended benefits have occurred. This will be a more complex exercise for the Tier C activities that involves both qualitative and quantitative data
Along with GEDSI, it will be important to explicitly state AWP projects employ transdisciplinary approaches for greater impact of AWP investment.
For Tier C activities in the future there will be an expectation we’ll need to undertake this approach – to ensure the activities are effective.
It is similar for Tier B but we need to be realistic and pragmatic as to what approaches to consider as the activities are generally smaller.
Please define what we mean by 'climate resilient water management' in the context of AWP work.
Resilient indicates a system can cope with disturbances and adapt to changing conditions. In regards to water management being climate resilient, we need to approach interventions that are resilient to climate change as most mitigation activities use water. We need to use systems thinking about what water will do to this management activity, taking a preventative approach.
I wonder if this approach looks a bit like we have the knowledge around climate resilient policies and it is a case of transferring it - rather than being a needs based approach - I know AWP believes strongly in responding to need using some of the lessons from Australia but not sure it is obvious in this framework?
This approach does not change the basic intent, which is responding to in-country demand. We are a demand-driven organisation – this is crucial. There needs to be a collective, joined-up effort where both partners are working together to achieve a common goal.
AWP is not moving away from a demand-led agenda. We are trying to encourage further thought about the complexity of change. The simple exchange of technical knowledge is not enough for change to occur in the countries we are working in.
If we are responding to demand then in the framework the monitoring clearly starts with the identified need, which should then be assessed against achieving a positive response to that demand – isn’t that a key measure?
There will be a number of ways in which AWP performance is assessed beyond the Theory of Change, but the key message is that this is a deliberate attempt to focus our effort on in-country outcomes – on changes within the countries that we’re working on.
While the classification of activities to tiers is useful, we seem not to be encouraging a sufficient level of ambition. Tier A for me seems to sound like acquiring knowledge for knowledge's sake. How will they be prodded to move to Tier B? Is there an intention to do that nudge from one tier to the other, acknowledging that Tier C is what we want in the end?
One of the motivations for this change is that AWP had a one-size-fits-all approach regardless of the size of the activity. We are moving towards an ambition of supporting countries to undertake sustainable change, this will require a lot more complexity, analysis and willingness to look on AWP activities as being more like traditional aid projects. Tier A projects do have a modest ambition of supporting partners acquire new knowledge and skills on water management but they may also be the means by which new partnerships which lays the foundation that leads to Tier B activities. Likewise, Tier B activities can lay the foundations for future Tier C activities.
There are thousands of different elements to climate resilient water management and each activity addresses one or more of these elements. The identification and selection of which element is being addressed is a joint activity between AWP and the partner country. The activity only proceeds if the host country identifies the need and AWP agrees that it is something we have experience in and agree it is important - then by defining and measuring against this need underpins the monitoring - this is subject of course to the need to advance diplomacy.
It’s all about demand. We’re in no position to force a climate-resilience agenda onto partners where it is not yet recognised. We intend to have conversations with partners around this issue going forward
There’s a recognition that some of our activities will explicitly integrate a more climate-resilience approach than others – not all will have this on their agenda.
For DFAT, there is an expectation that we try to consider climate resilience across everything we do and report on it – but we’ll need to get this balance right.
Maybe we need to state explicitly in Tier A, there are two type of activities - (i) specific assistance for diplomatic reasons etc and (ii) activity based on the theory of change. This will avoid any confusion in the mission of AWP.
We are of the view that they shouldn’t be separated. Activities we fund are all based on the theory of change and are driven by development outcomes. Then for each activity there will be different levels of diplomatic interest based on the activity. We are not seeking to fund activities that only have a diplomatic agenda.
We 'need to be more explicit about the core agenda driving this theory of change'. Is this about delivering the theory of change framework, or more about a way of working? If the audience of the framework is the people delivering the AWP projects, that takes their way of working beyond the project management delivery to a more structured guide to the way of working that will deliver these qualitative/meta outcomes. Is this the intent?
Yes
People often say if you can't measure you can't manage. This can be misinterpreted and drive people towards quantitative outputs, which could mean losing sight of the outcomes you are seeking.
The point we are emphasising here is that we need to be better at measuring in-country outcomes for both management and reporting.
I don’t think the framework explicitly acknowledges the critical nature of defining and measuring against the defined need.
It’s very important to define and measure what we do. Connected to this theory of change will be an operational approach – a set of templates – which we’ll cover in the next webinar.
The Tier C indicator mentioned about policies adopted "benefitting people equitably”. The report for 2021 only reports on the adoption but not on the second part of the indicator which is about equitable benefit. How do you go about measuring this?
Just a note that the performance report for 2021 was a trial itself, we haven’t formally published it.
This raises a valid point – for the Tier C activities we identified, data on the extent to which the communities have benefitted from the activity is not available – this was because it was not part of the activity design.
We would like to emphasise that this is looking towards how we manage future activities. We need to establish a clear baseline of who is meant to benefit from these activities that we are supporting countries to adopt, and how we would measure that change.
Adoption can occur at very different scales – some are at the regional level (Mekong), others are occurring at the country level, some at the provincial level or the organisation level. Scale is very different and needs to be understood when we consider beneficiaries of this change.
I’m interested conceptually on how we are measuring equitable benefit, and how we track the pathway from policy adoption to equitable benefit.
AWP is aiming to support activities which aren’t just involved in the trialling and adoption, but will also support countries in the implementation of these policies at scale. That support won’t be just technical in nature but also assist in measuring change, so countries can determine if policies and practices are achieving their benefits. This will still be a demand-led approach.
How much consultation/input was undertaken in the design of the templates?
The templates have been developed in-house, we are trialling these over the next few months and can refine as needed.
Are there particular implications and application (engagement, attribution of outcomes, reporting and evaluation) to co-funding projects with multi-lateral development banks such as the Asian Development Bank or World Bank where AWP is only a part of the activity?
We need to distinguish between our current portfolio and future portfolio, and need to be realistic about information we can collect under our existing activities. For the Tier C activities that we are starting to support – we need to be assured the extent to which Australian partners can support our in-country partners to establish the monitoring systems to assess the benefits that are received by different beneficiaries from the activity.
Will project proposals prioritised according to their Tier (eg. Tier C being the most significant?
Tier C is not “better” than Tier B. We are a demand-led organisation. The majority of activities we do in-country will continue to be focused on piloting processes and tools because that’s what our In-country partners want. We need to be mindful that AWP and our partners need to have the resources to do the project well – it’s not something we’ll take on lightly, but we want to be able to respond when the demand is there.
Does AWP have a tool, template or process for assessing activities as Significant / Principal against the Rio Markers? How were those judgements made? We are also looking at this in the Water for Women WASH Fund.
There is no current template, we have a process that is being reviewed as we go through the process. We are trying to measure the direct impact.
The templates and approaches proposed are helpful in articulating the impacts more clearly. Is it possible for AWP to facilitate Australian and in-country partners to achieve the intended impacts more rigorously?
We want to focus a lot more on the partnerships themselves and addressing issues as they arise, while preparing people to work in-country so we are able to achieve outcomes as planned.
How is AWP’s approach different from the one of ACIAR in terms of achieving intended outcomes and impacts?
AWP is moving to the mainstream aid management practice undertaken by partners across the region.
Do you think partner countries will have the resources to fill out the templates?
The intention is they’ll be very simple to fill out – might just involve an email communication between the in-country partner and AWP. Just establishing an additional channel to understand better the progress that has been achieved, and empower the partner more to feel they are managing the change and their views matter. This will be done in a proportional way according to Tier.
Looking forward, how should proposals for activities link to country level climate action, for example referring to relevant country NAPAs, NAPs, NAMAs or NDCs?
For each activity, assess the context and risk; Identify partners and best possible options, then prioritise what to do.
Will the AWP performance framework also have qualitative markers? It does seem that the current one is leaning towards more quantitative measures.
Yes, but we will need to consider what information will be required as part of a qualitative assessment on a case-by-case basis. The key contribution made, who has benefitted and to what extent will all be considered.
A question more on program content rather than the performance system: Do you expect the AWP program to see much activity in the climate mitigation space?
Depends on what you see as climate-mitigation activities. Most mitigation activities (e.g. transitions to renewable energy) use water, so there is room to consider these. NbS and hybrid solutions can have both adaptation and mitigation effects – blue carbon, green carbon – working with nature to manage water.
Keep in mind that climate finance accounting considers the intention rather than the outcome.
