Post Opening Project Evaluation (POPE) factsheet
Purpose
This factsheet outlines common evaluation limitations, contextual factors and data constraints that may affect the assessment of project performance in Post Opening Project Evaluations (POPE).
Our aim is to ensure consistency in interpretation across POPE reports and support transparent communication of evaluation constraints.
Evaluation findings should always be interpreted alongside the evidence available at the time of the evaluation. Where limitations exist, they do not necessarily indicate that a project has not delivered benefits. Instead, they highlight where evidence is unavailable, incomplete or not directly comparable, helping ensure findings are interpreted consistently and transparently.
How to use this factsheet
This factsheet supports the interpretation of POPE findings and the consistent communication of common evaluation issues. Users should refer to relevant sections when reviewing findings and use the explanations provided to ensure consistency. This factsheet supports interpretation of POPE findings but does not replace project specific analysis.
Overarching Evaluation Issues
Delayed Evaluation timing
What does this mean
Evaluation activities undertaken later than originally planned.
Why this happens
- delays in project completion
- data collection delays
- time required for environmental mitigation (for example planting) to establish
- changes in evaluation priorities
- other projects on the network affecting attribution of outcomes
Use of alternative evaluation years
What does this mean
Data from a different year than the planned evaluation period has been used.
Why this happens
- the most recent data had not yet been validated or published
- data quality checks identified issues with preferred dataset
- a more complete dataset was available from an adjacent year
- external events made a particular year unrepresentative
POPE themes
Customer journeys
Journey time impacts could not be evaluated
What does this mean
We could not determine whether journey times improved after the project opened.
Why this happens
- no suitable before-and-after journey time data available. If the network has changed significantly (for example, a new route or significant realignment) it can be difficult to source reliable before-and-after data
- baseline and post-opening data were collected using different methodologies. Depending on when data is required for or if there is route realignment it may be necessary to collect baseline and post-opening data from different sources
- route realignment may prevent consistent extraction of satnav journey time data
- incomplete or unreliable traffic data.
Traffic flow impacts could not be evaluated
What does this mean
We were unable to determine whether traffic flows increased following project implementation.
Why this happens
- issues with traffic counters
- baseline and post-opening data were collected using different methodologies
- incomplete or unreliable traffic data
Changes in speed limit
What does this mean
Changes in speed limits may influence journey times and project performance and are often implemented to improve safety on the project extent.
Why this happens
- safety is the primary objective of the project
- design changes implemented during delivery
Note: Reduced speeds do not necessarily indicate poorer project performance if safety outcomes improve.
Lower than forecast traffic flows
What does this mean
Traffic flows after the project were lower than forecast at appraisal. In some cases, the observed traffic flows were also lower than the counterfactual (the level of traffic expected without the project).
Why this happens
- a lower counterfactual than do-minimum forecasted flow suggests that the background levels of traffic weren’t accurately forecast, possibly due to events like recessions or the COVID-19 pandemic
- a lower after than do-something flow suggests that the anticipated benefits of the project may not have been realised and the forecasts were overly optimistic. For example, the forecast predicted that congestion would be reduced by the project, but congestion is still prevalent
- many projects that we are evaluating now (2026) were appraised using NTEM 7.2 and RTF 2018. These forecasts were replaced in 2022 so are now out of date.
Lower than forecast journey time savings
What does this mean
The forecasted journey time saving from the do-minimum to the do-something is greater than the actual journey time saving from the counterfactual to observed.
Why this happens
- the forecast was overly optimistic when estimating the reduction in congestion the project could achieve, resulting in less journey time savings than expected
- the forecast did not consider local developments that may have impacted traffic flows and journey times
- different models used to forecast traffic flows and journey times, leading to inaccuracies in forecasted journey times
Forecast and observed traffic flow and journey times could not be compared
What does this mean
We were unable to compare forecast and observed traffic flows and journey times.
Why this happens
- missing documentation such as traffic forecast report, economic appraisal report
- missing data within documentation
- difficulty interpreting data. For instance, forecasted data is presented within report, but without a clear explanation of how it corresponds to different sections of the route
Environmental traffic data was unavailable or incomplete
What does this mean
The traffic data needed to assess air quality, noise and greenhouse gas impacts was unavailable or incomplete.
Why this happens
- HGV data is not available
- forecasted flow data is not available or only available for certain metrics, for instance ADT and not AWT
Safety
Counterfactual test has not been undertaken.
What does this mean
We have been unable to carry out a counterfactual analysis to determine whether changes in road safety can be attributed to the project.
Why this happens
The sample size of before collisions falls under the threshold of collision numbers where we can confidently perform the counterfactual scenario.
Appraisal area has been descoped
What does this mean
We have been unable to reforecast safety value for money and have therefore used the ‘as forecast’ scenario.
Why this happens
Safety appraisal should normally include SRN links where the project was expected to result in a material change in traffic flow (typically 10% or more).
In some appraisals, this threshold was not consistently applied, resulting in an appraisal area that was not representative of where the project was expected to affect safety.
Environment
Air quality, noise and greenhouse gas impacts could not be fully evaluated
What does this mean
We were unable to fully assess the project's air quality, noise and greenhouse gas impacts because the available evidence was limited or only qualitative.
Why this happens
- suitable traffic data (for example flows, speeds, HGV data) was not available
- forecast (appraisal) information was missing, incomplete, or not comparable with evaluation data
Value for Money
Lower than expected journey time benefits
What does this mean
Journey time improvements were observed but were lower than forecast at appraisal.
Why this happens
- observed traffic growth is lower than forecasts
- nearby developments altered traffic patterns
- changes to signal timings or wider network management
- forecast assumptions differed from actual conditions
Lower than expected value for money
What does this mean
The outturn value for money assessment was lower than forecast.
Why this happens
- lower than expected journey time benefits
- increased project costs
- external factors affecting traffic patterns
- forecasted benefits are expected to be realised over a longer timeframe than assumed
- some benefits could either not be re-forecast, or were not taken into account during the appraisal, so were never monetised to begin with
