The IRIS project has defined goals and targets in the project proposal, and the monitoring and evaluation work package (WP) 9 will analyse to what extent the project reaches these goals and objectives. The monitoring and evaluation will also provide information concerning the performance of the different solutions demonstrated in the Lighthouse (LH) cities in IRIS which is important for the replication of the solutions both in the LH cities and in other cities. This is of importance for the replicability of the solutions, both in the LH cities (Utrecht, Nice and Gothenburg) and in other cities. The project consists of several demonstration projects which are divided by 5 transition tracks (TTs): TT1; Smart renewables and closed- loop energy positive districts, TT2; Smart Energy Management and Storage for Grid Flexibility, TT3; Smart e-Mobility Sector, TT4; City Innovation Platform (CIP) Use Cases, TT5; Citizen engagement and co-creation.
D9.5 is the result of 2 years of work with several iterative processes involving the LH cities and their partners with the ultimate goal to:
Define a set of Key Performance Indicators (KPIs) which evaluate the effectiveness and impact of the cities proposed measures.Setup monitoring plans for each IS to define how each parameter is being measured to ensure that the KPIs can be calculated.Define how the baseline and the targets are defined and measured.This work started as described in D9.2 (Report on monitoring and evaluation schemes for integrated solutions) [1] with:The definition of the initial list of KPIs and how to calculate them, based on Smart Cities Information System (SCIS) [2], the CITYKeys Project [3] and the IRIS project itself .The assignment of KPIs to relevant measures within the project.An evaluation plan to measure performance on project level, including aggregation of KPIs.
The process has continued with D9.3 (Report on data model and management plan for integrated solutions) [4] and D9.4 (Report on unified framework for harmonized data gathering, analysis and reporting) [5], which define the basis of the methodologies used to come to the results written in this report.
Feedback from several workshops on this topic has led to a guideline that supports the partners responsible for implementation of the demonstrators in setting up their projects such that:KPIs that are being measured are well understood.KPIs give a meaningful result.The right data is being measured to calculate the required KPIs during the implementation of the measures.
An important part of this process is to have a close look at the KPIs that are projected for each demonstrator, the calculation method of the KPIs, and the expected results. By means of KPI interpretation forms. By doing so:
• KPIs are defined and calculated such that only one way of interpretation is possible. This way results from different projects and cities are homogenized.
• It is well understood what result the measurement of a KPI leads to.The method and results of this process are described in this report, which is a revised KPI list where KPIs are added, removed or adapted.
In addition to this, the KPI interpretation forms created the basis for the formulation of detailed monitoring plans for all measures within the project. Together with template forms for reporting these plans and a common data structure, which were provided to the affiliated partners, these plans are obtained and described for all measures per Transition Track and per Lighthouse city in this report.
Another essential part of measuring the performance of the IRIS project is the establishment of the baseline measurements and review if targets are met. Tables with KPI data requirements, consisting of the associated parameters, data sources, baseline and (possible) targets for all measures are incorporated.
An important part of the monitoring strategy of the IRIS project is the KPI tool, which is described in detail in report D9.4 [5]. This tool is established to collect all relevant monitoring data from the IRIS project in order to calculate and visualize the performance of the project. The tool partly obtains it’s data by means of the City Information Platforms (CIP). The monitoring details combined with the updated KPIs, result in an inventory containing an overview of all data sources with as main objective:
• To make sure that all data sources are known and will be measured by the responsible partners.
• To know what kind of data needs to be collected by the KPI tool.
• To know when monitoring in each demonstrator starts and data can be expected.
• To have a clear overview for all responsible partners what to deliver.
Besides setting up the collection of the indicators data, D9.5 also continues the work on aggregation of KPIs. For each city a revised list is made that indicates which KPIs will be aggregated to Transition Track-, City- and IRIS-level.
In the conclusion the challenges that where met during the process of setting up the monitoring framework are described. Because of delays within the IRIS project, not all monitoring plans have been obtained yet. Therefore, a future update of this report will be submitted as soon as this information is available. Further on a perspective is described for future work to start gathering the data and visualize results of the IRIS project.
The target group for this report is mainly people who:
- Are interested in how to apply a unified monitoring and evaluation scheme into a large Smart City project with many different partners and stakeholders. For example, people working on comparable (Smart City) projects, or the follower cities within the IRIS project.
- Are interested in how the performance of several different Smart city projects can be evaluated.
- Are interested in the implementation of KPIs from projects such as SCIS and CITYkeys.
- Want to learn from project partners from within the IRIS project who work on similar projectsabout their monitoring. For example, partners from different cities affiliated with the same transition track or transition track leaders.
- Want to find out what kind of data can be expected from the IRIS project. For example, external researchers interested in the results of Smart City projects, but also partners working on WP4 (CIP) and WP9 (monitoring and evaluation).Want to learn what the current state is of the monitoring and evaluation of the IRIS project.