Final Report Summary - CEES (“C-E-E-S” (Citizen oriented Evaluation of E-Government Services: A Reference Process Model))
Research by Brunel University, American University of Beirut, and Turkat has assisted three governments in evaluating and improving their e-government services, to make them more user-friendly for citizens. CEES (Citizen-Oriented Evaluation of e-government Services, http://www.iapp-cees.eu) project, funded by EU’s Seventh Framework Programme for Research (FP7) People program, delivered a new evaluation model, called COBRA, for benchmarking e-government services from citizens’ perspective. CEES led to COBRA’s adoption in at least five countries throughout the world for their e-government service improvement. The model has been successfully applied in order to identify e-services for further improvement by a Turkish central e-government service provider (Turkat) which has 12 million citizen users. The application resulted in more favourable citizen attitudes toward the e-government services after their improvements based on the COBRA (Cost, Opportunity, Benefit, Risk Analysis) framework. Also, the research outcome enabled the launch of a new project, called I-MEET that will extend the COBRA framework and be applied in Qatar, Lebanon and the UK. E-government service providers such as ictQATAR and OMSAR (Office of the Minister of State for Administrative Reform) decided to adopt the COBRA framework to evaluate and improve their e-government services in Qatar and Lebanon respectively.
The research was conducted as a part of the EU FP7 project CEES between April 2009 and March 2013. The CEES consortium consists of three partners namely Brunel University, American University of Beirut, and Turksat. Brunel Business School has the coordinator role of the consortium. The research started on April 2009 and ended in March 2013. The outcome of the project is a citizen-oriented e-government services evaluation model, called COBRA, and a reference process model that defines the steps with which COBRA is applied to e-government service evaluation in different national contexts. The COBRA and reference process model were verified through four field trials conducted in Turkey (June 2010, Jan 2011 for COBRA), the UK (October 2012 for the reference process model), and Lebanon (Jan - Mar 2013 for the reference process model).
Evaluation of e-government services (e-services) is a challenging task involving several factors and a large number of stakeholders with conflicting interests, which may limit citizens’ satisfaction and e-services take-up. Each e-service achieves certain system performance levels in the various aspects of interest and each citizen has different online behavioural experiences. The project aimed at understanding the main factors that affect citizen’s satisfaction and proposes a novel analytic approach to measure satisfaction from an e-service. The approach integrates measurement factors derived from the literature on e-services and new measurement factors based on citizen’s behaviour and experience during the online interaction. For each factor, tangible and intangible variables were identified through literature review and focused group interview by Brunel University and AUB in the first 6 months of the project in 2009. These variables were grouped into cost; benefit; risk and opportunity categories. These factors and variables were validated using focus groups of academics, e-government experts and users. Statistical analysis was conducted to establish proof of relationships among the identified variables, factors and users’ satisfaction. The Data Envelopment Analysis (DEA) was then employed to balance the trade-offs between the input (cost and risk variables) and the output (benefit and opportunity variables) to derive an aggregate measure of satisfaction. For the purpose of improving inefficiencies, DEA focuses on the best practice of efficient e-services. Unlike traditional evaluation approaches, DEA identifies inefficient e-services and the magnitude of these inefficiencies then sets improvement targets based on best-practice rather than average performance. Usage experiences were reported on a sample of 2785 Turkish users of e-services provided by the e-government portal at TurkSat during June 2010 – November 2010. The analytical results demonstrated how to implement the new DEA-based approach for the measurement of user’s satisfaction with e-services. They also established a proof of validity for DEA as an alternative method to traditional methods that seldom identify and quantify the individual factors for dissatisfying e-services. Based on DEA results, managers were able to establish a national best-practice benchmark and e-service performance indices, track progress and measure impact of policies on e-service design and improvement over time. The research outcome provided Turksat with a starting point to improve their e-government service provision through the following process:
Firstly, the Turksat team analysed the research outcome that assessed their e-government services based on their relative satisfaction by citizens for further improvement. This task was undertaken by the Turksat business analysis team in which the Turksat members of the CEES consortium played an advisory role. The advisory members explained the meaning of the assessment rank for the services based on the CEES framework. The interpretation of the service ranks were performed based on how much the citizen satisfaction can be further improved compared with similar e-services that provided citizens with the highest satisfaction. Also, they explained what factors (for example, easy access to service, risks in sending their private information over the internet, and the experience with supporting staffs of the web sites) are critical in increasing citizen satisfaction. Secondly, the team set target objectives for the improvement of the inefficient services. This was based on the strategic importance of each service (for example, citizen health related e-service is more important than library complaint e-services, and is more widely used) and its potential for improvement (sometimes it is meaningful to allocate resources to improve citizen satisfaction for a service by 1% rather than improving other services by 10%). Thirdly, the team identified core areas to achieve target improvement for each service and made action plans to achieve the goals. For example, it emerged that one of the areas requiring urgent attention was the problem of disseminating user authentication credentials; many citizens had difficulties in logging in to the system due to miscommunication during getting the log-in credentials from the website. Also, e-services which require personal data including family registration and house addresses achieved very low assessment due to the existence of incorrect data. Lastly, the team implemented the action plans by updating the e-services provided to the citizens. The improvements’ implementation lasted for a year and additional data collection was undertaken to verify the impact of the changes in the provision of the services. It was reported by Turksat that the use of COBRA framework was a useful tool to improve their engagement with citizen users.
Following articles were published on the international journal based on the outcomes from the project.
- Lee, H., Irani, Z., Osman, I., Balci, A., Ozkan, S., and Medeni, T. (2008), Research Note: Toward a reference process model for citizen oriented evaluation of e-government services. Transforming Government: People, Process, and Policy, 2 (4), 297 - 310. 10.1108/17506160810917972
- Osman, I. H., Anouze, A. L., Irani, Z., Al-Ayoubi, A., Lee, H., Balci, A., Medeni, T., and Weerakkody, V. (2012) A New COBRA Framework to Evaluate E-government Services: A Citizen Centric Perspective, Government Information Quarterly, in second review process.
- Irani Z, Weerakkody V, Kamal M, Hindi M, Osman I, Anouze AL, El-Haddadeh R, Lee H, Osmani M (2012) An Analysis of Methodologies Utilised in e-government Research: A User Satisfaction Perspective. Journal of Enterprise Information Management, 25 (3), pp. 298 - 313. 10.1108/17410391211224417
- Tsohou A., Lee H., Zahir I., Weerakkody V., Osman I., Latif A., Medeni T., “Evaluating E-government Services From A Citizens’ Perspective: A Reference Process Model”, CD-ROM/Online Proceedings of the European, Mediterranean & Middle Eastern Conference on Information Systems (EMCIS), (Eds. Ghoneim A., Klischewski R., Schrödl H., Muhammed K.), 7-8 June, 2012, Munich, Germany
- Tsohou A., Lee H., Zahir I., Weerakkody V., Osman I., Latif A., Medeni T., “Proposing a Reference Process Model for the Citizen-Centric Evaluation of E-government Services”, Transforming Government: People, Process and Policy, forthcoming
The research was conducted as a part of the EU FP7 project CEES between April 2009 and March 2013. The CEES consortium consists of three partners namely Brunel University, American University of Beirut, and Turksat. Brunel Business School has the coordinator role of the consortium. The research started on April 2009 and ended in March 2013. The outcome of the project is a citizen-oriented e-government services evaluation model, called COBRA, and a reference process model that defines the steps with which COBRA is applied to e-government service evaluation in different national contexts. The COBRA and reference process model were verified through four field trials conducted in Turkey (June 2010, Jan 2011 for COBRA), the UK (October 2012 for the reference process model), and Lebanon (Jan - Mar 2013 for the reference process model).
Evaluation of e-government services (e-services) is a challenging task involving several factors and a large number of stakeholders with conflicting interests, which may limit citizens’ satisfaction and e-services take-up. Each e-service achieves certain system performance levels in the various aspects of interest and each citizen has different online behavioural experiences. The project aimed at understanding the main factors that affect citizen’s satisfaction and proposes a novel analytic approach to measure satisfaction from an e-service. The approach integrates measurement factors derived from the literature on e-services and new measurement factors based on citizen’s behaviour and experience during the online interaction. For each factor, tangible and intangible variables were identified through literature review and focused group interview by Brunel University and AUB in the first 6 months of the project in 2009. These variables were grouped into cost; benefit; risk and opportunity categories. These factors and variables were validated using focus groups of academics, e-government experts and users. Statistical analysis was conducted to establish proof of relationships among the identified variables, factors and users’ satisfaction. The Data Envelopment Analysis (DEA) was then employed to balance the trade-offs between the input (cost and risk variables) and the output (benefit and opportunity variables) to derive an aggregate measure of satisfaction. For the purpose of improving inefficiencies, DEA focuses on the best practice of efficient e-services. Unlike traditional evaluation approaches, DEA identifies inefficient e-services and the magnitude of these inefficiencies then sets improvement targets based on best-practice rather than average performance. Usage experiences were reported on a sample of 2785 Turkish users of e-services provided by the e-government portal at TurkSat during June 2010 – November 2010. The analytical results demonstrated how to implement the new DEA-based approach for the measurement of user’s satisfaction with e-services. They also established a proof of validity for DEA as an alternative method to traditional methods that seldom identify and quantify the individual factors for dissatisfying e-services. Based on DEA results, managers were able to establish a national best-practice benchmark and e-service performance indices, track progress and measure impact of policies on e-service design and improvement over time. The research outcome provided Turksat with a starting point to improve their e-government service provision through the following process:
Firstly, the Turksat team analysed the research outcome that assessed their e-government services based on their relative satisfaction by citizens for further improvement. This task was undertaken by the Turksat business analysis team in which the Turksat members of the CEES consortium played an advisory role. The advisory members explained the meaning of the assessment rank for the services based on the CEES framework. The interpretation of the service ranks were performed based on how much the citizen satisfaction can be further improved compared with similar e-services that provided citizens with the highest satisfaction. Also, they explained what factors (for example, easy access to service, risks in sending their private information over the internet, and the experience with supporting staffs of the web sites) are critical in increasing citizen satisfaction. Secondly, the team set target objectives for the improvement of the inefficient services. This was based on the strategic importance of each service (for example, citizen health related e-service is more important than library complaint e-services, and is more widely used) and its potential for improvement (sometimes it is meaningful to allocate resources to improve citizen satisfaction for a service by 1% rather than improving other services by 10%). Thirdly, the team identified core areas to achieve target improvement for each service and made action plans to achieve the goals. For example, it emerged that one of the areas requiring urgent attention was the problem of disseminating user authentication credentials; many citizens had difficulties in logging in to the system due to miscommunication during getting the log-in credentials from the website. Also, e-services which require personal data including family registration and house addresses achieved very low assessment due to the existence of incorrect data. Lastly, the team implemented the action plans by updating the e-services provided to the citizens. The improvements’ implementation lasted for a year and additional data collection was undertaken to verify the impact of the changes in the provision of the services. It was reported by Turksat that the use of COBRA framework was a useful tool to improve their engagement with citizen users.
Following articles were published on the international journal based on the outcomes from the project.
- Lee, H., Irani, Z., Osman, I., Balci, A., Ozkan, S., and Medeni, T. (2008), Research Note: Toward a reference process model for citizen oriented evaluation of e-government services. Transforming Government: People, Process, and Policy, 2 (4), 297 - 310. 10.1108/17506160810917972
- Osman, I. H., Anouze, A. L., Irani, Z., Al-Ayoubi, A., Lee, H., Balci, A., Medeni, T., and Weerakkody, V. (2012) A New COBRA Framework to Evaluate E-government Services: A Citizen Centric Perspective, Government Information Quarterly, in second review process.
- Irani Z, Weerakkody V, Kamal M, Hindi M, Osman I, Anouze AL, El-Haddadeh R, Lee H, Osmani M (2012) An Analysis of Methodologies Utilised in e-government Research: A User Satisfaction Perspective. Journal of Enterprise Information Management, 25 (3), pp. 298 - 313. 10.1108/17410391211224417
- Tsohou A., Lee H., Zahir I., Weerakkody V., Osman I., Latif A., Medeni T., “Evaluating E-government Services From A Citizens’ Perspective: A Reference Process Model”, CD-ROM/Online Proceedings of the European, Mediterranean & Middle Eastern Conference on Information Systems (EMCIS), (Eds. Ghoneim A., Klischewski R., Schrödl H., Muhammed K.), 7-8 June, 2012, Munich, Germany
- Tsohou A., Lee H., Zahir I., Weerakkody V., Osman I., Latif A., Medeni T., “Proposing a Reference Process Model for the Citizen-Centric Evaluation of E-government Services”, Transforming Government: People, Process and Policy, forthcoming