ch8

8 Evaluating ICT and Development

This chapter is based on Kivunike, F., Ekenberg, L., Danielson, M. and Tusubira, F. Towards a Structured Approach for Evaluating the ICT Contribution to Development, The International Journal on Advances in ICT for Emerging Regions. 7(1). pp. 1–15. 2014.

The evaluation of the contribution of information and communication technologies (ICT) to development has been challenged from theoretical, ethical and methodological angles. This chapter proposes a model to address some of these challenges that enables systematic evaluation of the contribution of ICT to development. The proposed model is conceptually motivated by Amartya Sen’s capability approach – which defines development as freedom – as well as the ICT for development (ICT4D) value chain. Development is a process that involves the provision of opportunities (capabilities) from an ICT resource, as well as actually exploiting the opportunities to realise development benefits. The conversion of resources to opportunities and opportunities to development benefits is facilitated or inhibited by various contextual factors. Development from the capability perspective is both people-centred and multidimensional. This requires consideration of both instrumental effectiveness and intrinsic importance. The proposed evaluation process involves assessing the opportunities to realise benefits as well as the exploitation of them. Five evaluation dimensions concerning social and economic development are proposed, namely: research and education opportunities, healthcare, economic facilities, political freedoms and psychological well-being. ICT4D evaluation indicators are suggested for each dimension and a multi-criteria decision analysis (MCDA) structured evaluation process is proposed to guide the evaluation.

The evaluation of the contribution that ICT investments make to development continues to be an issue of great concern for both researchers and practitioners in ICT related development. The realisation of development benefits from ICT involves complex interactions between the technology, people and the context. Development is a complex phenomenon, meaning different things to different people. The predominant or most promoted approach to ICT related development evaluation today focuses on the ‘D’ in ICT4D to tease out the development benefits. Progress has been registered especially in incorporating theoretical approaches such as development theories in the evaluation of the contribution of ICT to development. Given the inherent complexities, these studies/evaluations are typically micro-based, focusing on individual or community evaluation and are mostly achieved through qualitative in-depth descriptions. These approaches generate a large amount of qualitative data and are not suitable for macro/meso levels, as they would lead to complex analysis problems. However, there is still limited evidence of approaches seeking to establish the contribution of ICT even at higher macro/meso levels, such as national development goals, or multiple projects such as healthcare delivery, education, universal access and so on.

There is a need for structured approaches to facilitate an objective process for evaluation of the contribution of ICT to development. It is envisaged that the structured approach is intended to streamline the data collection and analysis process to ensure that the method is neither so simplistic that it overlooks essential details nor so elaborate that it inhibits proper reporting. It is believed that such structured approaches support large-scale evaluations which lead to inclusive development rather than selective development for only a few, as is normally reported for specific project evaluations. Moreover, a structured evaluation approach can also facilitate evaluation of a specific initiative at a micro level, for example, the contribution a community ICT facility makes to individuals’ overall well-being.

To contribute to a growing field of ICT4D evaluation, this chapter addresses some of these challenges by adopting an interdisciplinary approach to ICT4D evaluation. It adopts an indicator-based approach whose model is based on development and information system models. It then applies multi-criteria decision techniques to facilitate structured data collection, analysis and reporting.

A literature review of the current state of evaluating the ICT contribution follows. A discussion of the underlying conceptual foundations applied in this study as well as the composition and interactions of the proposed model are then presented in the next section. This is followed by a proposition of possible criteria, and an explanation of how a MCDA can be applied to perform the evaluation. The chapter also gives different scenarios in which the evaluation approach could be applied, before concluding with a discussion of limitations and recommendations for future works.

Evaluating the Contribution of ICT to Development

There is an increase in studies in the area of ICT4D evaluation, with the most prevalent development-based approaches being the capability approach, the sustainability livelihoods approach and others mostly used in international development evaluation such as logical framework and results-based management. The capability approach is being increasingly applied to ICT4D research especially for qualitative evaluation. Studies have recently emerged that apply the approach to the development of ICT4D evaluation indicators. These propose both quantitative and qualitative indicators for ICT infrastructure, uses and capabilities and also suggest other indicators to evaluate the contextual influences on uptake. Partially drawing from the capabilities aspect of the approach, ‘sustainable livelihoods’ is defined as comprising the capabilities, assets (including both material and social resources) and activities required for a means of living. There is also evidence of substantial use of the sustainable livelihoods approach (SLA) in ICT4D. The SLA framework is applied in development evaluation because it is considered to be flexible and therefore applicable to different contexts, since it considers a wide range of aspects pertinent to the development process. On the other hand, it is criticised for its complexity as it contains a multiplicity of variables that make it more costly and time-consuming to implement, and difficult to form conclusions and generalisations from.

Programme theories include a collection of approaches that aim to show the logic between programme or project activities and expected outcomes. Common to all programme theory approaches is the underlying causal logic model which may be implicit or explicit, depending on the sources of information. The influence of contextual factors on programme results and a mechanistic approach to determining causality are also central in programme theories. It is apparent that any social intervention involves several factors that would contribute to the realisation of the outcome; yet it is complicated to attribute change to specific interventions. The goal, therefore, in programme evaluation is to understand the contribution of specific interventions and not to attribute change to them. This is the basis upon which the evaluation criteria are developed in this chapter. Programme theories have been referred to by a variety of names including theories of change, impact pathways and pathways of change.

Examples of methods that apply programme theories to establish a causal logic model are the theory of change (also contribution analysis), logical framework and results-based management. They are typically formative evaluation approaches that assess project progress against objectives. A more recent application of programme theory in development evaluation is the use of randomised control trials (RCTs) or randomised impact evaluation. Proponents point out that RCTs combat selection bias, which is inherent in several social programmes. While they may not be appropriate for all development problems, they have been applied to various cases, and are only starting to be adopted in ICT4D evaluations.

The majority of the ICT4D evaluation studies cited above apply development approaches to perform in-depth descriptive analysis. As a point of departure and contribution to this body of knowledge, the model suggested in this study illustrates the use of a structured evaluation approach that relies on indicators in the evaluation of the ICT contribution to development.

Conceptual Foundations

Theoretical and conceptual foundations are essential to the realisation of sound evaluation approaches to support ICT evaluations. This facilitates the understanding of how technology interacts with society to achieve development. ICT4D studies fall within an emergent multidisciplinary field now referred to as ‘development informatics’ that seeks to integrate development theories within information systems, communication studies as well as computer science. This fairly new field resulted from the knowledge that there is more to ICT4D than just diffusion, adoption and use. The need to establish the real benefits of ICT in terms of what they are used for within various contexts called for new approaches. Consequently, there is a need for sound theoretical premises as a basis for research on how ICT is integrated and affects people’s everyday lives and businesses as well as national and international development goals. Starting with the ICT4D value chain as a guide, the focus of evaluation in terms of the ICT4D implementation lifecycle in this study is identified. The capability approach is then applied to facilitate the definition and understanding of what development is and how it is realised.

Figure 1. ICT4D value chain.

The ICT4D Value Chain

The ICT4D value chain model facilitates an understanding of ICT4D evaluation. It is based on the standard input-process-output model, linking resources and processes in order to systematically analyse the stages an ICT initiative traverses over time (see Figure 1). The input, an ICT4D intervention in combination with fulfilled prerequisites such as policies and implementation skills, facilitates the realisation of a deliverable, for example, a telecentre, or an e-library platform. These deliverables, once exploited by the target beneficiaries, result in outputs, which lead to outcomes and ultimately impact. The realisation of outcomes from outputs as well as impact from outcomes is affected by various contextual factors such as skills, institutional barriers and cultural or personal beliefs and so on.

Over the years interest in the domains along the value chain has shifted from readiness, availability and uptake towards development impact. This shift arises from the need for ICT4D initiatives to demonstrate that they actually contribute to social and economic development. However, the challenge in such evaluations is that as, one moves from outputs to impact, evaluation becomes more complex since the focus shifts from the technology to the development goals. As a result outcomes and impact cannot be attributed to a specific initiative since there are other factors or even initiatives that could have affected the outcome. To address this challenge, it is argued in this chapter that, rather than aiming at proving causality, emphasis should be placed on the contribution an initiative has made to social and economic development. This refers to the change in terms of social and economic development resulting from the presence of that intervention, within the boundaries of the contextual factors. Furthermore, focusing on the contribution is appropriate in situations where baseline studies were not performed to facilitate a longitudinal evaluation of the initiatives.

Moreover, the impact concepts – that is, outputs, outcomes and impact as per the value chain – have been variously defined based on the different approaches applied to the design and the evaluation of projects or programmes in international development. Generally, outputs are the immediate results of the programme or initiative. These can either be goods or services: such as workshops held, information produced, or changes in skills. In this study ICT4D outputs are the behavioural changes associated with technology use, consisting of the new information and decisions, new communication patterns and new actions and transactions that an ICT enables. Moreover, outputs in telecommunications are similarly defined as information made available and retrievable by computer. Outcomes (purpose), on the other hand, are the effects of outputs; in this study they are the direct benefits in terms of measurable (both quantitative and qualitative) benefits as well as costs associated with the outputs. Finally, development impact refers to the contribution of ICT to the broader development goals – impacts are less tangible. They are the long-term effects of the interventions. The output and outcome definitions adopted in this study are similar to the concepts of opportunities and achievements that are discussed in the subsequent section.

The value chain assumes a linear relationship between ICT and development, but this does not adequately represent the process since there are several aspects involved in explaining how and why development would result from an ICT4D initiative. For this reason, and because of the need to define adequately what development is and how it is realised in a given context, there is a need to adopt and integrate a development perspective as discussed in the following section.

The Capability Approach

A development theory perspective facilitates the definition of what constitutes development. For this purpose, Amartya Sen’s so-called capability approach has been adopted since it facilitates a multi-dimensional, people-centred approach to defining what constitutes development. Development, according to Sen, is the expansion of freedoms (capabilities or opportunities) to enable people to lead the lives they value. Development is more than the provision of access to a resource such as ICT: it is about what ICT can enable people to be or do given their contextual aspects. One of the reasons freedom is central to development is for purposes of evaluation. Sen points out that ‘assessment of progress has to be done primarily in terms of whether the freedoms that people have are enhanced’. Basically it (1) views development in terms of values, for example, being healthy, being educated or being happy; and (2) evaluates how these have been enhanced by, for example, access to the internet in a given context. The premise of the capability approach is that a vector of a resource is transformed into a capability set within the restriction of conversion (contextual) factors. The capability set consists of functionings – things one can be or do to obtain the life one values. Simply defined, the capability set is the opportunities a development initiative offers. Achieved functionings, on the other hand, are the opportunities one chooses to exploit given one’s specific context.

The capability approach also highlights the role human diversity plays in the realisation of development. Diversity mostly results from people’s personal as well as external factors. These factors, referred to as conversion (or contextual) factors, determine people’s preferences and choices of potential functionings. Conversion factors are classified as personal – individual characteristics such as physical disabilities, motivation, level of education, age, gender and sex; and social factors – the external legalities or societal requirements that may consist of public policies, social or cultural norms and discriminating practices. Another emerging category of social factors here is that of intermediaries, for example, non-government agencies that seek to promote ICT usage. Lastly, environmental aspects focus on location and accessibility of facilities, as well as technical aspects such as quality of service. An individual’s capability set comprises both well-being – the opportunities available for a better life – and agency – one’s ability to choose from the available opportunities based on personal values and circumstances. Agency takes into consideration the active involvement of beneficiaries in their development process; that is, whether they choose to exploit the available facilities for the improvement of their lives or not, depending on what they value and the prevailing circumstances.

The following are the multiple evaluation spaces within which policies and initiatives can be evaluated: well-being freedom which focuses on the capabilities or opportunities an initiative fosters; well-being achievement which is the achieved functionings; agency freedom which evaluates the freedom to achieve whatever a person decides he or she should achieve; and finally agency achievement which is the outcomes in terms of one’s values, including those of other people and things.

Proposed ICT4D Evaluation Model

As suggested by the capability approach, the realisation of development from an initiative is a process that besides the provision of the opportunities (capabilities) also involves the interaction of these capabilities with choice that is influenced by the conversion factors. This highlights two aspects: first, the need to perform a process analysis from capabilities to achieved functionings; and second, the need to explicitly establish the conversion factors that influence people’s choices. Focusing on achieved functioning alone denies one insight into the process that is essential given that development initiatives are highly dependent on context. On the other hand, focusing on capabilities alone offers limited development evaluation, focusing on what the initiative initially can do, and not what it has actually done. Focusing on capabilities alone may also be perceived as techno-centric since evaluation is only performed on the opportunities an initiative can offer and does not investigate whether these were achieved.

Figure 2. Proposed ICT4D evaluation model.

Based on the above discussion, the constructs of the proposed evaluation model include ICT characteristics, conversion factors, opportunities (capabilities) and achievements (choice, personal or community goals, and achieved functionings), as shown in Figure 2. The ICT characteristics that a resource enables (communication; production, processing and distribution of information) provide opportunities within the limitations of the personal, social and environmental factors. Achievements are the opportunities one chooses to exploit within the restriction of conversion factors, and choice is also explicitly evaluated as one of the achievements.

Although governments (as well as development partners) can provide opportunities, they cannot decide how people live their lives. It is assumed that if someone’s ability to make choices is increased or strengthened, it will enable them to choose to live the life they value. Outputs in relation to these definitions are the opportunities, while outcomes are the achievements. The achievement of certain functionings enables other opportunities: this is shown by the double pointing arrows between outputs and outcomes. For example, sensitisation to the benefits of using the internet empowers individuals to make wise decisions on how to use it.

ICT4D Evaluation Criteria

Sen proposes five instrumental freedoms that enhance people’s capabilities: social opportunities, economic facilities, political freedoms, transparency guarantees and protective security. It is argued that the extent to which these are secured is indicative of the level of an individual, household or community development. Since these freedoms are interrelated and supplement each other, they have been condensed to three: social opportunities, economic facilities and political freedoms. A fourth dimension, psychological well-being, is proposed as this evaluates the substantive freedoms such as choice and self-esteem.

Contingent on the nature of the initiative being assessed, all or just some of the dimensions may be applied. It is however imperative that psychological well-being is evaluated for all initiatives since it affects the achievements in other dimensions (e.g. as discussed in relation to choice). The dimensions are defined as follows:

Social opportunities: arrangements society makes available to enable an individual to live a better life. From the capability perspective, this specifically focuses on education and healthcare.
Economic facilities: opportunities that individuals enjoy to utilise resources for the purpose of consumption, production or exchange. This includes aspects such as productivity, employment etc.
Political freedoms: opportunities people have to exercise their political rights e.g. being able to participate in local elections, community development programmes etc.
Psychological well-being: physical, emotional and personal development opportunities. These are mostly a result of using ICT or participating in ICT4D projects. Examples include gaining respect from peers or gaining increased self-esteem. Psychological well-being has both substantive and instrumental value in that it enables people to exploit other opportunities in pursuit of development.

Depending on the nature of the evaluation, a set of criteria (as well as sub-criteria if necessary), such as ‘Improvement in research quality and innovations’ or ‘Improved access to health services’, can be defined for each dimension to facilitate an evaluation process. For each dimension, achievements (outcomes) and opportunities (outputs) are proposed. For example, it is presumed that to assess whether an initiative has improved access to formal or non-formal education (outcome/achievement) in the research and education dimension the following opportunities (outputs – what people do) are evaluated:

Accessing information in relevant online resources, e.g. research journals, online libraries
Participating in online research collaborations, e.g. through discussion forums
Producing and publishing research outputs, e.g. journals, patents etc.

These are further granulated to define output and outcome indicators, such as:

Accessing information in relevant online resources e.g. research journals, online libraries
Accessing information in relevant online resources e.g. online courses/tutorials, e-learning platform, research journals, online libraries
Accessing health-related information e.g. websites or short text messaging services that share information on good health practice, immunisation, pandemics etc.

Those indicators measure whether end users exploit the opportunity in terms of quality and usage. Quality seeks to establish whether end users actually value the opportunity, which will determine whether it is exploited.

The indicators proposed in this study are mostly qualitative and do not require precise data specifications. It is envisaged that the qualitative assessment facilitates a structured, approach that provides sufficient information to report the contribution of ICT to development. Elicitation of data for this approach relies on beneficiaries’ perceptions, which can be imprecise information about how initiatives have been of benefit to people’s well-being. Moreover, the use of structured approaches to evaluate the contribution of ICT to development is also recommended as a replacement for access and usage measures, which offer little as far as defining the actual ICT benefit is concerned.

Given the qualitative nature of the evaluation process, data collection, analysis as well as presentation of results can be supported by the more flexible systems-oriented and modelling techniques such as systems dynamics and MCDA. These facilitate the decomposition of complex decision problems for which quantitative approaches may be difficult or even inappropriate. For instance, the DecideIT platform for handling imprecise and vague information can be adapted for ICT4D evaluation. The approach facilitates multidimensional and multi-stakeholder assessment processes and evaluations, when the handling of uncertainty attributed to incomplete and vague information is necessary. This is a more instrumental alternative to the predominantly descriptive ICT4D evaluation approaches. An illustration of its applicability is presented in Chapter 19.

Benefits to the ICT4D Evaluation Practice

Recent empirical studies have proved that the proposed evaluation approach can be applied in different assessment scenarios depending on aspects such as the purpose of evaluation, level of analysis and availability of data. Examples include:

An evaluation of how an initiative or project contributes to one or more development outcomes. In this case the evaluation is of a single initiative aimed at achieving one or more goals. This was demonstrated in the evaluation of the contribution of online learning (the initiative) to students’ access to learning (the aim or goal). This study is reported in Chapter 19. While the study considered a single goal, the evaluation could be performed for more goals.
A comparative assessment of the performance of two or more similar projects or initiatives on various social and economic outcomes. The initiatives should be similar in the sense that they aim at achieving the same goal, and can be evaluated according to the same set of criteria (outputs, outcomes and contextual factors). A typical case is the comparative evaluation of the ICT contribution to improved healthcare delivery in rural healthcare facilities in Uganda.
An ex-ante evaluation of project proposals to establish perceptions of how they will perform on various outcomes and within the different contexts. Depending on the number of project proposals, as well as the target goals, this form of appraisal could take on the format in either scenario (a) or (b) above. For instance, if it is a single project aimed at achieving multiple goals, then scenario (b) would be the most appropriate. On the other hand, if there are multiple projects aimed at achieving one or more goals, then scenario (a) is the preferred option.
An evaluation of the influence of contextual factors on the development outputs and outcomes of one or more initiatives. This is achievable in various ways. In the first instance, the contextual factors are one of the criteria categories just like the outputs or outcomes in the evaluation model. Alternatively, the influence of the contextual factors on the outputs and outcomes can be explicitly performed especially when the aim is to assess project risks.

Furthermore, the application of such an approach to evaluating the contribution of ICT to development is particularly recommended to supplement access and usage indices that offer little as far as defining the actual benefits is concerned. For example, when the proposed approach is applied for the evaluation of telecommunications and communications policies, ICT resources are assessed in relation to their provision of a range of opportunities rather than quantities. In this way, these evaluations establish how well or badly the policy, such as universal access/service, performs in terms of which of the defined opportunities have been achieved, and those that have not been realised. This provides a rich analysis in comparison with the evaluations of quantities which have been predominant with policy evaluations.

A Structured Evaluation Model

This chapter proposes a structured model for the evaluation of the contribution of ICT to development. The model is based on the capability approach with aspects drawn from the ICT4D value chain as conceptual framework. One of the major challenges with the capability approach has always been its strongly and profoundly philosophical basis, which complicates attempts at its operationalisation. The work presented here contributes to the operationalisation of the capability approach as well as applying a development perspective to the evaluation of the contribution of ICT to development. However, unlike the existing applications of the approach, this study illustrates the use of indicators in the evaluation. Moreover the proposed approach offers more in comparison with the quantitative evaluations of availability and uptake. It is also multi-dimensional, explicitly considering the instrumental and substantive benefits of ICT, as well as the context in which they should be obtained. It further stresses the need to evaluate psychological well-being alongside the other dimensions, because this is both a means and an end in ensuring development.

The approach is envisaged to benefit ICT4D evaluation efforts for which in-depth descriptive evaluations are not possible due to various constraints related to budget, logistics or insufficiency of data. It may also serve for the comparative evaluation of multiple projects, ex-ante evaluation of development project proposals, and establishing the influence of contextual factors on the realisation of development benefits. To demonstrate its applicability, various empirical studies have been conducted, and a subsection of the proposed criteria was also applied in iMentors, an EU project developing a platform which will enable donors and development partners to review complete or existing projects to provide policy support and assist programme planning and implementation.

A limitation of the model is that it does not explicitly address unintended or negative benefits that are prevalent in any development initiative. In addition, the use of the more flexible systems-oriented and modelling techniques which facilitate the modelling of more qualitative, imprecise information are only in their infancy and would benefit from further studies to test their applicability. The proposed model could also provide a good basis upon which similar evaluations in other fields besides ICT could be built.