Linking the Processes of Design, Use and Evaluation
ComputerScope article 27/9/93 Words: 905
Liam J. Bannon
In recent discussions that I have had with senior IT managers, a common concern has been their difficulty in being able to demonstrate tangible benefits from their organisation’s investment in IT. They are sure that they do exist but do not know how to prove it to others. This raises the general issue of how one evaluates the success or failure of various IT initiatives. As soon as one mentions the term evaluation, various spectres seem to loom. A standard image is of several white coated, bespectacled, usually balding male ectomorphic figures charging on horses into the firm, ready to question everybody, take away and analyse mounds of internal documents, measure everything and anything they can get their hands on, and at the end of the day produce a report that is in part indecipherable, in part trivial, - and then have the temerity to charge an arm and a leg for the service! Of course, since any form of evaluation is also in part a political act, (e.g. who decides on the evaluation criteria ? ) these kinds of evaluations may still be useful in furthering one’s own goals within an organisation (e.g. by judiciously excising appropriate passages from the evaluators’ report) but the general value of such exercises to the firm at large is often minimal.
What I would like to talk about here is not so much these overly political economic evaluation exercises but forms of observation and evaluation that are intended to give immediate and direct feedback to the system designers and managers about their developing system. Within the systems development process there is an increasing awareness of the need for early prototyping of systems and for iterative design. The basis for iterating the design is the experiences of use gained with the prototype. The old idea that evaluation is a final phase in a linear stage model of the system lifecycle is becoming discredited. Thus we should see design, use and evaluation as interleaved and inter penetrating practices, not as distinct steps in a linear development process that moves from analysis to design through implementation to use and, ultimately, evaluation.
I have been surprised at the relative dearth of studies of real use (or non-use) of computer systems in the workplace. Of course there are a number of reasons for this. The developers of software are (unfortunately) often too busy trying to sell their product to spend time on investigating how people use it. On the other side, once an end-user organisation buys a product, there is often a marked reluctance to actually study how it is being used. This is because the decision has already been made, and any negative findings are likely to reflect poorly on the IT director who made the purchase in the first place, so this person is unlikely to see the utility of such studies. In some cases where an organisation has actually conducted some evaluation, either internally or contracted out, the results of such studies are often seen as highly confidential, as again, any negative findings might engender unfavourable comment from shareholders or give useful insight to competitors. The result is that overall, our knowledge of what works and what doesn’t, and more importantly - why? is slow to accrete, with a resultant loss for both software developers and end users.
Over the past few years, I have been attempting to distil some lessons from a number of empirical studies of use of different groupware systems that are publicly available. In the remainder of this piece, I will just mention one insight from this work that I find of particular interest. In a future article, I will discuss some of the other lessons that we can learn from these kinds of studies. The Information Lens system was developed as a research prototype by Tom Malone and colleagues at MIT some years ago, and the ideas embodied in it have now been incorporated into several commercial products. The system is designed to support people in managing their electronic mail through allowing users to define rules that can help sort and archive mail based on characteristics of the message. Wendy Mackay performed a field study of use of several evolving prototypes of the system. Her work shows the importance of the social environment in affecting the use and development of Lens, with a local expert exerting considerable influence. Mackay notes:" Software does not remain static when it is introduced into an organisation. People in the organisation evolve their individual patterns of use, share them with each other, react to external changes, both technical and non-technical, and sometimes pro-actively modify the system to produce significant innovations." In the current discussion, these observations are important, as they show how ongoing evaluation of prototype Lens systems led to new ideas about the very conception of Lens as a tool, which could be factored into the next iteration of the system. In other words, the very conception of what Lens was, and how it was to be used, changed for the design team as they witnessed the way people actually used their prototype system. This provides a very powerful example of how important ongoing observation/evaluation studies are. On another level, it changes how we think of the design process, and the actors which it encompasses, as we see original ideas about what the tool is and how it can be used coming from the "users" themselves.
Dr. Liam Bannon is Director of the Centre for Research in Computer Supported Cooperative Work (CSCW) and Senior Lecturer in the Department of Computer Science and Information Systems at the University of Limerick. Trained in psychology and computer science, he has worked in North America and continental Europe for many years. His research and consulting interests focus on empirical studies of use of groupware, user interface design, and human and organisational issues in the development and implementation of IT. He is the author of numerous books and articles in these areas. Contact: Phone: (061)-333644: Fax: (061) 330876: email: bannonl@ul.ie