ECSCW’93 Workshop on Evaluation Studies in CSCW
Milano, Italy, Sept 13th 1993
(Organizers: Liam Bannon & Thomas Schael)
Short Report by Liam Bannon
The following is a short report on a workshop held as part of the European CSCW Conference meeting in Milan, Italy in Sept. 1993. The full day workshop was organized by Liam Bannon (University of Limerick, Ireland) and Thomas Schael (RSO, Milano and HDZ/KDI der RWTH, Aachen). The initial call for participation listed a number of concerns that the organizers wished to address. These included:
• to examine empirical studies of CSCW systems done to date
• to investigate initial design claims and subsequent evaluations
• ......... the adequacy of many of these evaluations
• and their implications for re-design of CSCW systems
A secondary objective was to debate the relative merits of different kinds of evaluation techniques.
As is often the case in calls for participation in Workshops of this kind, the responses elicited were quite diverse in many respects, in terms of their themes, their depth, their conceptual concerns and simply their length. After an initial review process, the organizers selected 13 papers to be presented as input papers to the workshop. The papers could be loosely grouped into 3 categories, those presenting specific case studies, those that combined discussion of methods together with some case material, and those more exclusively concerned with methods and techniques ( A listing of the position papers appears at the end of this report). These were made available to all participants over electronic mail before the meeting - with one or two exceptions due to technical difficulties - so that participants could have a chance to view other people's material in advance. At the workshop proper, there were over 20 people present, as a number of interested students at the conference were allowed to attend, in addition to the original participants. The fact that 11 countries were represented in this workshop at least attests to the fact that Evaluation is seen as an issue across the world.
In terms of the papers submitted, quite a number addressed issues of evaluating particular CSCW systems in specific contexts via a variety of different techniques, and discuss the adequacy of these techniques. Another set of papers focus more generally on methodological issues of evaluation, and on the relative merits of specific techniques. However, overall there was a range of issues discussed in these papers, and the general level of argument was of a high standard.
Some of the issues that I personally wished to have aired at the Workshop centred around the whole concept of Evaluation per se, and its place in Design. There have been a large number of CSCW systems demonstrated that have had little use other than in the laboratories of their designers. What happens when these systems are moved out to confront specific workplace settings? As many of the techniques used for evaluation of systems come from specific disciplinary backgrounds and research areas such as HCI, we have to ask the question if these techniques are appropriate in specific use situations. What range of techniques are required in order to ensure that our evaluations are "ecologically valid"? Several of the papers address these concerns (van der Velden, Björklund, Tammaro, Macleod).
Some more open questions that I mention below were just intended as possibilities for further discussion during the Workshop.
• Evaluation relative to what? Who owns the problem and/or solution?
• The Need for Quantification - is there such a need?
• Suitability of specific methods for specific purposes
• Strengths and weaknesses of particular methods - can we agree on what they are?
• How to handle the disciplinary biases of researchers in practical situations
• What have we learned from our early "evaluations" of CSCW systems?
The actual workshop consisted of a short introduction to themes and issues by the organizers, followed by short presentations from each participant, followed by discussion.
Later in the day, some more general discussion themes were collated. As can be seen from the list of topics covered, the Workshop covered a wide variety of topics and perspectives, and so it was not exactly simple to present any single coherent outcome from the Workshop. Indeed, to expect such an outcome of a one-day meeting would have been over-optimistic. In discussion after the meeting, it was clear that for the majority of the participants, they found the workshop useful, not so much for in-depth discussion, as the format made this difficult, but as a chance to learn from people using other approaches and to discuss with them some of the different approaches possible, their respective strengths, drawbacks and limitations in the context of recent work in the area of CSCW. Interestingly, much of the work presented could fit quite readily into the familiar CHI type session on evaluation, as while there were a couple of ethnographic studies, the emphasis among this group had a more experimental focus and the specific issues involved in CSCW Evaluation did not surface as strongly as one might expect.
What certainly seemed to have been achieved was a greater awareness of other's perspectives and methods, and some discussion of how to use particular methods. For example, the study of Beguin introduced a number of participants to a form of ethnographic enquiry with a specifically French conceptual basis, concerning activities and actions. The need to understand the context in which data are collected was an issue in the discussion of Palme's paper on usage statistics and the interpretation of this data. Interest in learning more about network analysis (Eveland) was evident, and ethical questions about its use in particular settings discussed. Almost none of the papers discussed the interplay of design, use and re-design directly, although Segall did broach the issue in her case outline of user participation and cooperative design, and Wagner critiqued the design assumptions behind certain kinds of CSCW systems. There was also little in the way of direct critique of existing evaluation studies, something that might be of interest to many in the CSCW community and might be a specific topic to address in a future meeting.
While it was decided not to proceed with a formal publication from this Workshop, as the topics covered were so heterogeneous, there was agreement on the need for further discussion and elaboration of issues. A number of the authors were encouraged to flesh out their position papers and submit them for publication. Some of the participants at this workshop have subsequently been involved in further meetings on this theme, and it is expected that more material in this area will be published shortly.
List of Position Papers
Béguin, Pascal (CAD)
Field Evaluation and Collective Work Design in Practice with CAD
Segall, Patsy (cooperative design)
Modernisation of the Australian Tax Office
Ovaska, Saila (shared editing)
Position Paper: Evaluation Studies in CSCW
Palme, Jacob (COM system)
Use of Conferencing systems to do Research on them
Björklund, Lisbet (library information system)
Information Sharing - Evaluation of the ThemeCat
Tammaro, Susan (collaborative writing & scheduling)
Characterising Uses for Collaborative Tools : A Report on a Pilot Study
Marmolin, Hans
Ongoing Evaluation Studies of Collaborative Work within the Swedish Multi G Research Program
van der Velden, Jeroen M (3 level studies)
Exploring Methods for Evaluation and Research in CSCW
Newman, D.R et al. (CSCL)
Towards a Methodology for Evaluating the quality of Student learning in a Computer-Mediated Conferencing Environment
Macleod, Miles
Evaluation Methods for Collaborative Work
Eveland, J.D.
Uses and Limitations of Communication Network Analysis in the Evaluation of CSCW Applications
Scrivener, Steve et al.
The Use of Breakdown Analysis in Synchronous CSCW Systems Design
Wagner, Ina (critique)
Fragmented Exchange