Monitoring Virtual Team Collaboration: Methods, Applications and Experiences in Engineering
What distinguishes high-performance engineering teams from lower-performing ones in the design of complex software, products, and services? Finding answers to this question traditionally involves extensive protocol studies and retrospective assessments of team effectiveness, e.g., in terms of adherence to budget and timelines, customer satisfaction, or innovation. Little attention has been paid to developing applicable techniques for observing performance-relevant differentiators directly in the behavioral aspects of digitally-mediated creative teamwork. The expanding role of virtual collaboration in engineering project teams requires new computational instruments to efficiently study if and how effective designing is reflected in the implicit processes, tactics, and strategies carried out over today’s dense network of groupware, e-mail, and Web 2.0 services. This dissertation handles two important aspects in the realization of such an instrument. First, it develops an adaptable monitoring service platform called d.store to capture and analyze virtual collaboration activities “on- the-fly”, i.e., while a project is still ongoing. Secondly, it applies the services in global, small-group engineering teams to identify structural differences in the collaboration behavior that correlate with independent team performance measures.
With the services provided by the d.store platform it is possible to tap into heterogeneous online communication channels and to generate a descriptive model of how teams virtually communicate, interact, and share information over the course of a project. The semantics and temporal attributes of the identified actors, resources, and relationships are represented as so-called Team Collaboration Networks. The platform is evaluated in the conceptual design phases of eleven distributed, multi-disciplinary engineering projects over a period of eight months each. The activities monitored in the e-mail archives, Wikis, and file sharing systems provide the basis for a detailed visual and quantitative examination of differences and similarities in the collaboration behavior of the observed teams.
The analysis of the generated Team Collaboration Networks indicates that high-performance design teams produce different collaboration patterns than lower-performing ones. Furthermore, the patterns that correlate with team performance suggest that an adherence to basic design principles has positive effects: Teams who applied an ‘outside-in’ perspective by emphasizing interactions with team-external stakeholders, contacts to domain experts, or group-internal knowledge sharing were generally more satisfied with their work, explored more design alternatives, or received higher ratings from independent judges. This is relevant, because it demonstrates that automatically collected objective real-time collaboration metrics can provide valuable insights into performance-relevant aspects of teamwork.
The contribution of this work is a tested, non-interfering monitoring instrument, which establishes a technological foundation for the scientific observation, comparison, and analysis of virtual collaboration activities as a service. A pilot application in engineering design gives first evidence that meaningful team performance indicators can be drawn from this approach. The results encourage a continued and intensified utilization of the instrument to assist in the evaluation of IT-mediated collaboration processes, ultimately promoting a new paradigm in the conduction of real-time team diagnostics and support in engineering design.