Evaluation of Software Mediated Process Assessments for IT Service Management

Abstract

IT service organisations are cognisant that continual service improvement can be achieved by conducting regular process assessments. However, such assessments are expensive and so we have developed a Decision Support System (DSS) tool which uses the international standard for process assessment ISO/IEC 15504 to offer a transparent and efficient approach. This paper provides evidence of evaluation of this software-mediated process assessment (SMPA) approach which was based on ISO/IEC 15504, ISO/IEC 20000 and the IT Infrastructure Library (ITIL®). In a usability evaluation of the online tool, participants reported largely positive experiences finding the online survey easy to use trustworthy, comfortable, generally effective, and more transparent and less costly to implement than a manual assessment. However, to engage in process improvement, human judgment, and possibly expert assessment facilitators are necessary for assessment validation and improvement, that is, a fully automated online survey that is strictly standard-based is not very useful. Further clarification of the survey questions with relevant examples, clearer answer options and having more visible goal statements on every question page were suggested.