Determining the level of agreement of expert estimates

Abstract

This paper considers the techniques of determining the concordance of expert estimates. Expert methods of evaluating particular criteria may be practically used, for instance, in multi-criteria decision support systems, if judgements are in good agreement. This is expressed by the concordance coefficient calculated by using expert-provided data. The concordance coefficient can be calculated only if the considered criteria are ranked. If other evaluation methods are used by experts, the data should be transformed to rankings before the calculation of the concordance coefficient is made. There is the dependence of the agreement of expert estimates on a particular method used. The highest degree of agreement has been obtained by using a direct ranking method. The effect of equally assessed criteria, that is, the tied ranks, on the concordance coefficient and thereby on the level of expert judgements agreement is usually insignificant and cannot change the results of rating.