Assessing the accuracy of expert methods

Abstract

Multi-criteria evaluation methods are widely used now in handling complicated problems. However, many questions still remain unresolved in this area. One of the main problems of multi-criteria analysis is the determination of weights of sub-criteria describing a considered object from various perspectives. This is usually achieved by using expert systems. The evaluation is based on various methods. They include the ranking approach, direct method of sub-criteria weight determination, when the weight expressed in parts of one or any other number is preassigned, and the indirect method, when the criteria are scored on a particular scale (5, 10, 20, etc.) and an expert determines the criterion weight not associating the estimate with those of other criteria. The method of weight determination by pair wise comparison developed by T. Saaty which is based on pair wise comparison matrix is also extensively used. The calculations revealed the variation of weight values obtained by the same experts using different evaluation methods. The question arises, which of the above methods is the best in determining the proper weight of sub-criteria and if the proper weight which is unknown and can hardly be known can be actually determined. The answer to this question cannot be found in the literature; therefore, an approximate way of solving this problem is suggested in the present paper. In this approach, the deviation of the weights of all considered sub-criteria obtained by all particular methods from the average weight values of the i-th criterion calculated by all the methods is determined. The calculations performed by using the suggested technique have shown that the direct method of sub-criteria weight determination yields the results demonstrating the lowest deviation from the average value. Then goes the ranking approach followed by the indirect method and pair wise comparison technique.