Part two of the blog series "Evaluation of ERP Projects from a Consulting Perspective" will now focus on the development process and conceptual implementation of a measurement tool that can be used to critically evaluate internal consulting projects and make them comparable with one another. In the first part of the blog series , we took a look at the motivation behind such a performance measurement, the associated objectives and the scientific status quo in this area, as well as the thesis on which it is based. Thus, models and criteria could already be named, which are scientifically relevant for a success measurement in the field of ERP and management consulting. Now the focus will be on how these criteria can be linked and made measurable. The presented implementation is by no means an alternative-less and rigid solution for meeting this challenge, but rather shows with a possible way of implementation, which seems practicable according to scientific aspects and can and should be adapted, expanded and critically scrutinized at any time.
To develop a measurement instrument, it is first useful to base it on a construct, framework or model in order to give a basic form to the different levels of observation and criteria. On this basis, adaptations can be made that appear to be purposeful according to the use case - in this case, the adaptation to the evaluation of ERP projects from a consulting perspective.
Due to the discrepancy in scope between the underlying thesis and the blog, the rationale for the choice of decisions in model development will be largely left out in the following and only the implementation and results will be discussed.
Selection and definition of the framework
A paper by Rosemann & Wiese (1999) already justified the adaptability of the so-called Balanced Scorecard (BSC), which is a widely used strategic management tool that can be applied to the evaluation of success in ERP implementations, in the research paper preceding the final paper. Accordingly, this approach is also interesting for evaluating success from the perspective of ERP service providers. Thus, financial, process, and management-related factors, as well as the customer view and learning effects, can already be categorized using the usual four views (financial, customer, process, learning, and development). The proposal to develop the measurement tool based on the BSC as a framework was also accepted and welcomed internally.
The four views of the BSC were retained and described in detail in the context of ERP consulting and for the logical classification of the assigned success factors and dimensions to enable clear differentiation.
The Finance view is evaluated in terms of the financial impact of the project on the company. It is evaluated in relation to other ERP projects to identify whether its influence has a particularly positive or negative impact on the business capability. As is characteristic for services, personnel costs and turnover play decisive roles for the corresponding evaluations.
The Process view deals with the entire consulting and development process during the course of the project. Both factors internal to the consultancy and factors arising from the collaboration with the customer are taken into account. The focus here is primarily on success in the area of management and qualitative framework conditions. This view includes, above all, satisfaction with the consulting service and internal consulting satisfaction in the context of project product success as well as project management success.
The customer view reflects customer satisfaction with the system. An alternative name for the dimension would therefore also be System. While human factors play a role in the Process view, the focus here is on technical implementation. This view therefore primarily includes product satisfaction, including acceptance and use of the product in the context of project product success.
The learning and development view refers to the knowledge gained from the project and the consequences for the further development of the company. This can be financial as well as intellectual or interpersonal in nature.
In reference to the BSC approach of Rosemann and Wiese (2001) and adapted to the view of IT consulting, a four-dimensional construct is thus created as a starting point for the evaluation of ERP projects, which is shown below and with the help of which corresponding questions are to be answered for each area:
Definition of the measurement instrument: basic assumptions and structuring
As a result of the theoretical groundwork, the BSC views are filled with success criteria and differentiated according to aspects relevant for the further development of the measurement instrument.
Based on the literature analysis, it must be stated that although the literature continuously differentiates between success dimensions and the success factors acting as drivers for these, the assignment of the criteria to the two types is not always clear. Success factors from customer view were renounced due to redundancies with the considered success dimensions. Success factors from consulting research, on the other hand, are initially integrated into the model development as additional points of consideration.
Since an evaluation of success should be carried out in principle on the basis of success dimensions and an integration of success factors, if these are recognized as such, is not stringent, a differentiation is made for the model between success in the narrower and in the broader sense. It is to be noted here that with success in the narrow sense the success factors are excluded and with success in the broader sense the success factors are integrated as success dimensions and part of the evaluation. If necessary, the weighting of the success dimensions concerned is adjusted accordingly.
For the further procedure, the success criteria must first be summarized and structured, since many authors use different terms for the same or similar criteria. Subsequently, these are assigned to the appropriate BSC dimension and thus become part of the success assessment for this dimension. For each criterion, an exclusive assignment as a success dimension or success factor is made. In anticipation of the later operationalization and questionnaire derivation for data collection, it is necessary to assign each criterion to an addressee and a view. An addressee is understood here as the party or system to be evaluated with the help of the criteria. For the measurement instrument, the following addressees and abbreviations result for the tabular representations:
- Consultant (B): operational or strategic project participants of the consultancy,
- Client (K): operational or strategic project participants of the customer,
- System (S): the ERP system to be evaluated as the addressee.
A view is understood as the party evaluating the addressee based on the criteria. For the measuring instrument the following views and abbreviations result for the tabular representations:
- Consultant (B): operative or strategic project participant of the consultancy receives the evaluation order via the addressee,
- Client (K): operative or strategic project participant of the client receives the evaluation order via the addressee,
- Joint (G): Consultant and client receive the joint evaluation order in dialog via the addressee.
According to the explanations, the first phase of the model development of the basic design was therefore subject to the following procedure:
The result is the basic construct of the measurement instrument as the basis for the subsequent operationalization.
Definition of the measurement instrument: operationalization and value origin
With the basic construct defined and the modifications considered, a suitable basis for operationalization is given. In addition to the usual selection of suitable indicators, characteristic values and codings for measuring the latent constructs (criteria), this also aims to realize the structural modifications at the level of the criteria. In the first step, therefore, all criteria of the basic construct were first operationalized. As a result, in recurring feedback meetings with the scientific supervisor of the thesis and contacts at manaTec, the mentioned criteria were optimized in a fine-granular way by adding, adjusting, redistributing individual indicators or eliminating them in case of redundancies in content with other indicators.
The indicators for a more detailed description of the influencing factors of the criterion have been primarily taken from the suggestions of the indicated literature, appropriately reduced, translated and appropriately formulated for the application purpose. All indicators serve the evaluation of the respective criterion according to points, whereby a higher score represents a better result. Therefore, only the ordinal scale can be used for all characteristic values of the indicators in order to represent a ranking of given answer options. Care was taken to provide each indicator with a maximum of four and a majority of an even number of answer options in order to force a tendency of the recipient to choose one answer option and to counteract the bias towards medium answer options. Each characteristic expression is assigned a coding that specifies the maximum score of an indicator at the maximum (best possible response by rank) and the defined minimum score of zero at the minimum (worst response by rank). It is significant that zero represents the minimum score for each indicator in order to represent a percentage mapping of success across all indicators and criteria in the definition range of zero to 100 percent. Since the relative success in each criterion is used for accounting, the total number of indicators and thus points per criterion is irrelevant for the success consideration at the dimension level and only the distribution between the indicators is decisive. However, it should be noted that a higher number of indicators can enable more accurate evaluations for the criterion, since, for example, outliers or incorrect evaluations are less significant. However, it is problematic that this is accompanied by a larger number of questions, which can be disadvantageous for such a comprehensive measurement tool. For example, sentiment or attention could decrease with increasing questionnaire size, which can negatively affect data quality contrary to the original intention.
Each indicator of the criterion is indexed for the measurement instrument, assigned to a suitable question, provided with answer options as characteristic values and coded according to these. It is also determined whether the operational level (O), represented by key users and consultants, or the strategic management level is addressed for answering the question. This separate consideration by so-called seniority level is important because not every decision level of the consultancy has insight into every topic area of the project. From the combination of view and level, a clear value origin can then be defined. This corresponds to the target group for answering the respective indicator question, which contributes the actual values to the maximum score in the measurement table for the measurement instrument.
The questions underlying the criteria and indicators as well as the associated characteristic values cannot be presented here due to the scope. However, you can get an insight into this in the last part of the blog series, which also deals with the creation of the questionnaires in Odoo based on the operationalization.
Definition of the measurement instrument: Finalization via weightings
The structure for the formal weighting has been completed by defining the value origins as data sources of the actual values for the calculation of the success indicators, the indicators per criterion, the differentiation of the criteria according to success factors and dimensions, and the assignment to a dimension. The following figure presents the measurement instrument suitably reduced down to the level of the criteria for the sake of readability and to show the influence of the weights.
In addition to the weights, the summed maximum score per criterion is also presented using the extended standard coding. A special feature is the assignment as success criterion i. e. S. (ED) or i. w. S. (EF), which influences whether a criterion is involved or ignored for the desired evaluation. For success in the narrow sense, the success factors are not relevant and G(P₁) is weighted at 100%. For the success in the broad sense, G(P₁) and G(P₂) are balanced to 100%, so that the result of the success factors is included in the result of the process dimension with G(P₂). In the process view, therefore, there is a separation at the level of the success criterion and an additional weighting depending on the desired evaluation. In the other views, the weighted success of the criteria goes directly into the dimension. Concluding from this and as can be seen in the measurement table, there are three weighting levels for the dimensions (G(Dim.)), success criteria (G(EK)) and the criteria (G(ID)). Each level must balance to a weighting of 100% to allow success to be calculated in the range of values from 0 to 100%. This version of the measurement tool can be considered a formal template for use in other IT consulting firms to measure success in ERP projects. In addition to the target values, it describes all the necessary areas that must be defined for weighting and can be used directly in practice.
In order to be able to apply the weightings, it is first necessary to calculate the success at indicator level, which is included in the success calculation of the criterion depending on how it is offset against the other indicators. The maximum values of each indicator are determined by the coding and can be freely chosen by the consulting company. They are the basis of the success in each indicator, which is defined by the associated actual value from the responses of the recipients.
As an accounting variant in this case the sums of the actual values are put into relation with the sums of the maximum values for all indicators of a criterion. Multiplied by 100 %, the result is the percentage share of the maximum value of the criterion, which is referred to as the "success contribution" at this point. The special feature of this way of calculating the indicators per criterion is that the coding corresponds to a weighting at the same time. Alternatively, for example, the contribution to success could be calculated for each indicator and then the arithmetic mean calculated over all indicators of the corresponding criterion. In this case, an additional weighting level would have to be introduced for weighting the indicators, assigning a weight to each indicator.
For the multidimensional, visual evaluation of the project success and the comparison of the dimensional results, different forms of presentation are possible. For example, the network diagram offers a possible presentation variant that concisely illustrates the success in the individual dimensions in particular.
This illustration is only intended to be a teaser of other possible display variants, which will be examined in detail in the last part of the blog series with a view to report and dashboard generation. There you will learn how the model presented here is operationalized and filled with data in practice via Odoo surveys, as well as how this data can be automatically loaded into a report through the connection of Odoo to Power BI and brought together on a common dashboard in combination with further Power BI KPIs.
Are you interested in evaluating your projects internally or are you looking for a Business Intelligence solution? Contact us now nd we will be at your side as a reliable partner!