ANALYSIS OF SURVEY RESULTS
A detailed statistical analysis of the survey results is one of the fundamental computational features of the application. Its scope can be defined either by the consultant (in SaaS mode) or by the sponsor (in PaaS mode). However, the results should in any case be pre-assessed by the survey facilitator before their discussion is held by thematic expert panels. The analytical methods implemented in the survey management system are related mainly to descriptive statistics. They include an estimation of expected values, standard deviations and semi-deviations, as well as calculations of medians, quartiles, and quintiles. In its basic version the application is able to calculate any standard or weighted quantiles for a quantitative or quantified question.
The need to use weighted descriptors results from the assignment of weighting factors φij∈[0,1] to the experts’ responses during the survey. They are first determined based on the competence assessment at experts’ registration, then updated based on and the evaluation of the expert’s participation in earlier Delphi surveys or rounds. These factors can be used to optimize the results of advanced statistical analysis. Specifically, they can improve reply clustering, missing value imputation and correlation analysis. Updates of coefficients φij may be performed after each survey round or after a final assessment of the quality of group decision support resulting from the entire study.
Expert trust and credibility analysis allows the survey facilitator to identify those replies that despite their extreme views may best reflect the actual state. This conforms to the principle that the main aim of Delphi research is to detect events and trends that are unexpected, and characterized by values of their parameters which are often far from the average and may be articulated by a relatively small group of knowledgeable experts. Secondly, the weights can reduce the influence of incidental mistakes on the results of the entire study [link].
In order to properly use the information obtained during a Delphi survey, one or more of the following standard methods of enhancing reply processing offered by the here presented cloud application are recommended:
- Verification of the quantitative responses supervised by a survey facilitator or other experts. An acceptable form of verification is only the correction of obvious mistakes made by respondents, such as e.g. placing 1 000 000 instead of 1 in response to a question where the requested reply is in million units.
- Supervised verification of comments, which may result in the removal of an entire response in the cases where the comment or justification is clearly either unrelated to the subject of study or is found to have violated rules of the study or other rules.
- Semi-supervised assignment of appropriate numerical values to particular qualitative or descriptive replies, based on the extracted rules. Numerical values can be assigned automatically to Likert-scale replies making it possible to calculate statistical moments or quantiles weighted by credibility coefficients and returning to the qualitative assessments by finding the descriptor closest to the calculated numerical value.
- Complex quantitative indicators may be constructed, aggregating replies to a collection of questions related to the same or similar topics. Based on the qualitative value conversion presented above, the system enables a fusion of replies given in symbolic and numerical forms and their further statistical analysis.
- Statistical analysis and the identification of logistic trends is offered for questions that elicit probabilities of occurrence of a certain event in the future, for at least four forecasting horizons. The experts respond to questions of the type "What is the probability that event X occurs in the year 2020, 2025 or until 2030 at the latest? " If the logistic regression function is significant, this analysis also makes it possible to determine the expected year of occurrence of an event described by the estimated probability distribution.
The quantitative results of each round are generated in a collective form by the survey management system. An example of such an analysis is presented in Fig. below. The details of the calculations are described in Chapter 8 of [link]. All factors are calculated with reply certainty factors combined with trust coefficients and taken as weights (lower part of the screenshot) as well as without weights (upper part).
CONSENSUS ANALYSIS
In addition to the aforementioned basic factors calculation, the analysis of results includes standard statistical tests and consensus-related properties of the set of responses:
- Standard semideviations σ+ and σ- and their sum, σ+/-.
- Interquartile range (IQR), i.e. the difference between the 3rd and 1st quartiles.
- Interquintile range (IQVR, the difference between 4th and 1st quintiles).
- Number of reply clusters for each question based on Hartigans’ unimodality test of reply distribution [link].
As stated above, the survey support system can calculate a novel measure of opinion consistency, stronger than IQR, termed the interquintile range. As the difference between the fourth and first quintile, it quantifies the consistence of responses included between the 20th and 80th percentiles. By a certain analogy to the famous Pareto principle, this indicator is also termed "Pareto range"
The simplest way to determine the number of clusters consists of an analysis of reply histograms and combining it with the results of unimodality tests. This method can be used even when the number of replies is lower than 30. Other clustering methods, such as k-means, can be used as well when the number of replies exceeds 30. In addition, the Shapiro-Wilk normality test of the distribution of responses [link] plays an auxiliary role in determining when it is necessary to use Gaussian-mixture-based clustering [link].
To assess whether a consensus has been achieved for a given quantitative question q, the IQR, IQVR, and σ+/-,
are divided by the length of the interval [rq,max,rq,min], rq, which reflects
the span of all responses to q after eliminating outliers. This interval will be termed reasonability range for
replies to q. The rq may also be defined independently on the actual responses in the survey.
For example, if the respondents estimate probabilities, it may be predefined as 1. Dividing a selected consensus measure by
rq, yields the consensus ratio, to be compared after a j-th round to a certain acceptability threshold
γq in the following way
The value of γq may be common for all questions in the survey, in a section, or for a group of
questions analyzed jointly. Observe that for a given γ, the condition (1) is obviously stronger than the
condition IQR(s)/rq ≤γ. No such rule can be derived for the indicators IQR and σ+/-, however.
The selection of an indicator and a threshold to assess consensus achievement strictly depends on the type of questions and
information provided by the experts in the survey. Usually, it should be determined empirically.
According to the Delphi background [link], [link],[link], the lack of consensus cannot be identified with a failure of the exercise.
Nowadays, it is commonly accepted,[link], that reaching a consensus survey is not the main objective. Instead, the
appearance of two or more clusters in a set of responses to a Delphi question indicates that the experts used various sources
of information concerned with the topic in question. If the ex-ante credibility of these information sources is unknown, it
prevents us from assessing which group of responses is more trustworthy. In this case FORGNOSIS can calculate the sum of
weights for each cluster of replies and assume that its overall credibility is proportional to that sum.
INTRA-ROUND CONVERGENCE
A consistency condition of type (1) can also be used as a measure of survey inter-round convergence. This is preserved if the consensus indicators decrease in subsequent rounds. This process does not need to be monotonic, i.e.
where Y is a given consensus indicator (IQR, IQVR, or σ+/-) and s and t are round
indices. The consensus is achieved if
The intra-round convergence criteria (2)-(3) can also be applied to each of the reply clusters separately. In this manner, the
hypothetical, most reliable estimate of the value of the studied variable might be the median response or average taken from the
selected cluster. These values can then be used to build development scenarios for the technology, product or market under study, cf. [link].
HOW TO USE SURVEY RESULTS
The expert replies to Delphi questionnaires may be used in many different ways. They may serve to identify elements (subsystems, variables, structural coefficients) of socio-econometric and techno-econometric models [link] or to define characteristics of future decision problems included in anticipatory models. Delphi exercises can help us to discover, confirm or exclude causal relationships between events, trends, and variables. They can also be used to
- create rankings of technologies,
- technological trajectories and
- technology development projects.
- forecast financial factors, such as the average profitability of a user-defined ICT business model that may depend on sales volume and other factors related directly to the market, as well as on the implementation of new technologies.
- generate market forecasts such as the prices of technologies, products and services, sales volumes, as well as macroeconomic variables.
Future technological trends that should be considered when elaborating on technological strategies can be detected with the Delphi studies with appropriately formulated questions. Therefore, an analysis of responses to a Delphi survey can serve as an important decision support tool, which can contribute to increasing the competitiveness of enterprises that use survey findings in their strategic planning.
The Delphi survey results may also become significant components of an enterprise knowledge base that usually stores a time series with market data such as prices, sales and production volumes, as well as technological parameters of competitors’ products. It may also contain rules that determine ways of processing its overall information contents.
DELPHI-BASED INFORMATION FUSION
A Delphi survey can be either an independent way of knowledge acquisition about the future or a complementary method of acquiring knowledge for other future studies. The latter may include biblio-, webo-, and patentometric trend elicitation and analysis, expert panels and workshops, product and technology inventories, etc.
Moreover, experts participating in technological Delphi surveys supported by FORGNOSIS may agree to express their views and justify particular replies during panel meetings with the stakeholders, as an add-on service ensuring detailed clarifications and provision of additional information.
This is why Forgnosis can be particularly useful for research project teams planning to commercialize their results as well as for innovative companies assigning funds to technological development projects. The latter can actually be the later stages of commercialized research projects.