ORGANIZATION OF THE SURVEY

SURVEY DESIGN

The survey questions should aim so solve an important problem for the organization which commissioned the study. The particular questions most frequently touch upon:

  • The possible implementation time of a new technology or product.
  • The probability of various events occurring over one or more possible forecasting horizons.
  • Innovativeness, relevance of technologies, events, trends, etc.
  • Limitations and potential barriers or other difficulties that may arise in relation to the implementation of particular technologies.
  • Assessment of available research and development infrastructure.
  • Position of a competitive technology, industry, or product on specific markets or compared to other products, technologies or industry branches.
  • Social and environmental aspects of technology implementation, including their impact on employment.

As mentioned in the previous section, the scope of the survey can be determined by the sponsor, or in cooperation with a consultant, based on results of "Round 0". Survey statements and questions can also be derived in a similar way or supplied by the sponsor or consultant. The survey may consist of thematic sections containing complex statements (or "theses"), i.e. groups of questions concerning the same subject or hypothesis.

PRE-DELPHI TO OPTIMIZE THE SURVEY SCOPE AND MINIMIZE EXPENSES

The Delphi survey is often preceded by an exploratory study conducted in the form of an online questionnaire, which is not part of the main survey process. It is aimed at selecting research issues relevant for the stakeholders commissioning the survey-as-a-service. This initial needs analysis is often called the survey’s "Round 0". The participants can be, for example, the decision makers as well as the senior managerial or technical staff of the organization interested in results of the study. The latter will be termed the sponsor of the survey, while the survey provider will be termed consultant. "Round 0" can be also open to external participants who contribute independent views on the subject of the planned survey. Based on the results of this initial exercise, the sponsor and stakeholders, together with the consultant’s staff assisting the survey process, define statements and questions to be used in the first round of the survey.

BUILDING AN EXPERT TEAM

Recruiting experts is one of the most difficult phases of the Delphi studies, effectively limiting access to this knowledge source for companies other than large corporations seeking solutions to the problems of technological development and market expansion. This barrier can be overcome when using the ForgnosisTM platform as its services are offered with a team of experts recruited from different scientific, industrial and social areas by the platform provider. The thematic expert panel is developed as a part of the service. The panel members are selected from among the consultant’s expert pool. The qualified customer’s staff can also be included into the team if wished so.

All team members should fulfill the following criteria and constraints:
  • The expert skills/knowledge must cover the full scope of the survey.
  • The panel should consist of at least 20 experts; the maximum number depends on funding and on the survey subject. Usually, more experts can take part in medical surveys, and fewer in highly specialized areas such as future of knowledge repositories.
  • The proportion between the sponsor’s staff and independent experts recruited by the consultant should be predefined in the contract and observed while the survey is being conducted.

The survey sponsor can also fully rely on their own experts and perform an analysis of results on their own as well. This mode may be useful when confidential or classified data is being handled during the survey. In this case, the service is a pure cloud-based SaaS. In both cases, a survey facilitator should be assigned. This person is responsible for handling the platform and for the survey running smoothly.

A dedicated web application was developed to ensure the effectiveness of the expert selection and management process. This system effectively manages the consultant’s pool of experts, from registration and verification of expert candidates, through innovative competence management, and intelligent mailing, to experts’ remuneration.

The number of panel experts depends i.a. on the subject matter and scope of the study. Usually, the more specialized the questions, the narrower the group of experts competent enough to give a reply. In engineering sciences, including computer science, it is assumed that the number of responses to each first round question should preferably exceed 20. Lower values are acceptable for subsequent rounds as they may be limited to a subgroup of experts participating in the previous round selected in a certain manner. For example, 25 experts participated in a typical decision-Delphi exercise conducted within the MOVING project [link], which strictly follows the recommended number of 20-30 experts for this survey type. On the other hand, the policy Delphi in the project SCETIST [link], which covered intelligent systems, was carried out on a sample of over 100 respondents.

QUESTION TYPES

After completing the self-assessment form, the experts can start filling in an online questionnaire, where they:

  • verify asked hypotheses (so-called Delphi statements);
  • comment on future events and trends of a subjective character, i.e. not resulting from quantitative models;
  • determine future values of key parameters of given objects (products, markets, technologies, etc.)
  • determine the probabilities of future events or the conditions under which they may occur;
  • make quantitative forecasts of the future development of a given area, usually within a 15-50 year horizon;
  • specify the time horizon in which particular technological, scientific, market or social events may occur;
  • identify new products and technologies, their barriers, growth factors and impacts;
  • identify the consequences of the decisions made and define the reference points for the decisions.
  • Questions of any type can be accompanied by a request to specify the "certainty level of the reply" from a predefined list of values.

FLEXIBLE MULTI-ROUND INTERACTION

The ForgnosisTM Delphi Support System does not require any preliminary limitation of the round number or even their synchronization. Its characteristic feature is that new respondents may join the exercise at any time during the survey period, even if other experts have already begun participating in the second or a later round. The opinion processing scheme is presented in Fig. below.

Flexible Multi-Round Delphi survey
A scheme of expert opinion elicitation in the Flexible Multi-Round Delphi survey. Darker arrows denote expert transitions between rounds, lighter denote information flows.

After each round, the research results are statistically processed and analyzed by the survey system to detect possible gaps and other issues. The consultant and/or sponsor determine the consensus conditions for each statement. If a consensus was reached for a given question, there is no need to continue research on this topic and the question is removed from the next round. If on the other hand the survey facilitator identifies new problems that have arisen during the last round, new questions can be included in the next round. Particular attention should be paid to questions where a consensus was not reached and where inter-round convergence was slow. The second and subsequent rounds are generally directed to the respondents who most fully and reliably completed the questionnaire in previous rounds.

THE SURVEY BUSINESS PROCESS

Delphi survey business process
An organization chart of multi-round Delphi study, in which the specialized software available as a service was used.

The above presented multi-round online Delphi survey scheme does not impose any limits on the number of rounds. The subsequent phases of the study, which can be identified with completing the i-th round, are usually summarized at specialized panel discussions or seminars with both experts, sponsors, and research stakeholders. The purpose of panel discussions may also be a final description of the future development scenarios derived from the survey.

SURVEY IMPLEMENTATION FOR RESEARCH PROJECT EXPLOITATION PLANNING

As a Delphi survey case study, we will present the most important elements of the web-based platform used in the EU Horizon 2020 MOVING project (see: www.moving-survey.ipbf.eu). This survey sections are shown in Fig. below.

MOVING Survey sections
An example of the Delphi study on ICT technologies composed of three sections and 96 questions. Percentages indicate the extent to which the selected respondent answered the questions on particular topics.

Delphi survey questions related to specific issues are complex and include structured elements such as check boxes, numerical fields and text boxes which allow the respondent to type descriptive comments and justifications. The design of the survey can be accomplished with a sophisticated administration panel. A screenshot of the panel with sample questions designed for the survey on a digital knowledge platform future is shown in Fig. below.

Questions design
An example of questionnaire design for the MOVING project.

Respondents may indicate relationships between questions, e.g. by pointing out conditions that must be fulfilled to ensure a positive or negative occurrence of an event or trend. The next round respondents have access to a summary analysis of replies to all questions in the previous rounds. Influenced by the summary opinion of other experts, the respondent can maintain his/her opinion or change it.

In the extended SaaS and PaaS modes, the sponsor can use the administrative tools available in the panel to design the overall survey and enter the statements and questions. Standard questions in the MOVING survey have a tabular form, where the respondents enter their replies in the columns corresponding to the specified time intervals. These are interpreted as the final and intermediate forecasting horizons. A separate table is assigned to every statement.

Question number Present state (facultative) State in year R1 State in year R2 State for year R3 Further develop-ment up to year R4 Comments sources, justifica¬tions Degree of certainty of a reply
Reply Reply field Reply field Reply field Reply field Reply field Comment Pick list
The scheme of a simple question consisting of one row.

The questions of the second type may have additional, so called subordinate questions. The third type of questions consists of interrelated groups of questions. These groups usually have a common introduction or share part of a question. Type 3 can be, for example, a group of questions concerned with the structure of a certain market, where each question is related to the forecast of a different market segment. The analysis of replies to subordinate questions should take into account the method used to relate these questions. It is often assumed that the sum of single-expert replies to sub-questions should amount to 100%.

Questionnaire with replies
An example of questionnaire with replies to questions collected in Section I of survey performed within the MOVING project.

Due to the ease of reply quantification, the most popular and frequently used in qualitative questions, is the Likert scale, particularly in its 5-grade version [link].

TRUST AND CREDIBILITY MANAGEMENT

Trust is the key issue for the success of the survey and the overall study and its implementation by the enterprise. This is why this issue has been paid utmost importance when designing the survey support system. Before they begin answering questions, the experts selected to take part in this exercise specify their level of knowledge on the topics included in the different survey sections. They fill out a two-stage form, where their knowledge of a topic is specified e.g. as "expert-researcher", "expert-practitioner", " expert-researcher and practitioner", "I’ve just started studying this topic". They can also provide additional clarifying information. Moreover, the survey facilitator has the possibility of assigning to each individual expert additional competence factors. This external assessment is independent for each survey section.

Self-assessment form
Self-assessment form of experts answering the Delphi survey.

The self- and external assessments may be aggregated, yielding normalized credibility coefficients φij∈[0,1], for each i-th expert and j-th competence area. These are used as weighting factors in the statistical analysis of the replies. An inter-round assessment of anonymized individual replies makes it possible to assign or update the trust or competence coefficients of experts and to eliminate outliers, for all thematic sections separately. The latter features can be efficiently provided in computer-based Delphi systems only.

The results of the previous Delphi survey research with the ForgnosisTM system (e.g. these performed within the SCETIST and MOVING projects) show that taking into account trust, and competence factors improves the overall credibility of survey results and accelerates the arrival of a consensus. Trust and credibility coefficients can also be the criteria for selecting experts for further studies. The selection of respondents with better access to the source information is also made possible.