ABOUT FORGNOSIS

FORGNOSIS characteristic features:

• A Delphi survey support system with a rich variety of survey designs, from simple multi-round pools to forecasting analytics.
• A flexible survey system that offers a variety of question and/or statement types, as well as a user-friendly interface to reply or verify them. The questions can have various forms: binary ("yes/no"), qualitative - Likert-scale-valued, define probabilities of future events or other quantitative variables.
• Advanced statistical analysis, including reply clustering, outlier detection and management, Gaussian mixture analytics.
• Trust and uncertainty management with sophisticated expert trust and credibility management system and fuzzy-random variables to combine credibility with statistical indicators.
• Customers contracting the use of the application in SaaS mode simultaneously benefit from the knowledge and experience of a consultancy organization that provides an ICT platform and access to its expert pool.
• Forgnosis supports several methodological Delphi variants: decision Delphi, policy Delphi, wideband Delphi.
• The technical organization of the survey encompasses classical multi-round, persistent multi-round, real-time Delphi as well as their combinations. Each one is endowed with a set of dedicated statistical procedures
• The Delphi survey results may yield both quantitative as well as substantial recommendations. They may further be used to determine trends, build scenarios and plan technological investment.
• The survey support system is offered in SaaS mode, with some PaaS functionalities. The latter include a variety of programming tools for designing the organization’s customized survey.

The entire Delphi-based problem solving, including advisory assistance in formulating the problems (statements or questions) presented to experts, the recruitment of experts, monitoring of the application performance, and the final statistical and descriptive analysis of results, as well as development of final rankings, is provided as an advanced research service. The client does not need to handle the organizational aspects the survey, which are fully managed by the application, and specialized know-how is not required. Strict statistical and uncertainty handling methods yield consistent recommendations to the survey stakeholders.

AN INTRODUCTION TO THE DELPHI SURVEY METHOD

The expert survey method often referred to as Delphi analysis was developed at Rand Corporation [link] in the 1950s. Since then, it has been widely used in technological, economical, and social foresight, as well as in decision support processes, especially in the context of choice problems with multiple conflicting criteria. The essence of the online Delphi method is multiple round interviews of an expert group who answer questions in a structured e-survey and verify hypotheses called Delphi statements or questions. These can be defined by the client contracting the study or the core experts employed by the Delphi supplier, usually a consulting or research institution. Delphi studies are often interdisciplinary. For example, in technological foresight, economic, environmental and social conditions are also examined. Therefore, the surveys are usually subdivided into separate research areas, hereinafter referred as "survey sections". Usually, each survey section corresponds to an appropriate thematic panel of experts.

A standard Delphi survey questionnaire comprises of 20-200 questions subdivided into thematic sections. Each section usually contains 10-30 questions, which cover technological, economic and sociological problems or hypotheses. Each question is associated with a specific trend, event, technological or investment priority, market expansion direction, etc. Respondents can select sections or questions which correspond to their particular area of expertise. If an expert is not able to reply a question, it is possible to select the option "no opinion" and move on to another question. A characteristic feature of the Delphi method is its multi-round knowledge elicitation process, in most cases two or three rounds. Subsequent rounds contain the same, modified or extended questions, based on the results of the previous round. The aims of this procedure are listed below:

• Verification of results obtained in the previous rounds by making them available to experts responding to the same or similar questions. This is designed to facilitate a response mainly to questions which were ambiguously answered. In the case of questions with quantitative responses, this procedure aims to reduce standard deviations of previous-round replies.
• Clarification of replies with detailed questions that refer to particular aspects of previous-round questions, by asking for justification of particular responses, and showing the causes of identified trends or events.
• Including questions that complete or update the question set used during the previous round.

The credibility of subsequent rounds can be increased by filtering the group of experts so that the least reliable respondents do not continue to participate in the survey. After the final round, a panel discussion is often organized to validate the results.

The overall survey process can be greatly facilitated by using an interactive application to provide replies. Computer supported Delphi appeared relatively early in the digital era, in the 1970s. Chapter 7 in [link] . With the development of the Internet, email and web-based surveys almost completely replaced traditional paperwork. Due to the increasingly complex structure of surveys, including contingent questions, filling in a questionnaire sent as a file would significantly complicate further statistical analysis of the results. Therefore, specialized survey applications should be used.

Further information on the Delphi research method is included in the classic monograph edited by Linstone and Turoff [link], while information about applications and implementations of the online surveys can be found e.g. in [link] or [link]. The implementations of foresight support systems, including Delphi surveys, are discussed in a series of articles, cf. e.g. [link],[link] and [link]. For a discussion of the role of the Delphi method in decision support and forecasting, the reader can refer to the publications listed here

DELPHI VARIANTS OFFERED BY FORGNOSIS

The multi-round survey (Forgnosis^TM) provides a sophisticated, universal, yet affordable solution to any organization seeking specialized expertise on a given topic. It is offered as Software as a Service (SaaS), SaaS with consultancy support, and also in Platform as a Service (PaaS) mode due to its advanced survey design capabilities. Other features include a novel trust, credibility and competence coefficient management system, facilitating the achievement of a consensus in an efficient way, clustering replies conforming to the statistical analysis of reply distributions, concerning their unimodality, clustering with k-means and Gaussian mixture identification algorithms [link], data imputation, correlation and causal analysis of replies to different questions. Its web implementation enables decision support, both directly, through Delphi questions concerning preferences, and indirectly, by building a decision model based on forecasts and scenarios derived from the Delphi survey.

Standard simple surveys. The online applications supporting the Delphi exercises are most often simple survey software adapted to Delphi needs. They inherit a narrow scope of question types, most often confined to multiple-choice questions with Likert scales [link] and single numerical entries, rudimentary descriptive statistical analysis, and no machine learning mechanisms. Only a few Delphi-dedicated applications offering adequate statistical analysis are available on the market. Typical simple survey software offer no inter-round and a scarce multi-round data management. Forgnosis support simple surveys, but offers an adequate statistical analysis and trustworthy recommendations to the customer.

The development of online Delphi support systems made possible creating a variant of the method termed "real-time Delphi". This variant allows experts to enter an unlimited number of updates to their opinions, while the interim results of the survey can be seen immediately after an entry is made [link],[link]. If the replies are based on different prior information, this mode may affect their independence and statistical significance.

The "decision Delphi" is a survey variant that corresponds most strictly to the needs of organizations that look for expert knowledge concerning specific technological, market or other business problems. This type of survey also fits the needs of medical research on future therapy, epidemics, health care issues, etc. Its characteristic feature is the participation of the client’s staff or the decision makers themselves. The cloud-based application Forgnosis^TM is a modern implementation of a decision Delphi endowed with sophisticated analytic features.

The Wideband Delphi estimation method is the Wideband variant of the Delphi Method. This method is a consensus-based technique for estimating effort, developed as a forecasting tool in the 1950-1970 (Barry Boehm, John A. Farquhar and developed by Neil Potter and Mary Sakry). Compared to the Delphi Method, the Wideband Delphi Technique involved greater interaction and more communication between the participants.
In Wideband Delphi the estimation team comprise the project manager who selects a moderator, experts, and representatives from the development team, constituting a 3-7 member team. There are two meetings: kickoff meeting and estimation meeting. During the kickoff meeting the estimation team creates a work breakdown structure (WBS) and discusses assumptions. After the meeting, each team member creates an effort estimate for each task. The second meeting is the estimation session, in which the team revises the estimates as a group and achieves consensus. After the estimation session, the project manager summarizes the results and reviews them with the team, at which point they are ready to be used as the basis for planning the project.
Wideband Delphi method – steps:

• The project manager selects the estimation team and a moderator. The team should consist of 3 to 7 project team members. The team should include representatives from every engineering group that will be involved in the development of the work product being estimated.
• At the kickoff meeting the moderator prepares the team and leads a discussion to brainstorm assumptions, generate a WBS and decide on the units of estimation.
• After the kickoff meeting, each team member individually generates the initial estimates for each task in the WBS, documenting any changes to the WBS and missing assumptions.
• The moderator leads the team through a series of iterative steps to gain consensus on the estimates. At the start of the iteration, the moderator charts the estimates on the whiteboard so the estimators can see the range of estimates. The team resolves issues and revises estimates without revealing specific numbers. The cycle repeats until either no estimator wants to change his or her estimate or the estimators agree that the range is acceptable.
• The project manager works with the team to collect the estimates from the team members at the end of the meeting and compiles the final task list, estimates and assumptions.
• Review results.

The Wideband Delphi has been adapted across many industries to estimate many kinds of tasks, ranging from statistical data collection results to sales and marketing forecasts.

Trend-discovery Delphi – Unlike extrapolation-based forecasting, this technique is based on specifying future values of a forecasted variable, or probabilities of occurrence of a future event at different future time horizons. Fused information from survey participants makes possible to estimate future trend values with random-fuzzy variables, or odds related to a future event, the latter with a logistic regression. The method makes possible to estimate the year of occurrence of a future event at a given certainty level, and its confidence interval.

SAMPLE IMPLEMENTATIONS