Application of methods of organizing complex
expert examinations for increasing reliability
and failure robustness of information systems
and complexes
aMarina Bolsunovskaiya 1, Aleksandra Loginova1*, and Svetlana Shirokova2
1 Peter the Great St. Petersburg Polytechnic University, Institute of Computer Sciences and Technologies, 195251, Politeсhnicheskaya st., 29, Russian Federation
2 Peter the Great St. Petersburg Polytechnic University, Institute of Industrial Management, Economics and Trade, 195251, Politeсhnicheskaya st., 29, Russian Federation
Abstract. The article provides a review of approaches and methods used by Russian and foreign authors for solving tasks of increasing reliability and failure robustness of information systems (IS) and hardware and software complexes. The methods of organizing complex expert examinations (МОСEE) are described and the possibility and necessity of
МОСEE combined application for analyzing reliability and failure robustness of IS, their components and hardware and software complexes are proven. The following methods and models were used to perform a comprehensive assessment: PATTERN methodology, G.S. Pospelov’s
method of solving matrices, models based on the information approach of
А. А. Denisov. A number of modifications of the above МОСEE is offered; methodology of system analysis for solving a task of assessing reliability of information system and its components has been developed; a conceptual model of methodology of developing new solutions in the field of IS and IT has been built.
1 Introduction
One of the most important characteristics of information systems, information complexes and its components are the reliability and failure robustness. A problem of increasing reliability and failure robustness has been solving for IS as a whole and for its particular components, such as software (SW), transmission medium, electrical support systems, data storage systems (DSS).
Special techniques and methods are used in up-to-date information systems and complexes for providing reliability and failure robustness, for instance, full or partial
a The research is carried out with the financial support of the Ministry of Education and Science of
Russia within the framework of the Federal Program “Research and Development in Priority Areas
for the Development of the Russian Science and Technology Complex for 2014-2020”. Subsidy
backup of components, absence of single-point failure, etc. Besides, the devices of system status monitoring are used, including smart systems of monitoring and diagnostics of deficiencies. The development of suchlike diagnostic systems requires engagement of formalized methods.
The following groups of models are used when selecting mathematical tool for investigating systems failure robustness and reliability: Markovian chains [1-4]; models of queuing systems; models based on the graph theory [1, 5, 6]; methods of analysis and statistical estimation of distributions, theory of random labeled point processes, mathematical models of degradation failures [1-3]; differential equations [7]; logical networks, such as failure trees and functional integrity diagrams; logical equations [7]; models of multi-agent systems [8], etc. Based on the performed analysis one can come to a conclusion that presently there is no integrated solution in the field of providing DSS failure robustness, there are only individual solutions pertaining to the above sections. Thus, it should be recognized that a research problem of developing procedure of comprehensive evaluation of reliability and failure robustness of DSS as a IS component is actual.
The goal of this study is to develop methods of assessing reliability and failure robustness of DSS. It is offered to use the ideas of system-targeted approach and gradual formalization [9, 10] for building methods of assessing reliability and failure robustness of DSS, including convergence of methods of systems formalized presentation (methods of mathematical description of systems) with capabilities of methods of organizing complex expert examinations (МОСEE).
The important constraints in DSS development are the requirements of the system customer and capabilities of development tool infrastructure, as well as final equipment. It should be pointed out as well that the task of assessing future reliability and failure robustness of the system at the early stages of DSS life cycle, such as stage of designing, is the task featuring high degree of uncertainty. Therefore, МОСEE are used in this study for reliability evaluation of DSS; methods of this group make it possible to perform decision-making procedures with high level of uncertainty.
The work shows models built with the use of PATTERN methodology, G.S. Pospelov’s method of solving matrices, information approach of А. А. Denisov.
2 Approaches and methods
2.1 PATTERN methodology
One of the simplest methods of assessing alternatives is the use of relative-importance values from PATTERN methodology [1, 11, 12].
It is suggested to use PATTERN methodology in this investigation for assessing development project and incorporation of hardware/software complex.
The criteria for assessing complex components and weight coefficients thereof are expertly determined at the first stage:
1.
K
k
1,...,
k
n
k
ii
1
,
n
– set of criteria used for assessing components of a hardware/software complex (such criteria as cost criterion; criterion of compliance with demands of future users; criterion of complementary dependence, etc. are traditionally used).backup of components, absence of single-point failure, etc. Besides, the devices of system status monitoring are used, including smart systems of monitoring and diagnostics of deficiencies. The development of suchlike diagnostic systems requires engagement of formalized methods.
The following groups of models are used when selecting mathematical tool for investigating systems failure robustness and reliability: Markovian chains [1-4]; models of queuing systems; models based on the graph theory [1, 5, 6]; methods of analysis and statistical estimation of distributions, theory of random labeled point processes, mathematical models of degradation failures [1-3]; differential equations [7]; logical networks, such as failure trees and functional integrity diagrams; logical equations [7]; models of multi-agent systems [8], etc. Based on the performed analysis one can come to a conclusion that presently there is no integrated solution in the field of providing DSS failure robustness, there are only individual solutions pertaining to the above sections. Thus, it should be recognized that a research problem of developing procedure of comprehensive evaluation of reliability and failure robustness of DSS as a IS component is actual.
The goal of this study is to develop methods of assessing reliability and failure robustness of DSS. It is offered to use the ideas of system-targeted approach and gradual formalization [9, 10] for building methods of assessing reliability and failure robustness of DSS, including convergence of methods of systems formalized presentation (methods of mathematical description of systems) with capabilities of methods of organizing complex expert examinations (МОСEE).
The important constraints in DSS development are the requirements of the system customer and capabilities of development tool infrastructure, as well as final equipment. It should be pointed out as well that the task of assessing future reliability and failure robustness of the system at the early stages of DSS life cycle, such as stage of designing, is the task featuring high degree of uncertainty. Therefore, МОСEE are used in this study for reliability evaluation of DSS; methods of this group make it possible to perform decision-making procedures with high level of uncertainty.
The work shows models built with the use of PATTERN methodology, G.S. Pospelov’s method of solving matrices, information approach of А. А. Denisov.
2 Approaches and methods
2.1 PATTERN methodology
One of the simplest methods of assessing alternatives is the use of relative-importance values from PATTERN methodology [1, 11, 12].
It is suggested to use PATTERN methodology in this investigation for assessing development project and incorporation of hardware/software complex.
The criteria for assessing complex components and weight coefficients thereof are expertly determined at the first stage:
1.
K
k
1,...,
k
n
k
ii
1
,
n
– set of criteria used for assessing components of a hardware/software complex (such criteria as cost criterion; criterion of compliance with demands of future users; criterion of complementary dependence, etc. are traditionally used).2.
Q
q
1,...,
q
n
q
ii
1
,
n
– set of criteria weight coefficients, for which the following requirement shall be met:
1
1
n
i i
q
(1)3.
C
c
1,...,
c
m
c
jj
1
,
m
– set of components (modules) of a hardware/software complex.4. At the next stage every component will be assessed in accordance with i-criterion: Sij– assessment of relative importance of j-component according to i-criterion.
The relative-importance values Sij for every criterion can be calculated on the basis of
experts’ (or a single expert) appraisals using different methods depending on analysis peculiarities and formed group of experts. The methods of scaling, ranging, pair-wise comparisons, comparison on basis of linguistic scales, etc. can be used among others.
5. Combined value of j-component is calculated using its evaluation by each criterion and criteria weights:
n i
i ij
j
S
q
S
1
. (2)
2.2 G.S. Pospelov’s method of solving matrices (MSM)
The method of solving matrices was offered by the Soviet scientist G. S. Pospelov in 1970-80s and corresponds to a tool of stratified representation of problem under investigation and step-by-step acquisition of assessments. The MSM has been used in the course of building complex production facilities, during implementation of unique capital-intensive projects (e.g., in the field of space exploration), etc. [9, 12]. The method is quite labor-intensive and produces high requirements to qualification of experts. Example of model of DSS assessment based on MSM is detailed in para.3.
2.3 Denisov Information Approach
The fundamentals of information approach to investigating systems were described by
А.А. Denisov in 1970s; models built with the use of information approach elements have been utilized in control of technical systems and in social-and-economical systems management. An important peculiarity of these models consists in the fact that they include values of degree of alternatives compliance with the targets; as well as some models take into account dynamics of changes in the system under investigation; for assessing degree of compliance with targets a logarithmic regulation is used, which simplifies a procedure of processing acquired expert assessments [9].
This work gives examples of using information values of two types:
1) evaluation of alternatives (system elements) in static state with the use of expression:
H
i
q
ilog(
1
p
i')
, i = 1,…, n (3) where, Hi – information value of i-component (alternative); qi– probability of achievinga selected goal with the use of i-component (alternative); p’i – probability of using i
-component (alternative) in the course of implementing a particular goal.
ni i i
p
q
H
1
'
)
1
log(
; (4)2) assessment of the correspondence of alternatives (components) to system goals and probability of implementing different goals taking time factor into account.
3 Basic results
3.1 MSM modification for DSS assessment
It is proposed to use MSM at the first stage in the course of DSS project assessment in the following context (Fig. 1). Meaningfully, three levels presented in Fig. 1 are interpreted with regard to DSS assessment task, as follows:
1.
E
e
1,...,
e
k
e
zz
1
,
k
– level of experts (engineers, project managers on building and incorporating DSS, etc.). The assessments of relative importance are allotted to experts (based on the level of their qualification, degree of involvement in project): α1,α2, …, αz ,…, αk (
z
1
,
k
).2.
K
k
1,...,
k
n
k
ii
1
,
n
– level of criteria; every criterion also receives itsassessment β1, β2, …, βi ,…, βk,
i
1
,
n
(to be calculated according to formula (7)).3.
C
c
1,...,
c
m
c
jj
1
,
m
– level of alternatives (possible decisions on DSSorganization), assessment of relative importance of these alternatives γ1, γ2, …, γj ,…, γm,
(
j
1
,
m
) show, which alternative from the experts’ point of view meets the specified list of criteria to the fullest extent possible.The relations between elements of the neighboring levels are presented in the form of “solving matrix” with the number of columns equal to the number of elements at the upper level and number of lines equal to the number of elements of the lower level. The matrix elements in a general case are the assessments of significance of a link between two particular elements from the neighboring levels.
The algorithms of receiving evaluations for alternative DSS projects can be described in the following way:
1. The assessment of relative importance (“weight”) αz is allotted to every z-expert
(
z
1
,
k
) at every step of working with model. The “weight” of experts is standardized:1
1
k
z z
. (5)2. Every z-expert (
z
1
,
k
) at every step sets the values of assessing relations qiz(
i
1
,
n
,z
1
,
k
), which reflect the ability of z-expert to perform assessment of alternatives by i-criterion (if qiz = 0, then z-expert does not fulfill assessment by i-criterion).
ni i i
p
q
H
1 ')
1
log(
; (4)2) assessment of the correspondence of alternatives (components) to system goals and probability of implementing different goals taking time factor into account.
3 Basic results
3.1 MSM modification for DSS assessment
It is proposed to use MSM at the first stage in the course of DSS project assessment in the following context (Fig. 1). Meaningfully, three levels presented in Fig. 1 are interpreted with regard to DSS assessment task, as follows:
1.
E
e
1,...,
e
k
e
zz
1
,
k
– level of experts (engineers, project managers on building and incorporating DSS, etc.). The assessments of relative importance are allotted to experts (based on the level of their qualification, degree of involvement in project): α1,α2, …, αz ,…, αk (
z
1
,
k
).2.
K
k
1,...,
k
n
k
ii
1
,
n
– level of criteria; every criterion also receives its assessment β1, β2, …, βi ,…, βk,i
1
,
n
(to be calculated according to formula (7)).3.
C
c
1,...,
c
m
c
jj
1
,
m
– level of alternatives (possible decisions on DSS organization), assessment of relative importance of these alternatives γ1, γ2, …, γj ,…, γm, (j
1
,
m
) show, which alternative from the experts’ point of view meets the specified listof criteria to the fullest extent possible.
The relations between elements of the neighboring levels are presented in the form of
“solving matrix” with the number of columns equal to the number of elements at the upper
level and number of lines equal to the number of elements of the lower level. The matrix elements in a general case are the assessments of significance of a link between two particular elements from the neighboring levels.
The algorithms of receiving evaluations for alternative DSS projects can be described in the following way:
1. The assessment of relative importance (“weight”) αz is allotted to every z-expert (
z
1
,
k
) at every step of working with model. The “weight” of experts is standardized:1
1
k z z
. (5)2. Every z-expert (
z
1
,
k
) at every step sets the values of assessing relations qiz (i
1
,
n
,z
1
,
k
), which reflect the ability of z-expert to perform assessment of alternatives by i-criterion (if qiz = 0, then z-expert does not fulfill assessment by i-criterion).The requirements of standardization are to be met for every z-expert:
1
1
n i izq
. (6)3. Assessments of criteria βi (
i
1
,
n
) are calculated according to formula:,
1
kz z iz
i
q
i
1
,
n
,
(7)1
e
...
...
...
...
1
k
k
ik
n
...
...
...
...
1
с
с
jс
m
j
...
...
e
z...
...
e
k [image:5.482.76.428.87.347.2]z
i
izq
jiq
Fig. 1. Block diagram of model of calculating assessments of relative importance based on “solving
matrices” method.
4. Assessments of relations between elements of the second and the third levels of the model q’ji (
i
1
,
n
,j
1
,
m
) reflect a degree, in which the j-alternative satisfies i -criterion. If q’ji = 0, then i-criterion is not met for j-alternative. The following standardization requirements shall be met for every i-criterion:1
'
1
m j jiq
. (8)5. Assessments of relative importance of alternatives will be calculated at the last stage of model handling (possible decisions on organization of DSS):
,
'
1
ni i ji
j
q
j
1
,
m
.
(9)3.2 Building model of DSS assessment with the use of information approach
It is offered to build a model based on information approach of А. А. Denisov at the second stage for assessing variants of DSS building. Fig.2 shows an example illustrating the task of assessing variants of DSS implementation taking into account customer’s requirements and
In order to assess the variants of DSS organization, the quantitative criteria are traditionally used. The model based on informational assessments of А. А. Denisov makes it possible to calculate a combined assessment of alternatives conjoining quantitative and qualitative criteria. In case of assessment by qualitative criteria, a degree pi' of influence of
i-variant of DSS project (or of inclusion of i-component (combination of components) into DSS) on implementation of requirements to IS as a whole (0<pi'<1) is determined. These
assessments will be converted into assessment of a potential of the corresponding project variant (corresponding DSS component) for convenience of further processing:
Hri = qi log(1 pi'), (10) where, pi degree of influence of i-variant of DSS on reaching the goals (requirements) of
the customer; qi probability of selecting this variant.
[image:6.482.56.424.221.416.2]
Fig. 2. Model of assessing variants of DSS implementation taking into account customer’s
requirements and capabilities of development organization.
Designations: T1,…, Tl – customer’s requirements to DSS (quantitative and qualitative); V1,…, Vz –
variants of DSS implementation, B1,…, Bw– DSS components
The comprehensive evaluation of results Hri includes both assessment of DSS variant
acquired by means of assessing degree of influence on implementation of qualitative criteria, and the technical characteristics thereof corrected to informational values by means of calculating relative assessments pxi.It is possible to use natural units of measurement
together with cost units for assessing expenditures (e.g., labor-content of development of this or the other DSS component, necessary material costs, etc.), which get converted later into relative assessments pzi and Hzi comparable with assessment of Hri results.
Thus, the efficiency of every variant of DSS project can be expressed in the following way:
Эвi = Hri / Hzi. (11)
4 Directions of Further Investigations
B1 B2 B3 B4B5 B6 B7 B8
B9 B10 B 11 B12
B1 B2 B3 B4
B5 B6 B7 B8
B9 B10 B11 B12
B1 B2
B3 B4
B5 B6
T1 T2 T3 T4 T5 … Tj … Tl
V1 V2 V3 V4 V5 … V6 V7 V8 V9 … Vz
DSS2
In order to assess the variants of DSS organization, the quantitative criteria are traditionally used. The model based on informational assessments of А. А. Denisov makes it possible to calculate a combined assessment of alternatives conjoining quantitative and qualitative criteria. In case of assessment by qualitative criteria, a degree pi' of influence of
i-variant of DSS project (or of inclusion of i-component (combination of components) into DSS) on implementation of requirements to IS as a whole (0<pi'<1) is determined. These
assessments will be converted into assessment of a potential of the corresponding project variant (corresponding DSS component) for convenience of further processing:
Hri = qi log(1 pi'), (10) where, pi degree of influence of i-variant of DSS on reaching the goals (requirements) of
the customer; qi probability of selecting this variant.
Fig. 2. Model of assessing variants of DSS implementation taking into account customer’s
requirements and capabilities of development organization.
Designations: T1,…, Tl – customer’s requirements to DSS (quantitative and qualitative); V1,…, Vz –
variants of DSS implementation, B1,…, Bw– DSS components
The comprehensive evaluation of results Hri includes both assessment of DSS variant
acquired by means of assessing degree of influence on implementation of qualitative criteria, and the technical characteristics thereof corrected to informational values by means of calculating relative assessments pxi.It is possible to use natural units of measurement
together with cost units for assessing expenditures (e.g., labor-content of development of this or the other DSS component, necessary material costs, etc.), which get converted later into relative assessments pzi and Hzi comparable with assessment of Hri results.
Thus, the efficiency of every variant of DSS project can be expressed in the following way:
Эвi = Hri / Hzi. (11)
4 Directions of Further Investigations
B1 B2 B3 B4B5 B6 B7 B8
B9 B10 B 11 B12
B1 B2 B3 B4
B5 B6 B7 B8
B9 B10 B11 B12
B1 B2
B3 B4
B5 B6
T1 T2 T3 T4 T5 … Tj … Tl
V1 V2 V3 V4 V5 … V6 V7 V8 V9 … Vz
DSS2
DSS1 … DSSi … DSSn
It is possible to expand the proposed approach in the course of managing projects development and implementation, which envisage incorporation of an essential number of homogeneous components (e.g., software utilities) by appending with the models of assessing DSS development projects in dynamics [9]. The authors of this work offer to build a model of the process of DSS implementation as the model of development and incorporation of innovations with the use of the information values based on the А. А. Denisov’s information theory.
The considered approach to assessing DSS projects on the stage of development and implementation can be used as a basis for creating decision-making toolsin managing processes of IS and technical complexes design during development of new IT solutions.
Fig.3 shows a stratified conceptual model of the methods of developing new IT solutions.
Fig. 3. Structure of the conceptual model of methods of designing and developing new IT solutions.
5. Conclusion
The article shows a necessity of using МОСEE in the course of solving tasks of enhancing reliability and failure robustness of a DSS as a component of information systems and complexes. The advantages of МОСEE are determined by the fact that they grant a possibility of fulfilling a decision making process in situations with high uncertainty. At the same time the number of studies where the МОСEE are used for designing and assessing serviceability of IS and information complexes is low at the present. This work offers a number of modifications of certain МОСEE. A methodology of system analysis for solving tasks of assessing reliability of IS and its components has been developed. A conceptual model of a new IТ solutions development methodology has been designed.
References
1. F. Beichelt, P. Franken, Mathematical approach, 392 (1988)
2. V. Ya. Kopp, М. V. Zamoryonov, Yu. Е. Obzherin, М. Yu. Larin, Scientific and technical journal of St. Petersburg Polytechnic University. Information science. Telecommunications. Management, 3 (247) (2016).
Software Hardware
Structural and logical model of object region
Model of system dynamics
Algorithms
IТ solutions
[image:7.482.128.342.209.379.2]3. V. Ya. Kopp, М. V. Zamoryonov, А. А. Selkin, О. V. Filipovich, Journal of up-to-date technologies, 1 (1), (2016)
4. B. Saghaie, R. Ragel, S. Parameswaran, A. Ignjatovic, Proceedings of the 23rd ACM international conference on Great Lakes Symposium on VLSIDOI: 10.1145/2483028.2483051 (2013)
5. I. V. Asharina, А. V. Lobanov, XII All-Russia meeting on management problems ARPMP-2014, 7387-7392 (Institute of management problems named after V.А. Trapeznikov, RAS, 2014)
6. K. Reinshke, I. А. Ushakov, Assessing reliability of systems with the use of graphs, (Radio I Svyaz 1988)
7. I.P. Simakov, P.V.Kholodnykh, Information technologies in management (ITM-2016)
Proceedings of the 9th conference on management problems (2016)
8. S. E. Saradjishvili, А. V. Igumnov, Journal of Bauman Moscow State Technical University, 1 (2014)
9. V. N. Volkova, А. А. Denisov, Theory of systems and system analysis (YURAIT, Moscow, 2017)
10. V. N. Volkova, A. V. Loginova, S. V. Shirokova, E. A. Kozlovskay, Soft Computing and Measurements (SCM), 2016 XIX IEEE International Conference (2016)
11. М. М. Lopukhin, PATTERN method of planning and forecasting scientific works, 160 (1971)
12. Е. М. Smolina, V. N. Volkova, А. А. Pirogova, Technological perspective in the context of Eurasian environment: new markets and points of economical growth.
