Feedback of results during an experiment

During an on-going experiment the involvement of the clients is crucial. The analysis of results should not be left solely to the researchers. As researchers, clients, and other relevant competence are present at the test location, there is reason to use this opportunity for analysis based on the multi-disciplinary competences. Often, an experiment is conducted as a series of runs during an exercise. Thus, time is crucial if the results are to be interpreted during an on-going data-collection series during an exercise. This is especially accentuated in an action research approach (see section 3.2), where the results from the initial test runs should be input for further runs. However, the need of feedback during a single case study (see section 3.2) is also present as it might save time and resources invested in the post mission analysis.

7.4.1.1 Case 7A: Demo 05 V

Experimentation is an important feature of the Swedish Armed Forces’ transformation from invasion focus to a flexible net-work based operational defense. Demonstrations and tests of new technology and methods will be used to resolve uncertainties, test technical (and financial) feasibility, and illustrate alternative courses of action. The results will be analyzed and assessed through study activities before deciding on continued development activities. One important sub- program in this process is LedSystM which is responsible for development of command and control (C2) methods. As a part of the demonstrator program, command and control methods and principles will be tested in a series of experiments.

In the exercise Demo 05 V, held in April 2005, the focus of the experimentation was to further develop a procedure for planning under time pressure. The model for the procedure, Planning Under Time Pressure (PUT), is based on a satisficing rationale approach, that is, the purpose of the planning process is to find a good-enough option early in the process and then refine and corroborate it through war gaming. The model is originally developed at the Swedish Defence College and was first introduced in the Swedish armed forces as a tactical planning model in 2003. A full description of the PUT-model is provided by Thunholm (2003, 2005).

More precisely, the aim was to develop procedures for the cooperation between staffs and the distribution of commander’s intent in the context of the PUT model. The experiment was conducted during a series of war-gaming sessions. The test design was based on an action-research approach. The intention was to conduct the analysis according to the following procedure:

1. During, and instantly after, each war-gaming session a set of variables were measured with a number of different instruments, i.e. observers, questionnaires and registration, and classification of radio and data communication. The variables - task knowledge, cooperation, product quality, distribution of commander’s intent, and communication pattern - were used as indicators of organizational performance.

2. Between each war gaming session the result from the measurements were analyzed together with personnel from LedSystM. Participants of the exercise also conducted workshops on the theme “What can be changed in the planning process in order to improve the PUT process?”.

3. Based on results from the steps above decisions were taken together with personnel from LedSystM on what to change in the PUT process during the next war-gaming session. Instruments of measurements could also be changed.

4. During the next war-gaming session the organization was measured using basically the same set of variables. The basic question to be answered was if the changes introduced made any difference. If so, what was the nature of the effect? Refined and supplementary questions and hypotheses might be added to the analysis.

The process described above was implemented with the help of military personnel allocated to the work from LedSystM and is described in more detail below:

a) The initial task was to explain the results from the analysis of data from the first war-gaming session. Results showed a strong negative correlation between the commander of the joint staff and his staff members on the ranking task of the importance of different events that occurred during the session. This result might be interpreted as a failure of the commander to communicate his intent

within his own staff. The personnel from LedSystM used the data base from the digital questionnaires in order to find explanations of the result. It was also possible to trace data from staffs, observers, and personnel for further data collection. The most plausible explanation found was that the result was due to the fact that distribution of information to the subordinate staff had been prioritized. Thus, the commander’s intent had not been formally communicated in the joint staff. Another explanation might also be that officers from different services had different competence and traditions and therefore interpreted the same situation differently.

b) The next step in the procedure was to come up with solutions to overcome the identified problems. In this case, the major problem was judged to be the inconsistent perception of the situation in the joint staff and the absence of communication of intent within the staff. The opinion was that the most significant tool was that each staff should establish a timetable in order to push the planning process forward.

c) Finally, two changes in the experimental design were defined. First, changes of the measurement of distribution of commander’s intent were implemented. The aim was to overcome the problem with the difference in perspective on the mission between services. Instead of just ranking events of primary interest for the joint operational level, the ranking also included lists of events of importance for each of the services. Second, a minor experiment was also conducted between the first and second session in order to compare different principles to formulate a commander’s intent. In this minor experiment twelve officers were divided into two groups. Each individual in the groups acted as a liaison officer receiving a commander’s intent communicated by the commander of the joint staff. One group received commander’s intent expressed in terms of the ”required effect” while the other group received the same commander’s intent expressed in ”required capacity”. Communication between the officers within or between the groups as well as questions to the commander where not allowed. Measurement of distribution of commander’s intent consisted of ranking of importance of a selection of defined events. Each participants were asked to rank the order of a number of injects regarding their threat to mission success. The participants ranking of importance were compared within each staff as well as with the commander (whose ranking were considered to be the correct ranking in accordance with the communicated intent). Result of the data analysis indicated that the group subjected to commander’s intent expressed in ”required effect” showed a higher consistency in ranking of importance relative to the “correct” commander’s ranking.

Results from the intervening experiment were not put forward as a finding to be generalized. Instead, the results were used as a basis for an AAR discussion a few minutes after the “intervening” experiment. The purpose of the discussion was to try to explain the result and to come up with actions to improve transfer of commander’s intent. During the discussion a hypothesis, intended to be tested in the next war gaming session, was formulated: “Commander’s intent should be mediated in terms of “required effects” but the following dialogue should be based on the ”required capacity”. A template to support a dialogue based on this hypothesis was constructed and informally tested in the following session. The possibilities to observe this dialogue were limited due to limited space for observers.