Solution performance evaluation 24

The overall goal of any solution approach is to find a well-performing and feasible solution. However, there is no fixed definition of well-performing, but general algorithm performance measurements can be used in order to express the overall meaning of it. For that matter the overall evaluation criteria of Cordeau, Gendreau, Laporte, Potvin, and Semet (2002) are often used. They set four criteria to evaluate solution algorithms: accuracy, speed, simplicity and flexibility. Within these four criteria the abilities for software transferability and end-user adaption are also taken into account. These two aspects are often neglected in other evaluation methods. In order to evaluate an algorithm, the evaluation criteria are scored using a five-point scale (i.e., from very low to very high). The main benefit of this approach is that is can be used analysing all type of solution methodologies. This section describes the four evaluation criteria in more detail.

3.2.1. Accuracy:

Accuracy measures the degree to which the heuristic solution value deviates from the optimal value and is also related to consistency. Users generally prefer a heuristic that performs well given multiple occasions, rather than a heuristic whose performance fluctuates a lot (i.e., inconsistency). In addition to this, Cordeau et al. (2002) indicate that users prefer an algorithm that produces a good solution at an early state and increase the quality throughout the execution process, compared to an algorithm that requires more computational time and only returns a final answer.

The accuracy of a solution methodology is related to case specific performance measurements. These case specific performance measures represent the overall performance (i.e., efficiency) of a solution methodology. Efficiency relates to the use of the lowest amount of input in creating any given output (i.e., goals). How to measure the overall performance should clearly be defined at front. This in order to make sure that each reader has the same understanding. A good example for this is ‘workload’. The overall workload can be defined as the average number of events that is scheduled on a day, or as the percentage of time resources are not idle. In order to make sure that accuracy is a good directive for the quality of an algorithm, the clear definition (and understanding) of these case specific performance measures and their scale is important.

As the input and output differs per scheduling process, there is no single, specified objective that describes efficient scheduling. However, there is a common number of performance measurements that are used more often. Examples of such performance measurements are for example (Ramasesh, 1990):

I. Workload (average jobs per day) II. Mean and variance resource idle time

IV. Mean tardiness (+ number of jobs tardy) V. Maximum lateness

A typical objective when striving for efficiency is to minimize the weighted combination of costs (i.e., based on a numerical scale or real costs) (Bard & Purnomo, 2005). This is especially useful when multiple characteristics must be considered. This method assigns a scalar value to each event (in combination with a resource and/or time) and schedules based on these values. The scalar values assigned to the different (costs) components are company specific, and so it can easily be customized. In order to use such a performance measurement in line with the method of Cordeau et al., the scalar value of the performance measure must be converted to a value on the evaluation scale (i.e., 5 point scale).

3.2.2. Speed

Speed relates to the computational speed of the algorithm and is also referred to as algorithm running time. The importance of speed depends on the required level of accuracy. When dealing with real-life problems the algorithm must be applied instantaneously in relation to changes. This criterion is very important in relation to solving the research problem.

The algorithm running time is often denoted with the Landau-notation, also referred to as big-O notation. This notation specifies a bound for the algorithm’s performance. The expressed bound is an upper bound, as it gives the algorithm’s worst performance. The actual performance for a specific input may be lower than the upper bound, but it never exceeds the upper bound. If the worst-case running time is expected to be larger than polynomial time (based on the number of calculation steps) it is hard to find the optimal solution within acceptable running times. The number of calculation steps that is can be performed in polynomial time equals O(nk_{) for some nonnegative integer k, where n is the complexity} of the input.

The algorithm running time depends on the problem size, the software compiler, and the computer used. The software compiler and computer are assumed to be fixed, and so those cannot be affected by changing the solution algorithm. The problem size is defined by the number of variables and number of symbols needed to express the problem (i.e., combination and comparison of all variables and constraints that are required in order to form a decision).

3.2.3. Simplicity

Simplicity relates to the difficulty of understanding the reasoning of the algorithm. Algorithms that are difficult to understand due to any reason, such as containing too many parameters, are unlikely to be used. It is unrealistic to assume users understand all details of (scientific based) algorithms, but based on process knowledge the user should understand most basic solution steps.

In relation to scheduling problem, a minimum level of complexity is expected in order to capture the core of the problem and to return good results, but a simple code stand a better chance of being adapted. Therefore complexity and simplicity should be carefully balanced.

3.2.4. Flexibility

The flexibility of an algorithm relates to the degree in which the algorithm is able to handle with various side constraints, encountered in real-life setting, while still returning a feasible solution. General elements that influence the scheduling process are time, costs and required resources. As the scheduling engine is desired to be applied in multiple businesses, the scheduling decisions made by the scheduling

engine must be easily structured and so variables must be easily adaptable to customer settings. This implies that the boundaries of the automated scheduling algorithm must be flexible.

The flexibility of a solution is influenced by the constraints. Constraints are rules set for scheduling and limit the options. Constraints can relate to multiple aspects of the problem: it can be associated with a resource or an event, but it can also be related to resource allocation or time bounds. The identification of constraints is essential for the result of a solution approach while it affects the feasibility of a solution. This multitude of aspects for constraints makes the scheduling task challenging and difficult. Elements that are recognized as influences on the scheduling decisions are (Pinedo M. , 2005):

I. Time related issues; related to the release date, due date and duration of an activity, and available time-windows from both resource and activity

II. Capacity issues; related to the availability of resources

III. Resource issues; related to the configurations (e.g., capabilities) of resources IV. Tooling and material issues; related to the required assets in order to perform a job

V. Workforce scheduling issues; related to the time schedule of individual employees VI. Geographical issues; related to the operational area of a resource

VII. Costs related issues; for example related to lateness or denying customers

These four criteria are used to evaluate the performance of any solution approach. The best method scores well on all criteria, but often a compromise is necessary. In Chapter 1 it is said that the scheduling engine should provide good results. In terms of accuracy this implies that a certain threshold must be achieved.Chapter 1 also indicated that a solution must be created within reasonable time. This is due to the fact that we are dealing with real life problems and therefore a solution must be given (almost) instantaneously. Although it is not specified what is reasonable, we assume that a running time in second or maximum a couple of minutes is acceptable. For simplicity and flexibility no requirements are set in Chapter 1.