An alternative approach is made to design a high-throughput distributed processing cluster computer system that utilises a combination of all the processing devices. It uses the workstations, PCs, different types of servers, and small form factor computers as separate, mutually exclusive clusters that can work as intelligent processing devices to facilitate the compute-intensive calculations to be executed within manageable time scales and maximise the processing capacity. The calculation clusters are configured in a way that they can act as a single cluster or as part of a multiple-cluster system depending on how the distributed processing controller manages these clusters. By consolidating different clusters into an intelligent calculation cluster that can be used during out-of-office hours when most the resources are free to be used at full capacity, this is highly useful for batch processing tasks that take many hours to complete in a single server. An adaptive and self-tuning control mechanism is implemented that continuously fine-tunes the cluster performance and is able to function autonomously with varying parameters. The varying parameters can be hardware, software, or business rules. The cluster design approach has various degrees of flexibility due its bespoke-type design; hence, it can be modified with minimum impact to the business. The consolidated calculation cluster configurations are based on the following:
Workstations are configured as virtual machines using VMware software. Noncritical servers are set up as calculation nodes.
Surplus conventional servers are set up as calculation nodes.
Cluster controllers are modified for coordinated process management. SQL servers are configured for distributed data and query processing.
Clusters are designed using adaptive processing for efficient and robust operation.
CPU cores are used as calculation nodes with load balancing techniques.
Historical data are collected for adaptive load balancing and task scheduling. Static and dynamic load balancing algorithms are implemented. Various methods and techniques are implemented to improve the efficiency of the distributed processing.
166
The following tests are performed on consolidated calculation clusters: Various applications are used to monitor hybrid cluster performance.
Various tests are performed, and collected data is used to analyse cluster performance under different load conditions.
CPU core-level process executions using different applications and static and dynamic load balancing.
Control techniques applied to manage the process execution within calculation nodes to avoid process deadlocks.
Virtualisation techniques are used for workstations and servers. Multi-core processing and multi-thread techniques are employed.
Parallel processing is performed in a single workstation using MS-Excel applications.
Virtualised servers are used as dedicated calculation nodes.
Distributed data and query processing are implemented using local SQL database servers and linked SQL servers.
A combination of hardware is consolidated as physical and logical calculation clusters.
The followings are investigated using the consolidated hybrid clusters:
The systems that are currently used have been developed using VB6, VBA, VBScripts SQL server, and batch processing systems. How these applications can be incorporated within the distributed processing system at no extra cost or at minimal cost is investigated.
Implementation of reliable, robust, and efficient distributed processing system for different type of processor-intensive calculations at minimal cost. Currently, these calculations are performed as batch overnight and weekend processes.
Incorporating Disaster Recovery (DR) site hardware and software as part of the overall distributed processing system.
167
The Northwest distributed systems must have certain characteristics, hence, the overall design and development is focused on complying with the following characteristics:
The system should be scalable with a number of calculation nodes and able to add extra calculation clusters with minimum disruption to the business operations.
The system should be robust enough to any node failures of various types and should be able to manage the failure gracefully and the system continues to operate and delivers required critical calculations in the presence of node failures.
The system should be extensible for different types of data and applications that are used within the systems.
The system should have minimal management overheads and manual configurations.
The system should be portable to a variety of applications that are used in the company and able to upgrade the hardware and software with minimum disruption to the systems.
The system should provide low per-node overheads for all scarce computational resources including CPU, memory, I/O, network bandwidth, power consumption, and space required.
Figure 7.1 shows the high-level diagram of multiple cluster configurations. The multiple cluster-based distributed processing systems developed is partly being used for the following applications, and it will be improved and fully implemented to various compute-intensive applications that are currently used by the company once the distributed processing system is fully tested under strict conditions.
Used for pricing of different types of existing and new derivative instruments. Used for calibration of existing financial models and developing new financial
models.
Used for risk data calculation, stress testing, and scenario analysis. Used for valuation of portfolios using varying data for scenarios analysis.
168
Figure 7.1: High-level diagram of multiple cluster configurations