Effective Load Metric and Efficient Initial Job Placement for Dynamic Load Balancing in Cluster

Abstract

High performance clusters are being configured specially to give data centers that require extreme performance and the processing power they need. When the data is accessed across clusters the data latency time has significant impact on the performance. In the literature it is given that memory and I/O have become the new bottleneck, instead of processing power in achieving efficient load balance at higher performance for cluster computer systems. Initial job placement and load balancing are the key aspects affecting the performance. The proposed technique combines data access patterns, memory and CPU utilization and locality of memory to consider as load metric in the load balancing aspect across cluster. A scheduling algorithm based on this metric has been proposed to dynamically balance the load in the cluster. Initial job placement for a job in the cluster considers data access patterns and for load balance aspect metric constitutes CPU, memory utilization including locality of memory. Experimental results shown performance improvement to considerable levels with the implementation of the concept, specifically when the cost of data access from other clusters is higher and is proportionate to the amount of data.