Decentralized Resource Discovery and Management for Future Manycore Systems

Abstract The next generation of many-core enabled large-scale computing systems relies on thousands of billions of heterogeneous processing cores connected to form a single computing unit. In such large-scale computing environments, resource management is one of the most challenging, and complex issues for efficient resource sharing and utilization, particularly as we move toward Future ManyCore Systems (FMCS). This work proposes a novel resource management scheme for future peta-scale many-core-enabled computing systems, based on hybrid adaptive resource discovery, called ElCore. The proposed architecture contains a set of modules which will dynamically be instantiated on the nodes in the distributed system on demand. Our approach provides flexibility to allocate the required set of resources for various types of processes/applications. It can also be considered as a generic solution (with respect to the general requirements of large scale computing environments) which brings a set of interesting features (such as auto-scaling, multitenancy, multi-dimensional mapping, etc,.) to facilitate its easy adaptation to any distributed technology (such as SOA, Grid and HPC many-core). The achieved evaluation results assured the significant scalability and the high-quality resource mapping of the proposed resource discovery and management over highly heterogeneous, hierarchical and dynamic computing environments with respect to several scalability and efficiency aspects while supporting flexible and complex queries with guaranteed discovery results accuracy. The simulation results prove that, using our approach, the mapping between processes and resources can be done with high level of accuracy which potentially leads to a significant enhancement in the overall system performance.

1. Introduction Modern large-scale distributed computing systems are undergoing with the rapid evolution of processor and network architectures. And they have made possible: i) the integration of more and more cores into one single chip; ii) many-chips being interconnected into a single machine; iii) more and more machines getting connected with highly increasing bandwidth. This leads to the emergence of the next generation of manycore enabled large-scale computing systems which rely on thousands of billions of heterogeneous processing cores connected to form a single computing unit. Current large-scale computing environments such as HPC clusters (e.g., Infiniband-based distributed memory machines, Bewolf clusters), Grids and Clouds are

1 Javad Zarrin from Computer Laboratory, University of Cambridge, UK 2 Rui L. Aguiar from Universidade de Aveiro and Instituto de Telecomunicacoes, Aveiro, Portugal
3 Joao Paulo Barraca from Universidade de Aveiro and Instituto de Telecomunicacoes, Aveiro, Portugal

2 common scenarios when discussing enhancements to overall computing/system performance and resource/data/service/application accessibility through efficient sharing and utilization of the integrated infrastructures and hardware resources (such as computing, storage, data and network resources) in large-scale systems with high heterogeneity (in terms of resources, applications, platforms, users, virtual organization, administration policies, etc.) and high dynamicity. In such large-scale computing environments, resource management is one of the most challenging, and complex issues for efficient resource sharing and utilization, particularly as we move toward Future ManyCore Systems (FMCS). There are various types of techniques and methods to control and manage the infrastructure for each one of the aforementioned computing environments, which differ based on their main focus, embedded technologies and system architectures. And in fact, designing a resource management architecture which can be applied and adjusted to the requirements of these different computing environments is an extra challenge. In this report, by “manycore enabled computing systems", we mean future computing systems that support thousands of chips per compute node and thousands of heterogeneous cores per chip (as predicted in [1]). We address the problem of resource management for this future large- scale manycore enabled computing environment. In such large-scale systems (e.g., future multitenant Clouds, petascale manycores, and heterogeneous clusters) it is not feasible for the control system to centrally and statically have a complete and perfect knowledge of the entire system due to the magnitude and diversity of the amount of cores/processors and other resources. For uniformity of discussion, we will address all these co

…(Full text truncated)…