MIT SuperCloud Portal Workspace: Enabling HPC Web Application Deployment

Abstract—The MIT SuperCloud Portal Workspace enables the secure exposure of web services running on high performance computing (HPC) systems. The portal allows users to run any web application as an HPC job and access it from their workstation while providing authentication, encryption, and access control at the system level to prevent unintended access. This capability permits users to seamlessly utilize existing and emerging tools that present their user interface as a website on an HPC system creating a portal workspace. Performance measurements indicate that the MIT SuperCloud Portal Workspace incurs marginal overhead when compared to a direct connection of the same service. Keywords-Jupyter Notebook; HPC; MIT SuperCloud I. INTRODUCTION
The merger of traditional HPC systems, big data clouds, and elastic computing clouds has highlighted new challenges in managing a combined SuperCloud. One key challenge is addressing the need to run web applications and provide the end-users access to them [Kwan 1995, Thomas 2000, Alosio 2001, Milne 2009, Cholia 2010, Atwood 2016, Schuller 2016]. Web-based applications have become a common deployment method for many application development efforts, such as the Jupyter notebook interface to Python, R, Matlab, Octave, and Julia programming languages. Numerous projects expect their users to run private web servers and interact via the web browser. Enabling these same applications on a high performance computing (HPC) system encounters many challenges that do not exist in the local workstation context. The ability for users to start arbitrary web applications that are sufficiently open to the network for that user to connect to them raises obvious security concerns. The design of most HPC systems, in which compute nodes do not have direct external network connectivity, also complicates this goal. Traditional solutions for this problem, such as SSH port forwarding, are burdensome to teach to users who are not familiar with them. These solutions also can be a security risk in themselves if the tunnel endpoint on the user’s workstation is not correctly set up to limit access to just that user. The tunnel endpoints could easily be accessible to other users on the local network or on the same system if it’s shared by multiple users.
Even applications that have security built in do not solve these problems. Such applications either must be integrated into the HPC system to use its existing authentication systems or must set up their own parallel authentication system. If integrated into the HPC system, they are no longer a separate user application, as integration would require privileged access to the HPC system. If standing up their own parallel authentication system, that system must be evaluated against the security posture of the system and separately monitored for attacks, weaknesses, reported flaws, and updates that could affect the overall system security. Neither of these options can scale to meet the goal of permitting users to run arbitrary web applications for their own consumption. We have addressed these challenges by building a Portal Workspace technology capable of dynamically forwarding web applications. The portal acts as a reverse proxy to the internal network of the HPC system that is capable of being dynamically reconfigured by end users to expose their web applications to the external network. It handles encryption, authentication, and authorization of specific users’ access to exposed applications. The organization of the rest of this paper is as follows. Section II describes the technologies used to create the MIT SuperCloud Portal Workspace. Section III describes some examples of efforts enabled by this technology. Section IV shows the performance results and overhead of the system. Section V describes future work in this area. Section VI summarizes the results. II. TECHNOLOGIES The MIT SuperCloud Portal Workspace builds on a number of existing technologies for operating and managing a large, heterogeneous HPC system. These include the software stack, the scheduler, firewall, as well as the Portal Workspace itself. A. MIT SuperCloud The MIT SuperCloud software stack enables traditional enterprise computing and cloud computing workloads to be run on an HPC cluster [Reuther 2013] (see Figure 1). The software stack runs on many different HPC clusters based on a variety of hardware technologies. It supports systems with 10 GigE, and FDR InfiniBand or Intel OmniPath running Internet Protocol over InfiniBand

…(Full text truncated)…