Hadoop extensions for distributed computing on reconfigurable active SSD -

Abstract

In recent years, there has been an expanded usage and proliferation of cloud computing services. Along with the usage expansion, there has been extensive research on extending cloud computing capabilities to be more computationally efficient. One such example is the recent work that proposed a novel architecture for a distributed reconfigurable processing platform with SSD storage, called RASSD. The system is targeted for processing data-intensive applications at the storage node itself, without having to move data over slow networks. In this thesis, we extend the Hadoop cloud computing framework to serve as cloud computing platform for integrating hardware accelerating capabilities with cloud computing environments. In particular, we show how to extend Hadoop for the distributed RASSD environment. Our contributions include: 1. Additional components for extending Hadoop and the Map-Reduce model, 2. The changes needed to configure the usage of Hadoop for the RASSD environment, 3. Demonstrating how applications are mapped to the extended MapReduce model, and 4. evaluating the benefits and impact of hardware acceleration on overall application performance in the cloud environment. Our RASSD Hadoop work suggests a two-level implementation of MapReduce to support the capability of accelerating map functions on the active storage units. To support the evaluation of the distributed hardware acceleration system under different conditions, we built a Hadoop-based simulator to evaluate the RASSD environment. Additionally, network simulations were used to simulate the performance benefit with a larger number of remote RASSD nodes. The system was evaluated using applications provided with Stanford’s Phoenix MapReduce platform. Our implementations demonstrated the ease of programming in the extended Hadoop environment, while the experimental results showed significant performance gains with map acceleration on the remote RASSD processing nodes. With an assumption of 20X map acceleration on RASSD nodes, the overall p