Data and Application Migration in Cloud based Data Centers: Architectures and techniques

Gong Zhang
School of Computer Science
Georgia Institute of Technology

Committee:

Prof. Ling Liu (Advisor, School of Computer Science, Georgia Tech)
Prof. Sham Navathe (School of Computer Science, Georgia Tech)
Prof. Calton Pu (School of Computer Science, Georgia Tech)
Prof. Ed Omiecinski, (School of Computer Science, Georgia Tech)
Prof. Joao Eduardo Ferreira, (Department of Computer Science, University of Sao Paulo)

Abstract:

Computing and communication have continued to expedite the growth of digital data and the complexity of applications. Today, the cost of managing and scaling hardware and software systems ranges from two to ten times the acquisition cost of the hardware and software systems. Such cost continues to increase as data grows and applications become more complex.

We see an increasing demand on technologies for transferring management burden from humans to software. Data migration and application migration are popular technologies that enable computing and system management to be autonomic and self-managing.

In this dissertation, we examine important issues in designing and developing scalable architectures and techniques for efficient and effective data migration and application migration. The first contribution we have made is to investigate the opportunity of automated data migration across multi-tier storage systems. The significant IO improvement in Solid State Disks (SSD) over traditional rotational hard disks (HDD) motivates the integration of SSD into existing storage hierarchy for enhanced performance. We developed adaptive look-ahead data migration approach to effectively integrate SSD into the multi-tiered storage architecture. When using the fast and expensive SSD tier to store the high temperature data (hot data) while placing the relatively low temperature data (low data) in the HDD tier, one of the important functionality is to manage the migration of data as their access patterns are changed from hot to cold and vice versa. We designed and implemented an adaptive lookahead data migration model that can dynamically adapt the data migration schedule to achieve the optimal migration effectiveness by taking into account of application specific characteristics and I/O profiles as well as workload deadlines.

The second main contribution we have made in this dissertation research is to address the challenge of ensuring reliability and balancing loads across a network of computing nodes, managed in a decentralized service computing system. We have developed a distributed workload migration scheme with controlled replication. It utilizes a shortcut-based optimization to increase the resilience of the system against various node failures and network partition failures. In addition, we devise a dynamic load balancing technique to scale the system in anticipation of unexpected workload changes. Our approach is highly scalable under changing service workloads with moving hotspots and highly reliable in the presence of massive node failures.

The third contribution in this dissertation research is to study how to simplify the management overheads in migrating large scale enterprise applications/system from local data center to the Cloud. More and more enterprises are moving some workloads from their local data centers to Cloud, such as EC2, to reduce the cost of ownership and leverage the benefits provided by Cloud based data centers. However, such migration process turns out to be non-trivial. By in-depth analysis of some popular multi-tier middleware systems such as Hadoop, MySQL etc, we show the complexities and difficulties of application migration process and propose an autonomic management framework for migration configuration and migration validation, aiming at reducing typical operator errors, eliminating the risks of hidden migration pitfalls, and increasing migration assurance. In this defense talk, I will give an overview of my dissertation research and then highlight in detail our most recent work on application migration validation.