UW-Madison Logo

The ADvanced Systems Laboratory (ADSL)
Publication abstract

FATE and DESTINI: A Framework for Cloud Recovery Testing

Haryadi S. Gunawi1, Thanh Do2, Pallavi Joshi1, Peter Alvaro1, Joseph M. Hellerstein1, Andrea C. Arpaci-Dusseau2, Remzi H. Arpaci-Dusseau2, Koushik Sen1, Dhruba Borthakur3
1 EECS Computer Science Division, University of California, Berkeley
2 Department of Computer Sciences, University of Wisconsin-Madison
3 FaceBook


As the cloud era begins and failures become commonplace, failure recovery becomes a critical factor in the availability, reliability and performance of cloud services. Unfortunately, recovery problems still take place, causing downtimes, data loss, and many other problems. We propose a new testing framework for cloud recovery: FATE (Failure Testing Service) and DESTINI (Declarative Testing Specifications). With FATE, recovery is systematically tested in the face of multiple failures. With DESTINI, correct recovery is specified clearly, concisely, and precisely. We have integrated our framework to several cloud systems (e.g., HDFS [33]), explored over 40,000 failure scenarios, wrote 74 specifications, found 16 new bugs, and reproduced 51 old bugs.

Full Paper: PDF, BibTex