Ph.D. Thesis Proposal Announcement
Title: Elf+: Efficient Lightweight Fast Streaming Processing at Scale

Liting Hu
Ph.D. Student
School of Computer Science
College of Computing
Georgia Institute of Technology

Date: Tuesday, March 31, 2015
Time: 12:00 pm – 1:30 pm EDT
Location: KACB 3402

Committee
Dr.Karsten Schwan (Advisor), School of Computer Science, Georgia Institute of Technology
Dr.Ling Liu, School of Computer Science, Georgia Institute of Technology
Dr.Matthew Wolf, School of Computer Science, Georgia Institute of Technology
Dr.Greg Eisenhauer, School of Computer Science, Georgia Institute of Technology

Abstract: 
Stream processing has become a key means for gaining rapid insights from webserver-captured data. Challenges include how to scale to numerous, concurrently running streaming jobs, to coordinate across those jobs to share insights, to make online changes to job functions to adapt to new requirements or data characteristics, and for each job, to efficiently operate over different time windows, and also guarantee each job’s target end-to-end delay. 

The Elf+ stream processing system addresses these new challenges. Implemented over a set of agents enriching the web tier of datacenter systems, Elf+ obtains scalability by using a decentralized “many masters” architecture where for each job, live data is extracted directly from webservers, and placed into memory-efficient compressed buffer trees (CBTs) for local parsing and temporary storage, followed by subsequent aggregation using shared reducer trees (SRTs) mapped to sets of worker processes. Job masters at the roots of SRTs can dynamically customize worker actions, obtain aggregated results for end user delivery and/or coordinate with other jobs. 

An Elf+ prototype implemented and evaluated for a larger scale configuration demonstrates scalability, high per-node throughput, sub-second job latency, and sub-second ability to adjust the actions of jobs being run.