Title: Concolic Execution Tailored for Hybrid Fuzzing

Insu Yun

Ph.D. Candidate

School of Computer Science

Georgia Institute of Technology

Date: Tuesday, November 17th, 2020

Time: 1:00 pm to 2:30 pm (EST)

Location: *No Physical Location*

BlueJeans: https://bluejeans.com/362727935

Committee:

Dr. Taesoo Kim (advisor), School of Computer Science, Georgia Institute of Technology

Dr. Wenke Lee, School of Computer Science, Georgia Institute of Technology

Dr. Alessandro Orso, School of Computer Science, Georgia Institute of Technology

Dr. Mayur Naik, Department of Computer and Information Science, University of Pennsylvania

Dr. Weidong Cui, Microsoft Research Redmond

Abstract:

Recently, hybrid fuzzing, which combines fuzzing and concolic execution, has been highlighted to overcome limitations of both techniques. Despite its success in contrived programs such as DARPA Cyber Grand Challenge (CGC), it still falls short in finding bugs in real-world software due to its low performance of existing concolic executors.

To address this issue, this dissertation suggests and demonstrates *Concolic execution tailored for hybrid fuzzing* with two systems: QSYM and Hybridra. First, we present QSYM, a binary-only concolic executor tailored for hybrid fuzzing. It significantly improves the performance of conventional concolic executors by removing redundant symbolic emulations for a binary. Moreover, to efficiently produce test cases for fuzzing, even sacrificing its soundness, QSYM introduces two key techniques: optimistic solving and basic block pruning. As a result, QSYM outperforms state-of-the-art fuzzers, and, more importantly, it found 13 new bugs in eight real-world programs, including file, ffmpeg, and

OpenJPEG.

Enhancing the key idea of QSYM, we discuss Hybridra, a new concolic executor for file systems. To apply hybrid fuzzing for file systems, which are gigantic and convoluted, Hybridra employs compilation-based concolic execution to boost concolic execution leveraging the existing of source code. Moreover, Hybridra introduces a new technique called staged reduction, which combines existing heuristics to efficiently generate test cases for file systems. Consequently, Hybridra outperforms a state-of-the-art file system fuzzer, Hydra, by achieving higher code coverage, and successfully discovered four new bugs in btrfs, which has been heavily tested by other fuzzers.