Title : Concolic execution tailored for hybrid fuzzing

Insu Yun

Ph.D. Student in Computer Science

School of Computer Science

College of Computing

Georgia Institute of Technology

Date: Thursday, November 21, 2019

Time: 2:30 pm - 4:30 pm (EST)

Location: Klaus 3126

Committee:

---------------

Dr. Taesoo Kim (Advisor, School of Computer Science, Georgia Institute of Technology) Dr. Wenke Lee (School of Computer Science, Georgia Institute of Technology) Dr. Alessandro Orso (School of Computer Science, Georgia Institute of Technology) Dr. Mayur Naik (Department of Computer and Information Science, University of Pennsylvania) Dr. Weidong Cui (Microsoft Research Redmond)

Abstract:

---------------

Recently, hybrid fuzzing, which combines fuzzing and concolic execution, has been highlighted to overcome limitations of both techniques. Despite its success in contrived programs such as DARPA Cyber Grand Challenge (CGC), it still falls short in finding bugs in real-world software due to its low performance of existing concolic executors.

To address this issue, we first present QSYM, a binary-only concolic executor tailored for hybrid fuzzing. It significantly improves the performance of conventional concolic executors by removing redundant symbolic emulations for a binary. Moreover, to efficiently produce test cases for fuzzing, even sacrificing its soundness, QSYM introduces two key techniques: optimistic solving and basic block pruning. As a result, QSYM outperforms state-of-the-art fuzzers, and more improtantly, it found 13 new bugs in eight real-world programs including ffmpeg and OpenJPEG.

Enhancing the key idea of QSYM, we will discuss libHybridFuzzer, a new concolic executor for source code. Unlike the existing tools (e.g., KLEE, S2E and even QSYM), libHybridFuzzer shows how to benefit from the existence of source code by employing instrumented concolic execution. Moreover, it will apply a new technique called continuous pruning, which resolves problems in the aforementioned basic block pruning: incompleteness and high overhead. To further explore advantages of having source, we will also discuss the emprical study to evaluate the impacts of compliation in hybrid fuzzing, which is only possible with source code.  Many system software are performance critical, and they are typically implemented in programming languages that are efficient but prone to security vulnerabilities. Existing approaches to address vulnerable software tend to address some specific harmful effects (e.g., detection based on evidence of an exploit), and thus have limited effectiveness. This thesis proposal presents three tools which eliminate or analyze vulnerabilities to protect computer systems.