Title: An IR-based Fuzzing Approach for Finding Context-Aware Bugs in API-based Systems

Wen Xu

Ph.D. Candidate

School of Computer Science

College of Computing

Georgia Institute of Technology

Date: April 27th, 2021 (Tue)

Time: 2:00 PM - 4:00 PM (EDT)

Location: *No Physical Location*

BlueJeans: https://bluejeans.com/5992360268

Committee:

Dr. Taesoo Kim (Advisor, School of Computer Science, Georgia Institute of Technology)

Dr. Wenke Lee (School of Computer Science, Georgia Institute of Technology)

Dr. Alessandro Orso (School of Computer Science, Georgia Institute of Technology)

Dr. Qirun Zhang (School of Computer Science, Georgia Institute of Technology)

Dr. Weidong Cui (Microsoft Research Redmond)

Abstract:

API-based systems, a large group of security-critical software programs including web browser and OS kernels, accept program inputs being composed of API calls. Considering the scale and complexity of an API-based system, fuzzing proves to be the most effective approach for bug detection in practice. To effectively discover new bugs in an API-based system nowadays, a fuzzer needs to generate syntactically and semantically correct API calls, which are not declined at an early stage.

Grammar-based API fuzzers can generate random API calls in various syntaxes described by context-free grammars. Nevertheless, context-free grammars are unable to fully express API semantics and cause semantic errors. Meanwhile, context-aware API fuzzers maintain context information for evaluating API semantics and generate semantically correct API calls. However, their existing approaches are domain specific and cannot be easily generalized. In summary, an API fuzzer that simultaneously supports rich API formats and resolves API semantics do not exist.

To fill this research gap, we present RPG (Random Program Generator), a general and semantic-aware solution to random API generation for common API-based systems.RPG generates random API programs in RPG IR, a formal and context-aware representation that models various API syntax and semantics in a general form. We also propose ASL (API Specification Language), a formal language for programming API specifications. The API description in ASL can be translated by RPG into random API programs in RPG IR for testing. We evaluate the effectiveness of RPG by running it against WebKit with the ASL files that describe the DOM specification. As a domain-agnostic approach,RPG discovers a similar number of bugs compared to the state-of-the-art DOM fuzzer. In addition, RPG largely outperforms a grammar-based fuzzer by triggering 3x more unique crashes in a 24-hour experiment.