Title: Performant Software Hardening under Hardware Support

Ren Ding

Ph.D. Student in Computer Science

School of Computer Science

College of Computing

Georgia Institute of Technology

Date: April 26, 2021

Time: 3:00 PM to 5:00 PM (EST)

Location (remote via Bluejeans): https://bluejeans.com/8795017971

Committee:

Dr. Taesoo Kim (Advisor, School of Computer Science, Georgia Institute of Technology)

Dr. Wenke Lee (School of Computer Science, Georgia Institute of Technology)

Dr. Alessandro Orso (School of Computer Science, Georgia Institute of Technology)

Dr. Brendan Saltaformaggio (School of Electrical and Computer Engineering, Georgia Institute of Technology)

Dr. Yeongjin Jang (School of Electrical Engineering and Computer Science, Oregon State University)

Abstract

With a booming number of applications and end-users in the past decade, software security has been emphasized more than ever. Nonetheless, a consistent increase of security-critical bugs has been observed along the way, mainly due to the variety and complexity of existing software pieces. To mitigate the situation, software hardening in the daily development cycle typically involves three phases, including bug finding, runtime security enforcement, and fault analyses in case the prior steps have failed. Among the various software hardening techniques proposed, a considerable number of works have relied on available hardware support to achieve their goals. The reasons behind the noticeable trend are three-folded. First, the performance benefit from hardware can be substantial compared to a purely software-based solution. Second, compatibility and ease of use are also keys for more solutions to adopt hardware features besides the performance gain. Last, implementation with hardware support can consequentially present a smaller codebase, thus introducing less attack surface for attackers.

In this dissertation, I present three hardware-assisted solutions for performant software hardening. The first one is PITTYPAT, a runtime enforcement for path-sensitive control-flow integrity. By utilizing Intel PT, it computes branch targets with points-to analyses in an efficient and precise manner. The second one is SNAP, a customized hardware platform that implements hardware primitives to enhance the performance of coverage-guided fuzzing. Given the program states originated from the existing CPU pipeline, our prototype on the FPGA platform enables a transparent support of fuzzing with near-zero tracing overhead. Finally, I will present a nested virtualization framework for fuzzing non-user applications, such as hypervisors. With a snapshot mechanism supported by the x86 virtualization extension and a customized kernel for fuzzing execution, our system demonstrates a 72x improvement on the fuzzing throughput compared to the prior solutions, and finds 14 zero-day bugs among the real-world hypervisors.