The objective of the proposed research is to develop methods to leverage sparse natural language feedback to improve large language models on various tasks. The motivation for this work stems from the overarching goal in the artificial intelligence (AI) field of achieving alignment - that is, AI that advances the intended objectives of its users. Recent progress with large language models has given rise to capable chatbot agents that can solve myriad tasks described in natural language and engage with users in a conversational setting. Despite these advances, large language models still face fundamental challenges in aligning with human objectives. One such challenge is that of memory: language models trained with supervised fine-tuning and reinforcement learning from human feedback are static models that, while capable of adjusting to feedback in real-time, do not remember past interactions. To make effective use of this data, models must generalize from previously seen feedback to new, unseen tasks. This is equivalent to the problem of solving a task given a small subset of data annotated with feedback. We refer to this problem as sparse feedback. We seek to better understand the role of natural language feedback in language model response generation and to develop methods to improve responses in the sparse feedback scenario.