The objective of the proposed research is to develop an energy-efficient hardware accelerator for transformer-based models. Transformer-based language models have demonstrated remarkable performance in various natural language processing (NLP) tasks. However, their substantial memory requirements pose not only the challenge of enabling these models on edge devices but also the issue of managing their energy consumption on servers. To address these challenges, this study introduces a novel parameter quantization method for BERT. This method utilizes a Hessian-based sensitivity metric to allocate higher precision bit widths to important parameters and lower precision bit widths to less critical ones. Expanding on this method, the research proposes a new energy-efficient software-hardware co-optimization approach. This approach encompasses a hardware-friendly hessian-based row-wise mixed-precision quantization algorithm and an FPGA-based accelerator, allowing an FPGA to accommodate all necessary parameters of the BERT base model without the need for off-chip memory access during runtime. Future research efforts will extend to the development of effective quantization methods tailored for even larger and more complex language models, such as GPT-3 and OPT. Furthermore, the study will focus on designing energy-efficient accelerators specifically customized for these larger language models.