The object of the research is to design an efficient Deep Neural Network (DNN) accelerator and denoising algorithm for the application of modulation recognition of the received Radio Frequency (RF) signals. In order to reduce computational demand of the hardware and increase the operation frequency, a low complexity DNN model employing ternary weight quantization is demonstrated with co-analysis of the classification accuracy and the hardware design. To maximize the benefits of the ternary weight quantization, I propose a new hardware design dedicated to the ternary weight type. Also the new training method to compensate the limited training capability of the ternary weight quantization is proposed to improve the classification accuracy of the ternary weight based network. The physical design analysis shows that the suggested DNN model with the ternary weight quantization can reduce hardware cost significantly, and the proposed training method is suitable for improving the modulation classification accuracy of the RF signal. Considering the fact that the RF transmission channel is exposed to various type/level of the noise, I propose the DNN-based denoising mechanisms optimized for suppressing the noise of the RF signal, which are the reconstructing of the signal with untrained DNN and the noise-adaptive autoencoder. The proposed methods improve the classification accuracy of the low Signal to Noise Ratio (SNR) signal efficiently with minimal impact on the hardware design of the entire DNN.