In our modern societal context, Artificial Intelligence (AI) algorithms, particularly those rooted in machine learning (ML) and deep learning (DL), exert significant influence over our daily routines, owing largely to the convergence of abundant data accessibility and formidable computational capabilities. ML methodologies have evolved into indispensable tools across diverse domains, spanning science, engineering, medicine, and beyond, facilitating tasks such as material innovation and medical data analysis. Notably, ML algorithms have found particular utility in inverse design and knowledge extraction within nanostructures, surpassing conventional methods by virtue of their adeptness in processing high-dimensional datasets and revealing intricate data-structure relationships. However, a critical challenge arises from the opaque nature of these ML algorithms, which often function as enigmatic black boxes, obscuring the rationale behind their decision-making processes. This opacity poses a substantial barrier for engineers striving to optimize device precision. To address this challenge, this thesis undertakes a systematic exploration focused on elucidating the knowledge embedded within ML algorithmic decisions. This endeavor employs two distinct methodologies: firstly, pruning techniques streamline neural network architectures to unveil the precise impact of each input on the output, thereby enhancing interpretability and efficiency. Secondly, interpretable models, such as SHAP (SHapley Additive exPlanations), rooted in Game Theory, provide a comprehensive understanding of the contribution made by each design parameter to the decision outcome, enabling a comparative analysis of different approaches across practical metaphotonic structures. The thesis concludes with a thorough investigation aimed at identifying the most efficient neural network architecture for inverse design in nanophotonic structures, considering both statistical and computational complexities inherent in such design endeavors.