The objective of the proposed research is to enhance photonic signal processing capabilities in optical communication links, spanning both fiber-optic systems and silicon photonic chips. For fiber systems, this will be achieved by demonstrating the effectiveness of using machine learning (ML), primarily as a nonlinear equalizer, in pushing conventional optical digital signal processing (DSP) beyond its present performance limits. Conventional DSP algorithms handle linear impairments well but are ill-suited to the nonlinear impairments arising from fiber, amplifier and transceiver nonlinearities. As universal nonlinear function approximators, ML models are better suited to this problem. To strengthen the case for ML and move toward deployment-ready solutions, the proposed research focuses on (1) compression techniques for a compute and power-efficient model, and (2) improved interpretability via a physics-informed learning paradigm. Fibers are optimal for long-distance coherent transmission, but the data must ultimately be processed on compact semiconductor chips for short-reach applications in the data center. Advances in integrated photonic design and characterization methods are critical for higher performance, dense on-chip integration, and optimal operation in the nonlinear regime. To this end, the proposed research focuses on (3) topology optimization (TO) of photonic devices for commercial foundry platforms capable of handling multiple waveguide modes with high modal purity. TO is a powerful inverse-design technique that opens up a vast design space through millions of degrees of freedom and a user-defined figure of merit. Every TO structure must still be routed on-chip using waveguides and bends, making it important to know when the waveguide response shifts from the linear to the nonlinear regime as input pump power increases. The proposed (4) theoretical and experimental analysis of nonlinear loss in silicon waveguides will allow predicting waveguide behavior during design itself (before testing). Once validated against experimental data, the model can extract key system parameters, such as carrier lifetime, through curve fitting alone, without requiring additional experiments.