Project 8 aims to measure the absolute neutrino mass to within 40 meV using the energy spectrum of electrons from tritium beta decay using Cyclotron Radiation Emission Spectroscopy (CRES). The Phase III Cavity CRES Apparatus (CCA) aims to demonstrate the effectiveness of a cavity in enhancing the energy resolution to 0.3 eV - a goal that requires careful reconstruction of the raw time series signal from the apparatus at 403 MHz. The current classical reconstruction pipeline, which processes these time series through a Short-Time Fourier Transform, track finding, and sideband classification, suffers from three fundamental limitations: efficiency losses from discarding events in the double-valued frequency-to-energy mapping region, discarding events whose axial sidebands fall below the noise floor, and an inability to perform radial reconstruction which incurs a systematic energy bias that directly limits resolution. This talk demonstrates the ongoing promising effort on using state space model (SSM) based neural networks to bypass the classical pipeline entirely by regressing kinetic energy, pitch angle, and radial position directly from the raw time series data. We show that SSMs perform better on long sequences required for CCA reconstruction compared to other popular architectures such as 1D Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Transformers used in Large Language Models (LLMs). Using simulations from Kassiopeia and Locust, we show that a six-layer S4D model reconstructs energy, pitch angle, axial frequency, cyclotron frequency, and radius jointly, achieving proof-of-concept energy reconstruction at ~0.4 eV FWHM and radius reconstruction at ~0.2 mm FWHM on noise-free data. On noisy data, combining a denoiser with a time series regression network achieves a current best of ~2 eV FWHM energy resolution, with further improvements toward the 0.3 eV target under active development.