[Objective] To achieve high-precision monitoring of soil organic matter in saline-alkali farmland, this study proposes a framework integrating fractional-order differentiation (FOD), continuous wavelet transform (CWT), and spectral index construction. [Method] Two saline-alkali farmland plots in Jiashi County, Xinjiang, were selected as the study area. Ground-based hyperspectral reflectance data and field sampling data were used as data sources. The original hyperspectral reflectance was subjected to differential transformation with a step size of 0.25. On this basis, continuous wavelet transform was performed at six scales, and the optimal combination of differential order and wavelet decomposition scale was selected to construct a ratio index (RI), difference index (DI), and normalized index (NDI). Random forest (RF), extreme gradient boosting (XGBoost), support vector regression (SVR), and light gradient boosting machine (LightGBM) models were built to quantitatively estimate soil organic matter content. [Result] The results show that after FOD processing, the 0.75-order differential spectrum had the highest correlation with organic matter, significantly outperforming the original spectrum and integer-order transformation. After the combined FOD_CWT processing, the 0.25-order FOD combined with 25-scale CWT showed the best performance, with a correlation coefficient of 0.726. The correlation between spectral reflectance data and organic matter increased by 0.197, effectively enhancing the relationship between the spectrum and soil organic matter. The FOD_CWT_XGBoost combination showed the best prediction performance, with a test set R2 of 0.744. [Conclusion] The combination of fractional-order differentiation and wavelet transform can deeply extract spectral information, and the constructed model is suitable for estimating soil organic matter.