Focusing on high resolution optical spectroscopy, a coherent optical spectrum analysis (COSA) system is investigated in this paper. Principle is built to demonstrate the operation of COSA and its signal processing in both time and frequency domain. According to COSA principle, resolution bandwidth (RBW) filters are found to have significant influence on power accuracy and spectral resolution of the optical spectrum analysis (OSA). Much effort is paid to design RBW filters, including center frequency, bandwidth and type of filters. Two RBW filters are optimized to reduce the power uncertainty of different spectral resolution and satisfy different signal under test. Then, simulations and experiments are conducted to verify COSA principle and results show that the power uncertainty is less than 0.5% and 1.2% for high and medium spectral resolution application, respectively. Finally, experiments on the OSA of actual spectra indicate that COSA system can achieve a 6 MHz spectral resolution and has an excellent capacity in analysis of fine spectrum structures.