Taking advantage of the broad spectrum of the Cabannes-Brillouin scatter from atmospheric molecules, the high spectral resolution lidar (HSRL) technique employs a narrow spectral filter to separate the aerosol and molecular scattering components in the lidar return signals and therefore can obtain the aerosol optical properties as well as the lidar ratio (i.e., the extinction-to-backscatter ratio) which is normally selected or modeled in traditional backscatter lidars. A polarized HSRL instrument, which employs an interferometric spectral filter, is under development at the Zhejiang University (ZJU), China. In this paper, the theoretical basis to retrieve the aerosol lidar ratio, depolarization ratio and extinction and backscatter coefficients, is presented. Error analyses and sensitivity studies have been carried out on the spectral transmittance characteristics of the spectral filter. The result shows that a filter that has as small aerosol transmittance (i.e., large aerosol rejection rate) and large molecular transmittance as possible is desirable. To achieve accurate retrieval, the transmittance of the spectral filter for molecular and aerosol scattering signals should be well characterized.