A polarization averaged short-time Fourier transform (PASTFT) technique is developed for distributed fiber birefringence characterization based on counterpropagating stimulated Brillouin scattering (SBS) gain signal. This technique can be used for the birefringence characterization of the general elliptical birefringent fiber. A theoretical model on polarization matching of counterpropagating SBS process is established. The performance of the short-time Fourier transform (STFT) method and the PASTFT technique is analyzed by using the simulation of the theoretical model. Simulation results show that the process of polarization average could effectively reduce the birefringence characterization error caused by the polarization dependence of the local period of SBS gain. A less than 8% normalized root mean square error is achieved for the characterization of the length of the birefringence vector on elliptical birefringent fibers. The PASTFT technique is experimentally verified by the distributed measurement of beat length and differential group delay of a standard single-mode fiber via the Brillouin optical time domain analysis system.