A novel merit function was constructed using the spectral coefficient average error and standard deviation, which can simultaneously optimize the expectation of spectral coefficient error and the envelope of standard deviation. Thus, a multi-objective optimization strategy based on Non-Dominated Sorting Genetic Algorithm and Sequential quadratic programming was proposed. By comparing result of wideband anti-reflection film, cut-off filter and Infrared dual-band filter designed by the conventional algorithm and the new algorithm, the control effect of the new algorithm on sensitivity of film parameters error was verified. The results show that the novel design method has the characteristics of time-efficient calculations and is capable of effectively improving the production yield of the film system, which has practical significance.