OBJECTIVE Knowledge-based planning (KBP) offers the ability to predict dose-volume metrics based on information extracted from previous plans, reducing plan variability and improving plan quality. As clinical integration of KBP is increasing there is a growing need for quantitative evaluation of KBP models. A .NET-based application, RapidCompare, was created for automated plan creation and analysis of Varian RapidPlan models. METHODS RapidCompare was designed to read calculation parameters and a list of reference plans. The tool copies the reference plan field geometry and structure set, applies the RapidPlan model, optimizes the KBP plan, and generates data for quantitative evaluation of dose-volume metrics. A cohort of 85 patients, divided into training (50), testing (10), and validation (25) groups, was used to demonstrate the utility of RapidCompare. After training and tuning, the KBP model was paired with three different optimization templates to compare various planning strategies in the validation cohort. All templates used the same set of constraints for the planning target volume (PTV). For organs-at-risk, the optimization template provided constraints using the whole dose-volume histogram (DVH), fixed-dose/volume points, or generalized equivalent uniform dose (gEUD). The resulting plans from each optimization approach were compared using DVH metrics. RESULTS RapidCompare allowed for the automated generation of 75 total plans for comparison with limited manual intervention. In comparing optimization techniques, the Dose/Volume and Lines optimization templates generated plans with similar DVH metrics, with a slight preference for the Lines technique with reductions in heart V30Gy and spinal cord max dose. The gEUD model produced high target heterogeneity. CONCLUSIONS Automated evaluation allowed for the exploration of multiple optimization templates in a larger validation cohort than would have been feasible using a manual approach. A final KBP model using line optimization objectives produced the highest quality plans without human intervention.