In an earlier study, trained raters provided reliable scores for a simulation-based anesthesia acute care skill assessment. In this study, we used this acute care skill evaluation to measure the performance of student nurse anesthetists and resident physician trainees. The performance of these trainees was analyzed to provide data about acute care skill acquisition during training. Group comparisons provided information about the validity of the simulated exercises. A set of six simulation-based acute care exercises was used to evaluate 43 anesthesia trainees (28 residents [12 junior and 16 senior] and 15 student nurse anesthetists). Six raters scored the participants on each exercise using either a detailed checklist, key-action items, or a global rating. Trainees with the most education and clinical experience (i.e., senior residents) received higher scores on the simulation scenarios, providing some evidence to support the validity of the multi-scenario assessment. Trainees varied markedly in ability depending on the content of the exercise. In general, anesthesia providers demonstrated similar aptitude in managing each of the six simulated events. Most participants effectively managed ventricular tachycardia, but postoperative events such as anaphylaxis and stroke were more difficult for all trainees to promptly recognize and treat. Training programs could use a simulation-based multiple encounter evaluation to measure provider skill in acute care. CONCLUSIONS A trainee's skill in managing critical events can be assessed using a multiple scenario simulation-based performance evaluation.