Many soft materials yield under mechanical loading, but how this transition from solid-like behavior to liquid-like behavior occurs can vary significantly. Understanding the physics of yielding is of great interest for the behavior of biological, environmental, and industrial materials, including those used as inks in additive manufacturing and muds and soils. For some materials, the yielding transition is gradual, while others yield abruptly. We refer to these behaviors as being ductile and brittle. The key rheological signatures of brittle yielding include a stress overshoot in steady-shear-startup tests and a steep increase in the loss modulus during oscillatory amplitude sweeps. In this work, we show how this spectrum of yielding behaviors may be accounted for in a continuum model for yield stress materials by introducing a parameter we call the brittility factor. Physically, an increased brittility decreases the contribution of recoverable deformation to plastic deformation, which impacts the rate at which yielding occurs. The model predictions are successfully compared to results of different rheological protocols from a number of real yield stress fluids with different microstructures, indicating the general applicability of the phenomenon of brittility. Our study shows that the brittility of soft materials plays a critical role in determining the rate of the yielding transition and provides a simple tool for understanding its effects under various loading conditions.