Dense non-Brownian suspensions exhibit a spectacular and abrupt drop in viscosity under change of shear direction, as revealed by shear inversions (reversals) or orthogonal superposition. Here, we introduce an experimental setup to systematically explore their response to shear rotations, where one suddenly rotates the principal axes of shear by an angle θ, and measure the shear stresses with a biaxial force sensor. Our measurements confirm the genericness of the transient decrease of the resistance to shear under unsteady conditions. Moreover, the orthogonal shear stress, which vanishes in steady state, takes non-negligible values with a rich θ dependence, changing qualitatively with solid volume fraction ϕ and resulting in a force that tends to reduce or enhance the direction of flow for small or large ϕ. These experimental findings are confirmed and rationalized by particle-based numerical simulations and a recently proposed constitutive model. We show that the rotation angle dependence of the orthogonal stress results from a ϕ-dependent interplay between hydrodynamic and contact stresses.