Symmetry broken states arise naturally in oscillatory networks. In this Letter, we investigate chaotic attractors in an ensemble of four mean-coupled Stuart-Landau oscillators with two oscillators being synchronized. We report that these states with partially broken symmetry, so-called chimera states, have different setwise symmetries in the incoherent oscillators, and in particular, some are and some are not invariant under a permutation symmetry on average. This allows for a classification of different chimera states in small networks. We conclude our report with a discussion of related states in spatially extended systems, which seem to inherit the symmetry properties of their counterparts in small networks.