Sexual size dimorphism is generally associated with sexual selection via agonistic male competition in nonhuman primates. These primate models play an important role in understanding the origins and evolution of human behavior. Human size dimorphism is often hypothesized to be associated with high rates of male violence and polygyny. This raises the question of whether human dimorphism and patterns of male violence are inherited from a common ancestor with chimpanzees or are uniquely derived. Here I review patterns of, and causal models for, dimorphism in humans and other primates. While dimorphism in primates is associated with agonistic male mate competition, a variety of factors can affect male and female size, and thereby dimorphism. The causes of human sexual size dimorphism are uncertain, and could involve several non-mutually-exclusive mechanisms, such as mate competition, resource competition, intergroup violence, and female choice. A phylogenetic reconstruction of the evolution of dimorphism, including fossil hominins, indicates that the modern human condition is derived. This suggests that at least some behavioral similarities with Pan associated with dimorphism may have arisen independently, and not directly from a common ancestor.