The loss of fragile X mental retardation (FMR1) gene function causes fragile X syndrome (FXS), a common mental retardation syndrome. Anxiety and abnormal social behaviors are prominent features of FXS in humans. To better understand the role of FMR1 in these behaviors, we analyzed anxiety-related and social behaviors in Fmr1 knockout (KO) mice. In the mirrored chamber test, Fmr1 KO mice showed greater aversion to the central mirrored chamber than wild-type (WT) littermates, suggesting increased anxiety-like responses to reflected images of mice. Fmr1 KO mice exhibited abnormal social interactions in a tube test of social dominance, winning fewer matches than WT littermates. In a partition test, Fmr1 KO mice had normal levels of social interest and social recognition. However, during direct interaction tests, Fmr1 KO mice showed significant increases in sniffing behaviors. We further tested the influence of environmental familiarity on the social responses of Fmr1 KO mice to unfamiliar partners. In unfamiliar partitioned cages, Fmr1 KO mice did not differ from WT mice in investigation of unfamiliar partners. However, in familiar partitioned cages, Fmr1 KO mice showed less investigation of a newly introduced partner during the first 5 min and more investigation during the last 5 min of a 20-min partition test, behaviors consistent with initial social anxiety followed by enhanced social investigation. Our findings indicate that the loss of Fmr1 gene function results in altered anxiety and social behavior in mice and demonstrate that the Fmr1 KO mouse is a relevant animal model for the abnormal social responses seen in FXS.