The genomes of all retroviruses encode the Gag Pol and Env structural proteins. Human and simian lentiviruses have acquired non-structural proteins among which Nef plays a major role in the evolution of viral infection towards an immunodeficiency syndrome. Indeed, in the absence of a functional nef gene, primate lentiviruses are far less pathogenic than their wild type counterparts. The multiple protein-protein interactions in which Nef is involved all contribute to explain the role played by Nef in HIV- and SIV-associated disease progression. This review summarizes common and distinct features among Nef proteins and how they contribute to increasing HIV and SIV fitness towards their respective hosts.