The recent development of effective immune checkpoint inhibition (ICI), first demonstrated in melanoma, has revolutionized cancer treatment. Monoclonal antibodies blocking the immune checkpoints cytotoxic T lymphocyte-associated antigen 4 (CTLA-4) and programmed death 1 receptor (PD-1) have shown substantial clinical benefit in a subset of patients across tumor types and in both the metastatic and adjuvant settings. In this article, we review the interaction between the immune system and solid tumors, and describe modes of immune response failure and the physiologic role of immune checkpoints. We also review the known mechanisms of immune checkpoint inhibitors, focusing on US FDA-approved agents targeting CTLA-4 and PD-1. Within this framework, we classify hypothesized tumor intrinsic and extrinsic predictive markers for response and resistance to ICI, and map them to their putative underlying biological mechanism. Finally, we outline future directions in ICI, including the development of new therapeutic targets, rational combination therapies, integrated predictive models for individual patients to optimize therapy, and expansion into different disease types.