Single-domain antibodies (sdAbs) are a promising class of biotherapeutics with unique structural traits within their paratope region. The distribution of canonical conformations explored by their complementarity determining region (CDR) loops differs to some extent from conventional two-chain Fv fragments of monoclonal antibodies (mAbs). In this study, we explored in detail the canonical structures of sdAb CDR-H1 and CDR-H2 loops and compared those with mAbs from the IGHV3 and IGHV1 gene families. We surveyed the antibody structures catalogued in SAbDab and clustered the CDR canonical loops in Cartesian space. While most of the sdAb clusters were sub-populations of previously defined canonical Fv conformations of CDR-H1 and CDR-H2, our stricter clustering approach defined narrower clusters in sequence-space. Meticulous visual inspection of sub-populations allowed a clearer understanding of sequence-structure relationships. The packing densities within structural pockets contacted by CDR-H1 and CDR-H2 canonical conformations were analyzed on the premise that these pockets cannot be left vacant as they would leave exposed supportive hydrophobic residues. The fine resolution of the canonical clusters defined here revealed unique signatures within these pockets, including distinct structural complementarities between CDR-H1 and CDR-H2 canonical clusters, which could not be perceived with the previous coarser clusters. We highlight examples where a single residue change in CDR-H1 sequence is sufficient to induce a dramatic population shift in CDR-H2 conformation. This suggests that preferences in combining CDR-H1 and CDR-H2 emerged naturally during antibody evolution, leading to preferred sets of conserved amino acids at key positions in the framework as well as within the CDR loops. We outline a game of musical chairs that is necessary to maintain the integrity of the antibody structures that arose during evolution. Our study also provides refined CDR-H1 and CDR-H2 structural templates for sdAb homology modeling that could be leveraged for improved antibody design.