Bone's hierarchical arrangement of collagen and mineral generates a confluence of toughening mechanisms acting at every length scale from the molecular to the macroscopic level. Molecular defects, disease, and age alter bone structure at different levels and diminish its fracture resistance. However, the inability to isolate and quantify the influence of specific features hampers our understanding and the development of new therapies. Here, we combine in situ micromechanical testing, transmission electron microscopy and phase-field modelling to quantify intrinsic deformation and toughening at the fibrillar level and unveil the critical role of fibril orientation on crack deflection. At this level dry bone is highly anisotropic, with fracture energies ranging between 5 and 30 J/m2 depending on the direction of crack propagation. These values are lower than previously calculated for dehydrated samples from large-scale tests. However, they still suggest a significant amount of energy dissipation. This approach provides a new tool to uncouple and quantify, from the bottom up, the roles played by the structural features and constituents of bone on fracture and how can they be affected by different pathologies. The methodology can be extended to support the rational development of new structural composites.