The hub-and-spoke semantic representation theory posits that semantic knowledge is processed in a neural network, which contains an amodal hub, the sensorimotor modality-specific regions, and the connections between them. The exact neural basis of the hub, regions and connectivity remains unclear. Semantic dementia could be an ideal lesion model to construct the semantic network as this disease presents both amodal and modality-specific semantic processing (e.g. colour) deficits. The goal of the present study was to identify, using an unbiased data-driven approach, the semantic hub and its general and modality-specific semantic white matter connections by investigating the relationship between the lesion degree of the network and the severity of semantic deficits in 33 patients with semantic dementia. Data of diffusion-weighted imaging and behavioural performance in processing knowledge of general semantic and six sensorimotor modalities (i.e. object form, colour, motion, sound, manipulation and function) were collected from each subject. Specifically, to identify the semantic hub, we mapped the white matter nodal degree value (a graph theoretical index) of the 90 regions in the automated anatomical labelling atlas with the general semantic abilities of the patients. Of the regions, only the left fusiform gyrus was identified as the hub because its structural connectivity strength (i.e. nodal degree value) could significantly predict the general semantic processing of the patients. To identify the general and modality-specific semantic connections of the semantic hub, we separately correlated the white matter integrity values of each tract connected with the left fusiform gyrus, with the performance for general semantic processing and each of six semantic modality processing. The results showed that the hub region worked in concert with nine other regions in the semantic memory network for general semantic processing. Moreover, the connection between the hub and the left calcarine was associated with colour-specific semantic processing. The observed effects could not be accounted for by potential confounding variables (e.g. total grey matter volume, regional grey matter volume and performance on non-semantic control tasks). Our findings refine the neuroanatomical structure of the semantic network and underline the critical role of the left fusiform gyrus and its connectivity in the network.