CaMgAl-layered double hydroxides (CaMgAl-LDHs) were synthesized by a co-precipitation method to prepare sodium oleate-modified, borate-intercalated CaMgAl-LDHs (O-CaMgAl-LDHs) using in-situ intercalation and modification, and the LDHs samples were characterized by X-ray diffraction (XRD), Fourier transform infrared spectroscopy (FT-IR), field emission scanning electron microscopy (FESEM), and thermal gravimetric analysis (TGA). The FESEM observations showed that the as-prepared CaMgAl-LDHs had a lamellar structure with a particle size of 200~500 nm, while the O-CaMgAl-LDHs had a plate-like structure with a particle size of about 100 nm. TGA showed that O-CaMgAl-LDHs resulted in higher thermal stability at high temperature compared to CaMgAl-LDHs. O-CaMgAl-LDHs/ABS composites were prepared by adding O-CaMgAl-LDHs to acrylonitrile-butadiene-styrene resin (ABS) to test the resulting flame retardancy and mechanical properties, and the results showed that the limiting oxygen index (LOI) could increase from 18% to 26%, while the mechanical properties decreased significantly when the added fraction was 40% (relative to ABS). O-CaMgAl-LDHs, ammonium polyphosphate (APP) and expandable graphite (EG) were added into the ABS to prepare ABS composites, and the effects of different compositions on the flame retardancy and mechanical properties of the ABS composites were investigated. The results showed that, when adding 5 g of O-CaMgAl-LDHs, 1 g of APP, and 14 g of EG into 40 g of ABS, the LOI of the ABS composite reached 28.8%, and the composite prepared could meet the V-0 grade requirements of the UL-94 combustion test, while the flexural strength decreased only 21.9% compared to pure ABS, the smallest decrease compared to all of the other composites.