High-quality ultrashort electron beams have diverse applications in a variety of areas, such as 4D electron diffraction and microscopy, relativistic electron mirrors and ultrashort radiation sources. Direct laser acceleration (DLA) mechanism can produce electron beams with a large amount of charge (several to hundreds of nC), but the generated electron beams usually have large divergence and wide energy spread. Here, we propose a novel DLA scheme to generate high-quality ultrashort electron beams by irradiating a radially polarized laser pulse on a nanofiber. Since electrons are continuously squeezed transversely by the inward radial electric field force, the divergence angle gradually decreases as electrons transport stably with the laser pulse. The well-collimated electron bunches are effectively accelerated by the circularly-symmetric longitudinal electric field and the relative energy spread also gradually decreases. It is demonstrated by three-dimensional (3D) simulations that collimated monoenergetic electron bunches with 0.75° center divergence angle and 14% energy spread can be generated. An analytical model of electron acceleration is presented which interprets well by the 3D simulation results.