In modern ultra-precision polishing, sub-aperture technologies are prone to mid-spatial frequency errors due to identical patterns of a path. A random tool path on a regular point set is widely used to suppress mid-spatial frequency errors. In this study, two non-grid uniform point sets, the Fibonacci and the three-directional, were introduced into optical polishing. To solve the time-consuming problem caused by a large amount of distance calculation, a distance-based weighted random (DBWR) algorithm and a linear programming and connecting (LPC) algorithm were presented. The DBWR algorithm reduces the generation time by strengthening the weight of the neighboring points in a specific direction, while the LPC algorithm adjusts the order and distance of points artificially. Then a random stitching method was proposed for the large-scale point set applying to large-sized optical surfaces, which dramatically reduced the generation time. Finally, experiments validated that the algorithms for non-grid sets can be effectively used for optical surface figuring without introducing an apparent mid-spatial frequency.