Four-wave mixing instabilities are theoretically studied for continuous wave propagation in ultrasmall core photonic-crystal and tapered fibers. The waveguide, or geometrical, contribution to the overall dispersion of these structures is much stronger than in conventional fibers. This leads to the appearance of unstable frequency bands that are qualitatively and quantitatively different from those seen in conventional fibers. The four-wave mixing theory developed here is based on the full wave equation, which allows rigorous study of the unstable bands even when the detunings are of the order of the pump frequency itself. Solutions obtained using the generalized nonlinear Schrödinger equation, which is an approximate version of the full wave equation, reveal that it suffers from several deficiencies when used to describe four-wave mixing processes.