Tapered optical fibers are promising one-dimensional nanophotonic waveguides that can provide efficient coupling between their fundamental mode and quantum nanoemitters placed inside them. Here, we present numerical studies on the coupling of single nitrogen-vacancy (NV) centers (single point dipoles) in nanodiamonds with tapered fibers. Our results lead to two important conclusions: (1) A maximum coupling efficiency of 53.4% can be realized for the two fiber ends when the NV bare dipole is located at the center of the tapered fiber. (2) NV centers even in 100-nm-sized nanodiamonds where bulk-like optical properties were reported show a coupling efficiency of 22% at the taper surface, with the coupling efficiency monotonically decreasing as the nanodiamond size increases. These results will be helpful in guiding the development of hybrid quantum devices for applications in quantum information science.