Pairing in the cuprate high-temperature superconductors and its origin remain among the most enduring mysteries in condensed matter physics. With cross-sectional scanning tunneling microscopy/spectroscopy, we clearly reveal the spatial-dependence or inhomogeneity of the superconducting gap structure of Bi$_2$Sr$_2$CaCu$_2$O$_{8+\delta}$ (Bi2212) and YBa$_2$Cu$_3$O$_{7-x}$ (YBCO) along their $c$-axes on a scale shorter than the interlayer spacing. By tunneling into the (100) plane of a Bi2212 single crystal and a YBCO film, we observe both U-shaped tunneling spectra with extended flat zero-conductance bottoms, and V-shaped gap structures, in different regions of each sample. On the YBCO film, tunneling into a (110) surface only reveals a U-shaped gap without any zero-bias peak. Our analysis suggests that the U-shaped gap is likely a nodeless superconducting gap. The V-shaped gap has a very small amplitude, and is likely proximity-induced by regions having the larger U-shaped gap.