Rutherford discovered the nucleus by scattering alpha particles from a gold atom, and finding the scattering was identical to that from a very compact heavy central charged object. For smaller nuclei, he found the alphas bounced off the nuclear surface, giving a different scattering pattern, and making it possible to estimate nuclear size. The nuclear radius turned out to be a fraction of a ten-thousandth of the atomic radius.
All the details can be found in my lecture on Rutherford Scattering.
Note that in the above applet, a few percent of the alphas are scattered backwards. In the real experiment, if was of order one in a thousand. This was very tedious to watch! The incoming alphas pass at random distances from the nucleus, but we've cut back to those passing relatively close. (Remember in reality, the next nucleus is around ten thousand nuclear radii away.)