Rutherford discovered the nucleus by scattering alpha particles from a gold atom, expecting to find only small angle scattering based on his belief in Thomson's plum pudding model.
He was amazed to find occasional very large scattering angles, only explicable by assuming a very compact heavy central charged object: a nucleus.
For smaller nuclei, he found the alphas actually bounced off the nuclear surface, giving a different scattering pattern, and making it possible to estimate nuclear size. The nuclear radius turned out to be a fraction of a ten-thousandth of the atomic radius.
All the details can be found in my lecture on Rutherford Scattering.
Note that in the above applet, a few percent of the alphas are scattered backwards. In the real experiment, if was of order one in a thousand. This was very tedious to watch! The incoming alphas pass at random distances from the nucleus, but we've cut back to those passing relatively close. (Remember in reality, the next nucleus is around ten thousand nuclear radii away.)