Many folks have an opinion, so I might as well add my 2 cents also. From my gun room I can pick just about any caliber (except 270) I might want from 22 to 45, and I'm speaking of rifles. For varmint I usually take one of my Thompson Encore's, but they would work okay for smaller deer also if you were careful with your shot placement, and that should go without saying. I have a 22-250 T/C with a scope and rifle length barrel, I have the same thing in 204 Ruger and 223. I also have a few heavier calibers, but you are after something with limited recoil. There is no reason to have a heavy rifle if you are using calibers under 25, because most of these rifle calibers don't recoil all that much any way. Do you have a friend you can borrow a rifle from to check how it's recoil affects you? I think my choice would be a 22-250, both of mine are deadly accurate.