One thing my gunsmith does when accuracy testing a rifle he's built is (1) take the sling off, and (2) take the front sling post out of the stock. Both can hang up on the front rest in recoil and throw your shot and render useless any other consistency in hold that you have. Honestly, I get lazy and usually don't take the sling post out, but make sure it's set far enough ahead of the front rest that it won't hit in recoil.
For the light kickers like the 17 Rem and 243 Win, I always kept my hand off the forearm - as others have suggested - and always seemed to shoot better. Never liked doing that with the harder kickers for big game, for two reasons. One, the muzzle jump off the front rest, and two, I would always be holding the forearm shooting at big game so I wanted that consistency in hold between aim-in at the bench, and in the field. With no left hand now, the rifle forearm rests on my forearm, at least shooting offhand, or on a front rest of some sort (pack, log, etc), much like at the bench. Who knows, maybe I'll start shooting better!