Oldbull,
You don't mention what range you are shooting at or whether or not you are using a scope, open sight, aperture, etc. As CD said, first shot with a cold bore can be consistently different, which is why a lot of people will fire a "fouling" shot before shooting for group. Then, the preferred method is a 3-5 shot group. If you fired a fouling shot and then 3-shot groups, you might not have ever noticed the odd 5-shot phenomenon, which probably is due to the barrel heating up. This depends somewhat on your rate of fire, but it sounds to me like the solution you're looking for is to shoot 3-shot groups, with time for the barrel to cool, in-between.
One of the reasons for free-floating a bolt-action rifle is to eliminate pressure out at the tip of the stock, especially as the barrel heats up. Not really an option with your gun, but it does serve to illustrate what you are probably seeing. Try shooting 1 shot, then a 3-shot group, then letting the gun cool for 10-15 minutes...then another 3-shot group. Let us know what you find.
Jason