If you cannot accept the fact that handloading allows a shooter to customize his cartridge for whatever use he has, then you need to go back to shooting factory loads....If you have to slow down a 220 Swift, then it is not needed! Move on to a smaller and "slower" cartridge, and leave the Swift to those that can appreciate it for what it is.
Sure, 1/4 moa with a worn barrel....now just how many “very bad” 1/4 moa groups at 200 yards did you fire? No need to embellish dude, the Swift is a great cartridge...which benchrest shooters abandoned well over half a century ago....My last 220 Swift was a rear tang Ruger M77 Varmint that had over 2,000 rounds shot through it before it was gifted to another varmint hunter, and on a very bad day, it would still shoot 1/2" or less at 200 yards with those 55 gr. Sierra bullets.