I would opt for the .45, but you don't have to jack the velocites up to be effective. My SRH in .454 really likes the 347 grain hardcasts I have used as well as 360s. I have even killed a couple of critters with 400 grain bullets in my Casull. The key is keeping the velocity moderate.

I too use a red dot on my .475 SRH -- an Ultradot 30 ($135.00 shipped). They are great in low-light situations.