AJ,
I think it really depends on how and where you hunt. If you are a stand hunter and where hunting pressure is really high, it could be a factor. But if are a still hunter or spot and stalk hunter, you can always adjust to keep the wind to your advantage. If hunting pressure is low in your area, the game may not be as suspicious of vague foreign smells. After all, the oil on your gun or the breakfast on your breath will probably be as strong. I use leather military type slings on my rifles and wear leather boots and belts, and have never had problems (that I know of). All of my hunting (including southern white tailed deer) is done by still-hunting or spot and stalk. Always keep the wind to your advantage and it shouldn't be a problem.