Are you supposed to leave the "collet" (pilot) behind? Yeah. If you mean is it supposed to pull out, certainly not.
You can stop that by grinding or filing a small flat place on the pilot shank that will allow the set screw to have a secure place to set into.
The "tight" case mouth fit comes from from the cutter forming a burr that turns in under the pilot. You can reduce that effect by 1) insuring the pilot is fully inserted and 2) by drawing the cutter/shaft out to turn the burr a few times as you work.
After I used my Dremel tool to grind flats on the pilot stems and to withdraw the cutter occasionally, I love my Lyman Universal. Had it for over 30 years now and the shaft is a slighly loose fit, shouldda kept it better oiled.
Course, if I'd known I'd live this long I would have taken better care of a lot of stuff, including myself. Ah, well...