Do boxer punchers generally struggle against speedy swarmers?

I don't mind slow brawlers or swarmers at all. These types give me time to unleash my shots and they just get caught coming in every single time and have to start over... It's only the explosive, joe frazier esque ones that I can't control. They disrupt my jab and put my reflexes under much more stress than I'm used to. The angles employed also annoy me. It feels like my fundamentals get turned upside down. It's just hard to keep track of everything, I feel. Is this typically the style of fighting boxer punchers dislike? Note that I do catch them coming in, but at the end of my punches.... not enough to be safe.