He's not big at 147 but he's not small either. He's just sort of in the middle to lower these days. Certainly has never struggled because of any size issue and has never really had any size disadvantage at all anyway, apart from Oscar at 154 when he had only been at 147 for a short while.
It's like saying Duran was small at 147. He faced bigger guys than Floyd did, he was shorter, with less reach and started at a much lower weight too, and was already showing signs of being flabby and soft at 147 (apart from against Palomino and Leonard) but while he certainly wasn't an average solid guy there, he wasn't small either.
For a guy that's 5'8" with a 72" reach, 147 is not out of his depth at all. His one downfall was that he didn't have a big, solid frame to go with his height and reach. He's developed into a solid, pretty averaged sized 147 pounder now IMO.