I personally think it's because there are fewer guys at or near the top with the heavyweights. Sometimes if I'm watching FNF, I'll watch two smaller guys who are less experienced or talented, and I have a hard time dealing with it because it's so ugly, so it's not like there aren't guys who don't have good skills at the lower weights. We're spoiled with the top guys who do it so well... the Mayweathers, Cottos, etc. who are at the top of deep divisions with a lot of guys fighting.
Why is this? I tend to agree with the other sports causing trouble. Football and basketball basically demand that you be big for most of the positions, and if you are good enough to be wanted, they'll hand you tons of money up front. Baseball maybe not so much, but it helps in a lot of cases. In boxing, your size doesn't matter, you can go out and do it. A guy who weighs 140 and wants to do sports can't even look at football or basketball, so boxing suddenly is more appealing. And with the money, there isn't that much to be made if you aren't all that good. Sure you'll get a couple thousand bucks, but you've had to pay for your training too, it's not going to make you rich.