In fact an actual study was done- I remember reading it somewhere but I dont have all the details.
It basically rated sports on how demanding they were in terms of:
SPEED
STRENGTH
STAMINA
HAND-EYE COORDINATION
and they conculded that boxing was the most demanding sport. And stated that it's the hardest sport to succeed in because you need to be near perfect on every count.
Wierdly F1 racing driving was near the top of the list too
:S
Depends on the individual, I guess.
Boxing is voted the nr.1 demanding sport in front of hockey and american football.
But most demanding and hardest are two different things.
I'd say there's alot of sports that's harder than boxing, but none more demanding.