Fighter's should fight the best, but not until they are ready. One loss can derail a career. Loma is an exception that was ready almost right away. With most guys that isn't the case.
Manny in his prime, with knockout power, had a chance to catch Mayweather. Unfortunately it doesn't appear he quite has that anymore and he won't be outboxing money for 12 rounds.
Walters said basically the same thing on Tuesday at the open workouts. Something along the lines of, I've been boxing for my entire life and if you aren't going to cash in when you're fighting the best then when will you? He has a very valid point.