Patterson's legacy gets hurt because he was utterly dominated by the best competition he faced at heavyweight. That being the two Liston fights and the 1st Ali fight.
Because Liston and Ali dominated him in different ways in terms of style, it hurts him. Frazier's losses to Foreman are chalked up as a bad matchup. Patterson doesn't get this excuse for the Liston fights because of how Ali beat him. Rather, it makes it seem that he clearly doesn't belong with the top heavyweights of all time.
I think it's slightly unfair, he was never going to beat Liston, but I'd have like to see a prime Patterson face Ali. By no means do I think he would have won, but he might done enough to save face and make the Liston fights be viewed as bad matchups.