Two men - Ingemar Johansson and Sonny Liston - have had a tremendous impact on Patterson's place in boxing history. Had he not fought Liston, he would have been highly regarded on the strength of his trilogy with the swede. What he built with those fights was destroyed by his two-time anihilation and the hands of Sonny Liston, which placed him right back to where he was before he fought either man, plus the fights that followed, including the two with Ali.
In summation, If Liston never happened, Patterson would have rated much higher; if Johansson never happened, he would have been rated even lower. Since Liston and Johansson did happen, they cancel each out as if they never did, which leaves us with Patterson's perceived underrating.