There's more to do with what drugs Ali was on leading up to the Holmes fight then there was with Larry Holmes.
In all fairness, Frazier did put Ali out on his feet in their first fight, and Ali himself said he was going to quit in their third fight, had Frazier's corner not called it off. Even so, Ali missed much of his prime fighting years, and came back to dominate the deepest heavyweight division in history. He beat the hard working, the seemingly invincible, and the proud.
Top ten lists are supposed to include title reigns, level of opposition, dominance, with head-to-heads way down near the bottom of the list. I, for example, rate Joe Louis as #1, with Muhammad Ali as #2, yet I firmly believe that Ali would cut Louis up and down the ring. There is also no proof whatsoever that the conditioning of fighters from recent years would give them an edge again the fighters of old. Joe Frazier, for example, worked so hard during the course of a heavyweight fight, that he would tire out almost any fighter from recent years. He simply punched and slipped and punched and slipped until his opponents couldn't go anymore. Even the good heavies from the 90's didn't carry that sort of work ethic, short of perhaps Holyfield.
Louis was not a boring fighter, by the way. He was very tactically sound, and could figure out almost any opponent. His only "real" loss was avenged in brutal fashion, and he didn't lose again until waaaaaaaaaaay later in life, after serving in the military, against two guys who are definitely not bums.