Primarily the fact that he was leading Germany into an unwinnable war was the reason. The early attempts prior to his invasions were more politically motivated. I beleive. I'm sure there were those who were privately horrified by the genocide but most Germans today will say with honesty that even if they had heard about the camps, they either disbelieved the rumors or didn't oppose them very strongly. Especially because by that point winning the war was more important to the german people.