It’s Martin Luther King Jr. Day, a day on which Democrats and Republicans alike will revere the great civil rights leader, a day on which you are likely to hear again and again the same few fragments of his I Have a Dream speech. It’s a day on which we must remember that during his lifetime, King was a divisive figure, someone who Gallup never found to have majority approval, someone who 31% of people told a pollster had brought his assassination on himself.

There are reasons for that: The cause of civil rights for Black people was itself not overwhelmingly popular among white people, for one. And King never stopped pushing the nation to do better. If he had lived decades longer and kept pushing, Republicans would still probably hate him.

On Martin Luther King Jr. Day, let’s remember the full scope of King’s politics, not just the oh so convenient, out of context, “I have a dream that my four little children will one day live in a nation where they will not be judged by the color of their skin but by the content of their character.”

Let’s remember that when King spoke out against the Vietnam War, as he did repeatedly, both The New York Times and The Washington Post editorialized against him, with the Post charging that the speech had “diminished his usefulness to his cause, to his country, and to his people.” And remember that not just when you roll your eyes at Republican invocations of the man’s greatness, but when you question the tactics of today’s movements for racial justice.

King’s focus in the final years of his life on poverty and its relationship to civil rights also drew criticism, which he fully understood, saying in May 1967: “You see, the gains in the first period, or the first era of struggle, were obtained from the power structure at bargain rates; it didn’t cost the nation anything to integrate lunch counters. It didn’t cost the nation anything to integrate hotels and motels. It didn’t cost the nation a penny to guarantee the right to vote. Now we are in a period where it will cost the nation billions of dollars to get rid of poverty, to get rid of slums, to make quality integrated education a reality. This is where we are now. Now we’re going to lose some friends in this period.”

There were of course those who brutally opposed integrated lunch counters and hotels and the right to vote—one major party is still committed to limiting voting rights, though with less personal violence than in King’s lifetime. But as King saw, those changes, however important they were, threatened fewer people’s complacent comfort than calling for the nation to fix the economic results of centuries of slavery and segregation. That’s a fight we’re still having, and that’s one reason why if he had lived, King would likely still be a controversial figure today rather than a depoliticized saintly figure.