For me, William Hartnell, Tom Baker and (I know I am going to get yelled at for this) Sylvester McCoy are "The Doctor". The others are just actors playing the Doctor.
I still can't bring myself to watch "Carry on Sergeant" or "Brighton Rock". Seeing "grandfather" in a different role would be too much like finding out that Father Christmas doesn't exist.
Nothing against Jon Pertwee. Jon Pertwee was the first Doctor I met. For a couple of years, he was just "the Doctor." When you said "Doctor Who", you meant "suave space traveling buccaneer with a slight undercurrent of buffoonery." It took me a while to get used to the guy in the scarf. But Pertwee, with his flashy cars and gadgets, with girls who were referred to as "assistants" (rather than "companions") and a chorus of square jawed soldier-boys... If this was "the Doctor" then none of the others were. Much as I like the UNIT episodes as episodes -- the sparring between the Doctor and the Brigadier, the wierdly respectful relationship between him and the Master -- I tend to think of it as part of a different series.
And nothing against Patrick Troughton. Colin Baker and Sylvester McCoy have both said they think he was the greatest, and considered as a performance, I think it was. Where Tom Baker waves his hands around a declaims, Patrick Troughton under-acts. You spend the first half of an episode thinking "what is supposed to be so great about this guy: all he is doing is reading out the lines". And then you suddenly realise that you believe, totally, and without question, everything this funny little man tells you. He's 400 years old, and there are Yetis in the tube. Fair enough. If you say so. But the one thing I didn't believe was that he was the same person as the white haired old buffer who kidnapped Ian and Barbara in episode one.
Maybe he wasn't. The metaphysics of regeneration have never quite been sorted out. Are we supposed to think that it's the same person, but with re-jigged cells and a slightly scrambled personality? Or has the Doctor mind slinked off into a different host body, like those slug-things in Star Trek? There is a strange undercurrent in some versions which suggest that the different Doctors are actually different people. The novelisation of "Tenth Planet" has our hero announce "I am the new Doctor" as if we were talking about someone taking over a role or position. The novel of "Five Doctors" seems to envisage the First Doctor "living out his days" in a restful garden, even though we know that he "died" at the south pole. And while it is not exactly canonical, "Dimensions in Time" has the Fourth Doctor sending out a message beginning "calling all Doctors", and thinking of his previous incarnations as other people -- "thank you, my dears."
It is harder to defend Colin Baker and Peter Davison. I rather liked what JNT was trying to do with the Davison era: three companions, instead of just one; expanding the TARDIS itself beyond a single control room; one companion in league with the bad guys; another one killed off; breaking some of the rules. But casting a young, good looking, heroic type, already know from many other roles as the Doctor was fundamentally misguided. Davison didn't stick in the role for long enough for us to get used to him. As for Colin Baker, the most that one can say is that he could deliver lines with great gusto, and it would have been interesting to see what he would have done with a decent script.
And then there was Sylvester. Sylvester McCoy started his tenure as Doctor Silly. He spent his first season being suspended over vats of boiling sugar by Bertie Bassett, and threatened by evil caretakers and their killer cleaning machines. He was stuck in an identikit "Doctor Who" costume, and JNT was still convinced that it was cute to use question marks as a motif on the Doctor's clothes. (It is regarded as an awful faux pas to refer to the Doctor as "Doctor Who", but evidently, it's okay for him have question mark umbrellas in the TARDIS wardrobe.) Yet somehow he struggled against this nonsense, and gave us a Doctor who was interestingly different from what had gone before. Darker; more manipulative; traveling the universe according to some purpose which he knew and we didn't. ("Ace tells the Doctor about her worst nightmare. So he takes her there" has got to be one of the the great Radio Time blurbs of all time.) Fans now know that the script editor, Andrew Cartmell, was engaged in the first stages of a "masterplan" which would have emerged over several seasons and put a new spin on the characters of the Doctor and the Time Lords. Some of these ideas were subsequently incorporated into the first cycle of post-TV novelisations, ("The Virgin New Adventures") and they seem pretty ghastly to me; another boring re-working of the very boring back-story to the very boring Time Lords. But what emerged in the transmitted episodes was rather wonderful; the hint that underneath the clownish exterior was Doctor as cosmic-entity who could stare down a villain by force of will. When you looked into eyes, you half-believed that he really was far more than a time-lord; that there really was a big dark secret just wating to be revealed.
The character of the Doctor is weirdly flexible; but not infinitely so. We can't define it, but we know it when we see it.