I avoided going to a doctor for about 15 years (mid-30's through 40's) and finally went when enough of the people around me looked at a mole on my face and told me 'uh, I'd have that looked at.' Turned out to be nothing but they did find my BP was sky-high and now I have to check in pretty regularly.
As for my comfort level, I just feel like I have to be a little pushy about what kind of treatment I'll deal with (like, I'd rather talk about dietary changes than to start another pill). I think he's genuinely interested in my health but I don't always feel like he wants me to be an equal player in making the care choices.