Sounds like the doctor isn't taking you seriously. I agree with above posters. Get a second opinion.

One thing to remember about people is that they're amazingly bold and brave when not confronting you in person. I bet the guy was full of it, or worse, perhaps one of those people who believes in conspiracies and thinks the government is out to destroy him. I hope you don't have to field such calls for a long time after this.