It's a matter of degrees...

Put it this way, if somebody is sitting on your computer, watching without telling you, it's the equivalent of having a camera pointing at you all day, including during your break or lunch when I can legitimately check my email. I have a position of responsibility and trust, and tend to believe that mutual trust and respect leads to a much more productive working environment.

Personally, I think if somebody is going to do this, s/he should give you a call first to say they are dialing in and why. It's basic professional courtesy.