The first thing to do is find out if T-Mobile still offers voice/SMS-only plans if you provide your own phone. They allowed me to use a jailbroken iPhone without requiring a data plan but that was almost two years ago.
The next thing would be to make sure the used phone you're planning to buy is (or can be) unlocked. Just because a phone is out of contract doesn't guarantee that it's unlocked. If the phone was originally used with T-Mobile service you may be OK, but if it came from another carrier it may or may not be unlockable.
Keep in mind that AT&T and T-Mobile's 3G data networks are incompatible. If you are able to activate an old AT&T smartphone on T-Mobile - and later decide that you do want a 3G data plan - you'll need a different phone.