The kind of magnet needed to damage a hard drive would be a large and powered electromagnet, with an oscillating field. It would also have to be fairly close to the laptop's bare hard drive, within a couple of inches. A simple refrigerator magnet, or one used to close cabinets, would never be capable of interfering with modern hard drives.
To give you an example, try taking an old hard drive apart, preferably one that's already broken. You'll find a ridiculously powerful pair of magnets ALREADY in that drive! Hard drive manufacturers use balanced magnetic fields from high-gauss rare-earth magnets for the drive's servo arm. Rarely in the consumer world will you find magnets that powerful, with several pounds of pulling force, in such a small area.
To make it short, don't worry about that little magnet. It cannot hurt your laptop.
(Oh, and once you pull those magnets out of that old hard drive, try not to put your skin in between the two. Some of those are powerful enough to pinch the bejeezus out of you.)