Often it's people raised there and never moved away
and of course there is always towns that have a job source, and it goes away. Lots of smaller towns or rural areas may have one large industrial mill/plant/factory that supports most of the community when you include all the business there just because the primary is.
The primary closes, or just drastically reduces it's manpower, and you lose a lot more jobs just the primary losses. You end up with a lot that have a difficult time moving, home investment, no money to move, etc.