It's a bonding thing. When they wear one of your shirts, it can be more of a primal thing - they want something that smells like you, especially if there's a hint of your cologne, or just a hint of YOU on it. Also, the whole "what's yours is mine" thing is at work.
"The best kind of pride is that which compels a man to do his best when no one is watching."