Hanging out with miserable people makes you miserable too. You're constantly trying to bring him up with you and all he does is fights it.
You may consider hanging out with him less/cutting him loose.
You may consider hanging out with him less/cutting him loose.