You should not be driving your son to his father's house. That's ridiculous. If his father doesn't come to get him then so be it.
My custody order says that my ex gets my son Friday night through Monday mornings every other week, plus Wednesday nights through Thursday morning every week. This adds up to 10 nights every 4 weeks...over 1/3 of the time. Guess what? He stopped taking my son on Wednesdays about 6 years ago (when he got remarried), and stopped keeping him Sunday nights about the same time....because it was "inconvenient" for him to take him to school. It was only inconvenient because my ex moved about 30 miles away from me. His loss.
My son, who is now 13, fully understands that his dad is a flake and can not be depended on. I have NEVER said one bad word about his father, but I have not shielded him from the truth either. I used to tell him that his father wasn't coming because he was working late or whatever. Finally I just stopped making excuses for him and started letting him listen to the voice mail excuse messages his father would leave on my cell phone. He gets it now. It takes a while...but they know who takes care of them.
I would suggest you stop transporting your son and let his dad get him if he wants. Don't provide the transportation for him.
I would also go back to court and get a real child support order. If you were in Texas he would be paying 20% of his income (only his income, not his wife's income, and it's regardless of other children), plus health insurance, and YOU would be the only one legally entitled to claim him on your taxes. It sounds like you might be better off with a standard custody order than the "arrangement" you currently have. Fight him for everything if he's going to be such a jerk about it.