Hi Mourning82...
You're in a tough spot because your M never got a chance to take root before all the collaspe. The hurt feelings your H is having is his "ego and pride" as in any man. It hurts when you know your W has been PHY with another man. But it can be overcome.
If he looks at you while you're preg. he will look at you more after the child is born. I don't know why he will wait to leave after the child is born? Maybe he wants to see if it looks like him? That's natural in your case.
Your greatest healing tool, is to denounce the past OM to your H and let him know OM was noway better than him, but just an emotional run to.... You have to denounce and degrade OM to your H (I mean degrade him as the slim of the earth! more worthless than the snot that drips from a cows nose, a narrow minded sewer rat, he's smaller than the worm in a chipmunks nose, ok you get my drift) build up your H with acts of love, words, go the extra mile, break soul tie with OM, claim your home back, create a new dream and vision with your H. Learn his love language. Find ways he can trust you, be open. Men don't walk away from sex so easily unless they have something else to run to, so don't get so caught up in he's leaving if there's no one in the picture.
Men, when hurt say I'm leaving to run from the pain, not always the person, they'll find a person to medicate there pain, called a rebound or affair, until it wears off then they'll go back home. I would find healing words and actions. When a man knows he can have sex with a woman, but she also solves and validates issues he's dealing with, he won't go to far. Most men are kept by women who believe in them....
Take your counseling tools and use them to the max.
In looking at how fast marriages end today. I would tell everybody who's married or plan to, "Plan to have a marriage ceremony every 3 years and renew and rededicate to one another in great love and passion". This keeps things fresh and exciting. You used to be able to wait 15 or 25 years for this, but not today.
So work things out and ask him what about starting over, he is going to try to do this with somebody, it might as well be you.