Quote
he twists and turns it all to make it out like its nothing and that I'm overreacting

That's classic gaslighting.

Quote
I'm just saying that I think I'm going to have a hard time getting past all that

This is a very important thing to work out in MC. Frankly, I think it's something your WH needs to hear from you in a deadly serious tone of voice. BTW, have you ever asked him if he has any idea why you had an affair?