This is why you do not confront the WS until there is irrefutable proof. And when you do, you dont give them EVERYTHING that you know. That way, the next time you bring it up, you can give them a little more. Eventually, they will think you know everything, and they will spill the beans.
In law enforcement, it is how they get admissions of guilt many times. The investigator will take known facts and slowly let them out over time. The suspect will be stonewalling for awhile. But eventually, due to the little info that has been brought out by the investigator, as well as "bluffing" by the investigator and insinuating that they know everything...the suspect ends up thinking there is nothing left to lose since they know everything...and they spill their guts.
You have to be smart and strategic. You have to be coy, to make him think you know everything. He will then question you on what you know. Of course, you dont tell him. You just say "I know a lot...and I know enough."
In the meantime, you continue to gather the data needed to build your "case." When law enforcement is getting info on a suspect that they expect to arrest, they do so methodically...but dont let the suspect know what they are doing. Law enforcement also knows that eventually, the suspect will slip up and say something or be careless in their actions. They ALWAYS do. And then, they can make the arrest.
So, be quiet about this stuff right now. Start gathering evidence. Get recorders/GPS equipment/internet monitoring software, etc. Hire a PI. Or get a good friend to tail him a few times. You know your situation, so tailor your intel gathering to it.
Plan A your butt off. But part of that is exposing the affair. But you cannot expose what you cannot prove. So, get the goos...quietly. Then, when you have him in a corner, you let it out...and he will admit to everything...even what you didnt know.
In His arms.