phoenix, do you remember what you told him in your plan B letter? You gave him the steps he needed to take to come back into your life, ie: end his affair, etc. By allowing him to come back without meeting that condition, you are giving him the impression that you were not serious. You are forfeiting your credibility.
And if you won't take your boundaries seriously, neither will he. If he thinks he can have you and the OW, then he will pursue that, and will have no motivation to end his affair.