He has an incredibly sexy foreign accent (definitely have to reduce, maybe eliminate phone time)
I would eliminate all contact, via phone or in person unless your husband is by your side and even that might need to change in the future.
It is the innocent contact while getting some EN met that leads to falling for someone. You don't even have to knowingly participate in flirting and sexual banter for it to get to a point where stopping is hard, too hard when our own spouse is not meeting some specific EN for us.
And we don't even have to have a reason in our mind that would justify it to begin with, we just have to recognize it and run from it.
There are going to be people from time to time that come into our lives that we just simply hit it off with as if there was some magic in the moment. It might be a strong physical reaction/attraction or just a feeling of "this person is really cool." In almost every one of these cases nothing will come of the attraction and we just move on with our life. But if we linger in the moment, if we begin to think of "what if" in regard to this person, we need to recognize that at once and move away quickly.
If we are getting something from this person in the way of ENs of any kind whether attraction or conversation or honesty, it can become quite easy to start to justify continuing down the road we know we are on. An attraction becomes a crush becomes an obsession becomes an affair with no real concept of how we got there.
There was a thread a while ago about the ten second rule. In the early stages of any relationship we make conscious decisions to move one direction or another in the relationship. When something is leading us in the wrong direction, we have about ten seconds to make up our minds. We either back away from the inappropriate behavior or we engage in it and once that road is in front of us, it becomes harder with each step to turn around and run.
So when my cute 20 something coworker suggests I join her for a drink after work, it is at THAT instant in time I need to decide to remain faithful to my wife. It isn't that I necessarily want to have an affair with her but simply going out for drinks with a women that could lead to a problem. Or when that new girl in the office touches my arm as she hands me a cup of coffee fixed just the way I like it, that is the moment when I decide if I am going to remain faithful or if I am willing to flirt not with her, but with disaster for my marriage.
It isn't the first ten seconds in the relationship that count but the first ten seconds in which we know there could be something more than the innocent act before us. That is when we have to see it as dangerous and step away.
My wife seldom travels for work, but it does come up sometimes. But if she were to suddenly have to begin traveling with one of the men she works with, it would be a huge problem for me. It isn't even a matter of trusting her, but that I could identify a man and woman travelling together, spending hours together in a car, eating together, sharing strategies for business together etc as being a huge potential for a lowering of boundaries. Her affair in 05/06 was not with someone she would normally be attracted to. The only thing they had in common was talking about idle chit-chat at the mailbox or over the back fence. I could see the trouble ahead but she couldn't and she ended up ready to divorce and leave me for a guy she can't even speak the name of today without gagging. She let her boundaries down because it was all innocent behavior and it nearly led to disaster.
Ultimately it is having boundaries that prevent us from heading toward a problem more than a line we will not cross that will prevent us from doing what we know we should not and had no intention of doing when we began. It isn't our bodies nor our hearts we have to stop, but our minds...
Affairs happen not so much when we act, but when we don't.