You initiated the conversation, so you are in control. It is not going to be a discussion, you will lay out the map and what route you are willing to take. If he wants to take a wrong turn, let him know that your route is set and he knows what it takes to join you.

He has faked to invest in the relationship with boattrips and presents, but never put in the effort to really make it work. Don't be fooled if he is trying to buy you without doing the work. Do not negotiate. Either he is on board, or you are on this route without him. The safety of your children is not negotionable.

One of the guys at my work told he stopped being a fool after he woke up while his 3y/o son tried to drink from left over beer bottles, while he had fallen asleep after using marijuana.
You won't want your children to lick up 'sugar'.