While YOU should enforce your personal boundaries... you should be thankful that he is willing to spend some time with you. This does not mean you have to be a doormat or engage in risky sexual behavior though.
You also need to keep in mind that while he is being kind about this right now... he ceratinly has the option of taking his daughter with him when he goes.... keep that in mind when you are dealing with him concerning your child. Do what is in her best interests right now and that means spending time with her father and not saying anything bad about him. I don't remember how old your daughter is... but if it is age appropriate you might want to let her know that you acted in a way that hurt daddy and that he is trying to cope with that. Kids need to know what is going on and not be lied to about things.