I am a betrayed spouse too, so I can't identify completely with your husband. I am a man, though, and I know that I have trouble sometimes with non-sexual intimacy. I don't think it's uncommon for guys to equate sex with loving feelings. He may not understand what took me a painfully long time to learn, that husbands and wives have very different natural ways of feeling love and showing love. When after D-day my wife was slow to restart our sex life, it was hard for me not to reach the conclusion that she doesn't love me. She needs hugs and cuddle-time like you do. If it has been a while since intercourse, I can be afraid to have a cuddle session, because I know what I'll be thinking about. It isn't so much a problem when we're on track. I've learned that I'm not a neanderthal to feel this way. I've also learned that after an affair, sexual intimacy may be slow in coming back. During that time, as hard as it was as a BS, I had to try to meet that need of hers.
I am NOT suggesting that you renew SF before you're ready. We tried too soon and it set us back. I hope you can help your husband understand that just because you're not ready to again enjoy that particular part of your relationship, it doesn't mean that you don't love him or that you don't want to work on the marriage. Explain to him that for most women, SF cannot be truly enjoyed unless the couple is close emotionally. Together you can rebuild that emotional bond, and the rest will take care of itself in time.
Please don't think there is something wrong with you. Don't withhold SF to punish, but you're not weird if you do not feel ready for that kind of closeness yet.