"The opposite is not true however. I don't say this to "keep score", I say this because there is a whole inside of me that feels hurt that my wife doesn't "want" to show me much affection, and also that doesn't receive affection from me with enthusiasm I guess? I'm not sure what I mean exactly, but from my perspective, when I show my wife affection she seems indifferent to it. Its like I could be rubbing her shoulders or not rubbing her shoulders and she's totally fine either way. I'm not sure what this means, while I feel this way about it, but I feel compelled to figure it out because its driving me crazy."

Maybe you still haven't it on the things she likes? You still seem to be judging a lot based on what you like and expect. For example - my partner likes to be lightly stroked (like caressed on the arm) ( He tells me he likes it). I don't like that most of the time - I prefer a much firmer touch. The light stroking is actually irritating to me, so I need to remind him to apply more pressure when he forgets (as he does often because we all tend to do what we like). If he were going to judge my worthiness based on my reaction to what he likes we would be in constant conflict.