Is it the girls responsibility.....?

My boyfriend and myself had a huge discussion and sorta fight about this last night. (We are both in our 20's) He feels I am not in the mood enough to fool around with him. We have jointly made the decision to not have , but we do do other stuff. He expressed that his body is on another schedule from my body and that he needs to be "taken care of" more often; and basically it's my responsibility to make sure that happens. I feel that it isn't. If I'm not in the mood to fool around, then I'm not in the mood and too bad for him, deal with it. He thinks that it's unfair that I pretty much have all the control when it comes to physical stuff. I just think that I'm the girl, that's just the way it is. Am I crazy, or he is wrong? I am confused....