Am i being too sensitive about my fiance saying he is turned off by me?

So we were touring gyms today. He made the comment that I should go to the gym with him from now on. I told him that I hate gyms and the idea of sweaty people everywhere and people watching me work out is not my thing. He then said 'GOD, that is such a huge turn-off, if not the biggest." And scoffed at me like I was gross. Lately he has been getting on me about working out. I mean, I'm in decent shape as is. I only weigh 125, but I know he wants be to have a super-hot, trimmed body. So that comment offended me and kinda made me depressed. It makes me really self~conscience about myself, which is already added on to the fact that I already hate my physical appearance as is. And it makes me feel like he doesn't think I'm good enough for him yet and he is just settling for me and has to work with what he has. He thinks I read into the comment too much and I'm overreacting. Am I or is he genuinely a jerk?