I'm really tired of the "fat" stereotype... so I'm going to apply my favorite thing to it: Math.
People seem to think that being fat means you eat grossly more than most people; fact is, it's simply not true. To gain five pounds in a year, you need to eat 48 calories extra per day. For reference, 48 calories is about half a slice of whole-wheat bread. That half-slice isn't exactly pigging out--it's little things, like a couple of extra potato chips or a whole apple instead of a half. You won't see someone gaining weight at the rate of 5 pounds a year eating appreciably more than someone at a typical weight level. And if you gain 5 pounds a year, and if at age 16 you weigh 110 pounds, then by the time you are 35, you will weigh 205 pounds... a morbidly obese BMI for anyone who was normal weight at 110.
The activity stereotype isn't very good, either. The idea that fat people sit around on their butts and don't do any exercise is pretty common, but the fact is that many fat people do exercise, some even as competitive athletes. There's nothing stopping you from gaining muscle under the fat, nor does extra weight stop you from increasing your endurance. Let's take that 110-pound person again. If they gain five pounds a year, the difference between their activity level and that of a person who gains nothing is the equivalent of taking a single flight of stairs instead of the elevator to a job on the second floor. If those two people eat the same and exercise the same otherwise--whether they run five miles a day or do nothing else--the person who doesn't take the stairs still ends up at the same 205-pound weight at age 35. (Eat 48 calories extra AND skip the flight of stairs, and you end up at a whopping 300 pounds!)
Or, y'know, it could be a natural difference in basal metabolism. 48 calories per day difference between appetite and energy output is not that much of a gap, and yet, it all adds up (see above).
The differences between lifestyles for people who gain weight versus people who don't are tiny differences--not nearly enough to be observable unless you were to look at the less typical case of gaining twenty or more pounds in a single year (at which point the difference amounts to an extra candy bar or a mile of walking per day). You simply can't say that this is a defect of character--not unless you want to assume that anything but perfection is unacceptable!
Of course, the naturally skinny people get it, too. I don't know what it would be like to be really skinny and assumed to be anorexic, but that can't be any easier. If you think about it, that's basically people assuming you are mentally ill when you are not, and that can be extremely annoying. How many times have you guys had people assume AS is a mental illness or been asked whether you are "taking your meds"? It's bad enough to have to deal with if you have a mental illness, let alone if you don't.