I don't believe it all is just being caught either. just like other animals some just get bigger.
Of course not. Some fish are genetically predisposed to grow faster, live longer, whatever it takes to be larger than most of the other fish.
However factors like catchability and food supply play a big role too though.
Case in point: Caught a yellow perch 5 oz. bigger than the state record and just over 16 inches in one of my ponds back in 2005 ice fishing. (This before I started raising and hatching my own fish.) State would not allow it to be entered citing I feed my fish. If feeding was the only factor I would have produced another one. I have not done so in 15 years. There was something about this fish genetically that caused it to get to that size.
OTOH most of my brook trout in the pond exceed the Indiana state record. That's because brook trout have a short life span and feed is not as plentiful in the wild as it is in my pond that I fish. Also the environment for brook trout in Indiana is suboptimal.
Ironically as anglers we can be our own worst enemy when it comes to a body of water producing big fish. We tend to harvest the largest fish.
Interestedly evolution favors that largest fastest growing fish for survival as they are quicker to avoid predation by their larger size. We as anglers tend to short circuit that.
I have a survey of Maxintuckee Lake done by 2 California professors for the then Department of Conservation at the turn of century (1900) It mentions 14 inch bluegills!