Hobbies And Interests
Cost
Might as well mention it right away. Even the fanciest, most expensive optical system will have some sort of cost limitation. For anything less than the fanciest, most expensive optical system, it's very likely that cost will be a very important factor, and for some systems, cost is THE most important factor. The larger the lens, the higher the cost. So for cost-driven applications, the trick is to make the lens as small as possible. Take a look at the lens on your cellphone camera. Now you know why it's so small.
Light-Gathering Power
Ever taken a night-time picture with that cellphone camera of yours? If so, you know one limitation of small lenses: they don't gather very much light. If the detector in your cellphone was replaced with a very sensitive, very expensive, low-light CCD then you could take high-quality night-time pictures. But if you want to have the chance to make good images with little light, then a larger lens is the way to go. That's why larger aperture telescopes are better: they collect more of what little starlight there is.
Resolution
The next issue associated with lens size gets a little tricky. It's a consequence of the physical nature of light. Without getting tangled in the math and physics, the simple fact is that it's impossible to focus light down to an infinitesimally small spot. Put another way, a spot of light can only get so small. The tinier the spot your lens can create, the more detail you can see -- the less blurring, if you want to think of it that way. The larger the lens, the smaller the spot can be.
F/#
The F/# (pronounced "f-number") of a lens is the ratio of the focal length to the diameter. Very small F/# lenses are very highly curved, very expensive to make, and very sensitive to errors in material or fabrication. So if you want a 500 mm focal length, you might want a 500 mm diameter lens to maximize the amount of light collected, but that's an F/1 system. F/1 lenses are really tough to make well. It's more reasonable to use a 125 mm diameter F/4 lens. But, as with everything in optical design, it's all a tradeoff among things that are important to you.
Weight and Thickness
Lenses most often have a uniform curvature. That is, their front and back surfaces are like parts of a sphere. Because of that, the larger a lens gets, the greater the difference between the thickness at the middle and the thickness at the edge. But if the lens is really thick in the middle, then it's really heavy, so the edge can't be made too thin or the lens won't be able to hold itself up. So the edge needs to be thicker. But that makes the whole lens even heavier and that much more expensive and difficult to handle. That's why the largest usable telescope lens is the 40-inch diameter lens at Yerkes Observatory.