Define the Requirements
The first job of an optical systems engineer is to define system requirements . This is THE most important step; cutting corners here will haunt you later. The requirements will get refined and modified as the design progresses. Define the requirements comprehensively as soon as possible.
For example, take that door peephole. You could say the requirement is to allow people inside to see people outside the door. But then a glass door would be the simplest solution. So the real initial requirement would be something like, "allow people inside a door to identify anyone within 6 feet of their front door without allowing the person outside to see inside."
Constraints
Identifying constraints is an extension of defining the requirements. For example, the peephole would have a constraint like, "the system must be small enough not to compromise the structural integrity or fire barrier of the door." Every optical system has one overriding constraint: cost. Even the most expensive systems have a cost limitation somewhere.
The requirements and constraints start out as ideas that the optical systems engineer translates into numbers. So the requirements for the door peephole could be something like: "a passive imaging system sensitive enough to provide high contrast under light from a ceiling fixture, weighing less than six ounces with a circular field of view of 80 degrees, with an overall system length of two inches, a diameter of half an inch, and angular resolution of half a degree."
Throughput
One of the primary concerns for the system design is the optical throughput. It's theoretically possible to build an optical system with extremely good imaging characteristics, for example, but one that doesn't deliver enough light to the eye, film or other detector. How much light will be present in the conditions under which the system will operate? How much of that will the optics collect? How much will make it all the way to the detector and get translated into a usable signal? Throughput will determine the diameter of the optics, the sensitivity of the detector, the number of optical surfaces and the coating on those optical surfaces.
Modulation Transfer Function
The modulation transfer function, or MTF, of an optical system is a measure of how well it transmits details from within the system to outside the system, or vice versa. It varies with spatial frequency; that is, it changes depending upon how fine the detail is. For example, if you look at a field of corn from a distance, you will see green plants brushed with puffs of gold. When you get closer you can see the gold is due to a tassel protruding from the top of the plant; closer still, and you see the tassel is made up of individual grainlike pods, and even closer you see the tiny pollen on the surface of the pods. You can't see the detail when you're far away because your eye doesn't have an MTF high enough at that fine spatial scale. As you get closer, the details appear bigger, and they fall within the good MTF region of the eye.
Whether welding metal with a laser or imaging details of a distant galaxy, MTF is important. MTF is affected by the size of the optics, the surface quality of the lenses and mirrors, the size of the pixels on the detector, the throughput--and more.
Scratching the Surface
Optical systems engineers spend their careers improving their knowledge and ability. There are dozens of other concerns: what wavelength will the system use and how well are the different wavelengths sent through the system? Is polarization important for this application, and how will it be affected? How will the system be tested to verify it works? How well will the system work five years from now, and how can someone predict it? Will the light be so bright it can damage the system? Is the system going into an environment that will make it degrade, and how? Although this summary gives you an idea what the job entails, it takes many years to become a competent optical systems designer.