Read Scenery Books
Purchase or borrow books about scenery. Books published decades ago are less useful because they do not incorporate newer (and often better) techniques. Reading a wide selection of scenery books will give you an idea of the range of techniques being used today in the hobby, many of them quite imaginative. This is the best way to decide which approach best suits one's capabilities, budget, time and interests.
Plan Scenery from the Beginning
The table design will determine what type of scenery can be built. Using a sheet of plywood makes the construction of the layout table simple, but offers little flexibility for scenery (a layout that is perfectly flat looks unrealistic, since even the flattest real-life terrain has irregular or slightly rolling areas). Open benchwork offers better chances for interesting scenery, including the possibility of ground levels below the tracks, but it does require more planning and the purchase of more hardwood.
Develop a Good Track Plan
A solid track plan is essential. Model railroad magazines, and books such as John Armstrong's classic "Layout Planning for Realistic Operation," will assist in this effort. A track plan that allows interesting operation will also give reasons for scenery and terrain features, for instance a hill being put in a corner of the layout to "force" a curve in that spot.
Tracks Come Next
Once the structure of the layout is in place, the next priority is careful and sound track laying. The track plan should be well-executed; ramps and sub-roadbed well fastened (in the case of open benchwork); the exact path of the tracks marked on the wood in pencil; and the cork or synthetic roadbed carefully laid, attached and smoothed. Once the track is in place with no rough spots or bad joints, and with solid electrical connections made, a lengthy period of track testing and train running to eliminate derailments and trouble spots should follow before any scenery is added.
Delay Track Ballasting
While many modelers "ballast" their track at this point by adding and gluing a multitude of tiny scale-sized rocks between the ties and rails, it is wise to wait to do this until after the trackside scenery is built. It can be a nightmare to clean off white plaster or other scenery materials that are accidentally dripped onto the painstakingly applied track ballast.
Build Scenery
Purchase and/or scavenge scenery materials according to the scenery book or method that has been chosen. One may use a self-contained kit such as those sold by Woodland Scenics, traditional "hardshell" technique (wet plaster strips over a cardboard webbing) for building landforms, or innovative ways of collecting and using outdoor materials as described in Lou Sassi's book, "Basic Scenery for Model Railroaders." Model trees can be bought at a hobby store or built from scratch using wire, clay and spray-painted steel wool. It is also helpful to bring in a friend or family member who may have little interest in trains but does have an artistic streak. Try small areas at first and keep things simple. Scenery that is not perfect can always be ripped out later and replaced with a better design.