Inverts
In certain types of printing, stamps went through a printing machine twice, once for a border, and again for an image. If some stamps were ever fed through one of the machines the wrong way, this meant that they would appear "inverted." It is rare that these mistakes made it to market, and so dramatically increases the value of a stamp. The most famous example of this is the inverted Jenny stamp, which is more commonly remembered as the upside-down airplane. This particular stamp cost 24 cents initially (1918), but was approximated to sell for $525,000 in 2005.
Famous Old Stamps
The very first stamp was the Penny Black (1840), which features the profile of Queen Victoria. These stamps will go for anywhere between $200 and $3,000, depending on the condition of the stamp.
Some stamps are not as valuable but are still famous for their iconic images. The Albert Einstein stamp of 1966 and any Norman Rockwell stamp will make fine additions to any stamp collection.
Rare Stamps
The rarest stamp is the 1-cent Z Grill. It was released in 1868 and features the profile of Benjamin Franklin. This is the most valuable stamp in the world, estimated at $3 million dollars. Grills are small, indented patterns on stamps; applying them was intended to discourage reuse (a practice that was abandoned due to impracticality). Grills are categorized by size and shape and given a letter to denote their difference (an "A" grill covers the entire stamp, for example) and the 1 cent Z grill is so valuable because only two instances of this size of grill exist on this stamp.
Other well-known rare stamps are the Treskilling Yellow (only one known to exist), the Mauritius "Post Office" stamp (26 exist) and the British Guiana 1c magenta (one known copy). Authentic copies of these stamps could be worth $1 to 2 million.