Price
The word "gem" implies a stone with a price tag. A gem with no value is a rhinestone or a bauble, fit to decorate a costume but not to set in durable jewelry. The precious gemstones - diamond, ruby, sapphire and emerald - generally command the highest prices. Semi-precious stones such as garnet, peridot and amethyst carry smaller prices, but even the most economical gem must have some value or it becomes just an attractive rock.
Durability
Archaeologists have uncovered gems in burial caches that are millennia old. Even if the other grave goods decay, gems endure. Diamond is the hardest mineral known at a 10 on the Mohs hardness scale; ruby, sapphire and topaz are also high on the scale. Durable items hold greater value when families can pass them down and transfer wealth to succeeding generations; costly gems allowed early cultures to preserve familial wealth in this way.
Scarcity
Before the advent of cultured pearls, people paid more for natural pearls than for any other gemstone. Healers attributed great power to them. One medieval recipe for curing virtually any illness was a glass of wine with crushed pearl dissolved in it. Natural pearls cost more than cultured pearls, but even they received a demotion to semi-precious gem status. Gem values also change as sources run dry or as prospectors find new gem-rich land to mine. Diamonds only appear scarce. Their apparent rarity is a product of careful price control and monopoly.
Beauty
Cost, scarcity and durability alone won't turn a rock into a gem. It must also look appealing. Gems almost invariably have sparkle, fire, brilliant color or an unusual luster. Sometimes their beauty arises naturally, as with pearls and amber. Other gems reveal their beauty only when a lapidary polishes or facets them. Before cutting techniques allowed jewel-cutters to facet diamonds, the unbreakable stones seemed like little more than a curiosity. To become the sought-after gem they are now, diamonds had to wait until gem-cutting technology caught up with their toughness.