Instructions
Study other table-top roleplaying games to better understand their gameplay mechanics. Keep a detailed record of what aspects work for you and which don't. Analyze how other games balance classes and describe player abilities. Given the vast quantities of D20 games out there, there's no reason for you to start from scratch. Learn about what's already been done. If you find a system that really works well for you, there's no reason you can't base a differently-themed campaign on the same game rules, or even by combining rules from multiple games.
Choose a genre for your game. Your gameplay mechanics will depend heavily on the style of game you are creating. Even simple details like the names of spells and abilities will be dependent on your setting. For example, an elven wizard in a fantasy setting may be able to cast "Magic Missile," while a genetically-enhanced mutant in a cyberpunk setting might cast "Psionic Burst." In terms of mechanics, these are the same abilities, but their names help to establish the fictional world your players will find themselves in.
Create detailed descriptions of your game setting. A good roleplaying game relies heavily on the players' abilities to imagine themselves in another world. Make maps and written background stories to help get your players oriented. If you are artistically inclined, draw pictures of the races or classes that are available to your players.
Write the rules of the game. This is the most difficult step of the entire process, and can take a considerable amount of time. If this is your first time creating a D20 game, basing your game off of an existing set of rules is the easiest way to learn the ropes. You will need to have clearly defined rules for player statistics, leveling up, skills and abilities, combat dynamics, casting magic and saving throws, to name a few. Will your casters rely on mana or spells-per-day? Will you make your players lose permanent sanity points when they encounter mind-bending terrors? How will you distribute experience points? Every aspect of gameplay must be clearly written out before you begin crafting a campaign.
Test your game out with some friends. Before you put together a full-scale campaign, gather a few trusted buddies to test out how well your new game works. Take note of any class balance issues or confusion that arises over statistics or saving throws. Getting your game just right can take some trial and error, so don't be discouraged if it doesn't play well the first time around.