Instructions
Develop a concept for your game. RPGs are basically tools that allow players to tell their own stories. Devise a world in which they can play in, settings they can explore, supporting figures they can interact with, called non-player characters or NPCs, and equipment they can use. The specifics depend on the kind of genre you want your RPG to be. For example, for a spy-based RPG, develop a super-secret agency that the heroes work for, a villain or villains they will fight, exotic locales they will visit and high-tech gadgets they will use.
Decide what role the RPG's players will take on in your world. Players adopt the persona of fictitious characters--called player characters, or PCs--who will serve as the heroes of the story. Provide a set of parameters that defines who those heroes are without limiting players' options. For example, in a spy-based RPG, all of the player characters should probably be secret agents; you don't want them playing surfers or CPAs who have no bearing on a spy-style story. But allow for as wide a variety of secret agents as you can, letting players create the characters they want while staying true to the genre.
Develop a mechanical system for your RPG. Tabletop role-playing depends on die rolls to determine the outcome of events within the story--whether a character hits something he shoots at, for example. Your system should have a series of statistics representing a character's traits, such as strength and intelligence, and skills, like the ability to defuse a bomb, which can be measured in numerical terms. Develop a means by which those skills are tested and a means of determining success or failure. In addition, provide a means of creating characters: Assign points to possible skills and ways to improve those skills as the characters learn, grow and get better.
Test your system by running sessions with groups of players to make sure it is balanced. No one's skill should be more powerful than any other, no character should be essentially unstoppable, and no obstacle should be devised which cannot eventually be overcome. Aim for "breaking" the system during these sessions by deliberately searching for problems. As playtesting proceeds, look for places where your system needs repair, and adjust the numbers you use in order to maintain a good system balance.
Contact an artist who can draw pictures for your RPG. Though they aren't always necessary, most RPGs have at least a few pictures to help break up the text. If you don't have access to an artist, consider checking the Internet for copyright-free images that match your game setting. Find a good image for your cover as well, preferably in color and able to blend with the title of your game in a pleasing fashion.
Contact a print-on-demand publisher to publish your RPG. He can set up a print run for you and provide set guides for page count and size which you should adhere to.
Design a layout for your RPG based on the page size and page count your publisher gives you. Use easy-to-find software such as Adobe Photoshop or InDesign, or hire a graphic designer who can provide a nice border for your pages and good graphic elements. A graphic designer can also provide a good cover for your RPG if you don't wish to use an artist.
Lay out your RPG once the design is finished and the system is balanced. Organize it in any way you like, though it's usually best to separate the overall setting, the system mechanics and the process of creating characters into distinct sections. That makes finding particular information easier for your players and gives your RPG a dependable structure. Include any pictures you have commissioned, and set aside room for any charts or lists if your system demands them, such as a list of specific skills characters can have or the kinds of equipment he can buy.
Send a copy of the completed RPG to your printer for final publication.