Things You'll Need
Instructions
Decide on your overall concept for the class. The more detailed the class is in your mind, the easier it will be to design.
Figure out if the class will be a base class or a prestige class. If the class has a very specific focus, a limited area where it can function or limited ability set, then create a prestige class. If the class has more generic abilities and operational areas, then create a base class.
Pick the approximate number of levels your class will have. Base classes always have 20 levels-whereas prestige classes usually have 3, 5 or 10 (but can have any number). This will give you a guide as to how many special abilities the class will need, the amount of spell casting, attack power and so on.
Choose your basic numbers for your class: base attack bonus, saving throws, hit die and skill points. This is where your concept will really help. The majority of these will be obvious (e.g., spell-casters nearly always having a high Will save), but if you have any that you are not sure about, mark these and you can alter them later.
Begin to work out the special abilities of your class. Make the additive abilities progress in a logical manner (e.g., Rouges getting Sneak Attack every other level or Rangers getting Favored Enemy every five levels). Don't create haphazard lists. It's not uncommon to have to switch things around from your original ideas at this point. Do your best to make sure that the class gets something on every level, even if it's just attack or save increases; nobody wants empty levels.
Select class skills. When picking skills, keep in mind the number of skill points the class will possess. If the class only possesses 2+Int for points then they'll also have a limited skill set as well. It's important that this is done after the special abilities are chosen in case an ability requires a skill check.
Deciding on spells and spell progressions can be tricky. Spell progressions are easily taken from existing materials. The number of spells using classes is very high, so this will keep you from having to work out exact progressions. Depending on the concept of the class, spells can be a pre-existing list or a custom-built one. When customizing a spell list, it's important to think of the specialty of the class; some spells may be easier to cast and some more difficult than standard. Lastly, consider how many spells the class should have access to; some spell lists should only have a handful of spells available at any one level.
Look over your class, and decide if all of the decisions that you have made work out. Don't be afraid to change parts of your class if you think they are over- or under-powered. Double check all of your numbers to ensure that they work out with the ideas you had after choosing them. Most importantly, make sure the class is what you want. It's easy to get sidetracked as you work on something like this and come up with a different outcome than anticipated.
Playtest your class. If you're running a game where a player is playing a custom class, make sure they know the class abilities can change as the game progresses. If they aren't cool with that, then you need to find a different player to test it. An alternative is to have an NPC use the class. This will require more work on your part, but the mechanics will be easier to test and change.