Things You'll Need
Instructions
Decide what kind of space your game system will work with. Chess, for example, uses an 8x8 array of squares (although the chess "system" encompasses any size grid). War games (tabletop or computer) take place on some sort of map. Tag, hide-and-seek and athletic sports take place in "real" space.
Make a list of the kinds of components your game system will handle. These are the things that will be manipulated in the game space, such as the men on a chess board, but they also include things such as the players' actual bodies in athletic games (or their avatars in computer games), equipment and tokens, markers, background objects and so on. You don't need to identify each and every object; just focus for now on the kinds of objects. For example, if you were designing something such as chess, you would not need to invent pawns and rooks and knights at this stage; you would only need to note that you will have pieces that have various ways of moving on the game board.
Decide on a time structure for your game system. Will it be turn-based, and if so will players alternate turns, as in chess, or will they make their turns simultaneously, as in rock-paper-scissors? Note that even real-time games can be conceptually broken down into separate phases (and generally have to be if they are implemented as computer simulations, even if those phases are invisible to the player). At a minimum, those phases will include a decision phase (when the player makes a choice) and an action phase (when the choice is executed and effects a change in the game space).
Devise rules for how the various components of your game system will interact. This is the "grammar" of your system: How do things behave? In a game such as chess, this amounts to defining the valid moves for each piece. In a more complex simulation game on a computer, it can be nothing less than devising an entire physics engine. Some games rely on randomizers, such as dice or spinners (or random number generators in computer games) to help resolve some of these interactions.
Build a simple mock-up of your game model. Often this can be done entirely with pencil and paper, though for computer games you may need to write some test code. Use the mock-up to run a few simulations to test how your game system works in various scenarios. You likely will find that a mechanism is unwieldy or inadequate in some way. Don't be discouraged; this is part of the process. You often will need to go back to a previous step and redesign components. It's not uncommon to have to go back and redo the whole model from scratch, but each time you do, you know more about what does and doesn't work.