Modifying "Tekkit's" RAM Use
Open the folder containing your "Tekkit" server, and then right-click the "launch" file and select "Edit" from the context menu. The launcher's code opens in your text editor. On the first line, the section reading "-Xmx3G -Xms2G" specifies the Java heap size -- essentially, how much RAM your server can use. Change the number after "-Xmx" to your desired maximum value in gigabytes. The number after "-Xms" determines the minimum amount of dedicated memory, and must always be the same or less than the "-Xmx" value. To use more specific values, change the "G" after each number to "M", and then enter values in megabytes. Save the file, and then relaunch your server to run it with the new RAM values.
Java RAM Limitations
All 32-bit programs have a 4GB limit on RAM use. Due to Java overhead, however, the 32-bit version of Java caps at around 1.4 to 1.6GB on 32-bit versions of Windows, while 64-bit versions of the operating system approach the 4GB threshold more closely. If you have a 64-bit operating system, running your server with the 64-bit version of Java will remove the 4GB cap entirely.
Using 64-Bit Java
You can download the 64-bit version of Java from the Java website. Rather than replacing the 32-bit version, it installs alongside it so that you can continue to use Java in 32-bit Web browsers. Programs normally use the 64-bit version by default if you have both installed, but if "Tekkit" still has trouble with large heap sizes after installing the 64-bit version, force your server to use it by changing "java" in the launcher code to the location of your 64-bit Java installation, such as "C:\Program Files\Java\jre7\bin\java.exe" (without quotes).
Picking RAM Values
To select proper values for "Tekkit," only change the defaults by small amounts as necessary. If you find your server begins running poorly after several hours or when a large number of people connect, increase the values marginally -- by 1GB or less -- and try again. Once the server remains stable, there is no benefit in continuing to add RAM. Allocating more RAM to the server leaves less available for other tasks, so don't dedicate too much if you plan to use your computer while running the server. You can check how much RAM your system has available in the Performance tab of the Windows Task Manager.