Syncing
Unlike the Xbox 360 and Wii versions of the RedOctane Guitar, the PS3 versions of the peripheral require a specialty dongle that plugs into the front of the PlayStation 3 game console. Once this dongle has been plugged in, you can the use the sync button on the front of the peripheral to get it to respond to your PlayStation 3. If it doesn't work on your first attempt, your controller may be broadcasting an incorrect signal. So try it several times.
Compatibility
It's very important to ensure that the RedOctane peripheral you're using is compatible with your console. All Wii, PlayStation 2 and Xbox 360 controllers are completely incompatible with the PS3. So if you have a controller for another console, you'll need to return it for one that will work.
Test for Defectiveness
If you're using the wireless RedOctane Les Paul controller that was bundled with Guitar Hero III, then chances are good that you have a defective product. There was a known issue with the guitar peripherals that shipped with this game, and if you cannot get it to sync, and have made sure that it's the correct peripheral, then simply return it and ask for a replacement product or a refund.