Syncing the Controller
The PS3 wireless Rock Band controllers require a USB dongle to interface with the PS3 console. Make sure the dongle is properly connected to the PS3, then press the "Sync" button on the guitar the initiate a synchronization attempt. The controller should automatically recognize the dongle after a few moments.
Ensuring Compatibility
Make sure you have the appropriate wireless controllers for your system. The PS3 does not interface with Wii or Xbox 360 controllers, so you will be unable to use Xbox Rock Band controllers on a PlayStation system, or vice-versa. There also are compatibility issues between Guitar Hero and Rock Band instruments and games, so check the compatibility matrix linked the resources section.
Possible Defects
If you still can't get your wireless controller to work with your PlayStation 3, you might have a defective product. Search for information related to your model number to see if your product is eligible for repair or refund. This is particularly notable in the case of the Guitar Hero III wireless "Les Paul" guitar controllers, which had a large number of defective shipments in its initial print run.