Instructions
Open up a new message screen and select the text field. This will bring up the on-screen keyboard.
Use the directional buttons on your PS3 controller to select the Symbol button in the bottom left corner of the keyboard. Press "X" to select the button. This opens a window displaying available symbols that aren't listed with the letters on the keyboard.
Scroll down below the standard keyboard symbols to view a list of pre-made emoticons. Select one that you wish to use and press "X" to add it to your message.
Select the Symbol button and enter individual symbols to create a traditional emoticon such as ":)" or ":(" if you prefer the look of old-school emoticons over the cutesy faces that are pre-made for you.