The small square / shape next to the website title is usually a standard icon (in Google, the icon that identifies what it is: doc, site, etc.; in other sites, the logo of the site: the Twitter bird, etc.). In GoogleSites but also in other sites, follow these steps to change that icon to what you want it to be:

  1. Identify the image you want to use.
  2. Change its size to 16 x 16.
  3. Save it using this filename (exactly; there are no variables here): favicon.ico
  4. Upload it to your attachments.
  5. This should change that icon.

(This definitely works on GoogleSites and should work on others, as the favicon.ico protocol is recognized by browsers (as opposed to being used by a specific site design tool), though each site creation method I suspect will have its own location for storing this file.)