The search engines run what are called spiders that effectively search a site looking at keywords, particularly titles menus and what is called 'meta data' which is information web designers hide in the source code that gives clues as to what the page is on about. A good web designer can include a page called 'robots.txt' that helps steer the spiders to the relevant info whilst also being used to 'hide' other parts of a site. This can be because multiple sites can be run from a single peice of web space( we call this virtual hosts).
here is an example of teh BBCs
http://news.bbc.co.uk/robots.txt
You can see what parts of the site are 'forbidden' or just useless.
Hope this helps G