Each search engine has something called an algorithm which is the formula that each search engine uses to evaluate web pages and determine their relevance and value when crawling them for possible inclusion in their search engine. A crawler is the robot that browses all of these pages for the search engine.
More: continued here