Here are a couple of factors that are given Top priority,
when it comes to the rate at which GoogleBot crawls and indexes your website.


1. Relevant and Authoritative Backlinks

Backlinks help crawlers find your site and can give your site greater visibility in the search results. Links from relevant content and authoritative sources are considered more powerful by search engines, and therefore are more likely to bring SE robots to your website. Submitting your site to reputable and well-categorized web directories or major social networking sites helps your site get more exposed to crawlers.


2. Content Update and Pinging

Regular and frequent content update is another important factor that attracts search engine robots. For example, the purpose of Google’s fresh crawl is to detect content update, and reflect the change in the search engine results immediately.

If your site is a blog, you can try existing pinging services such as pingomatic.com to proactively inform search engine robots of new posts and content changes.


3. Internal Link Structure

Another factor that affects search engine’s crawling rate is how the current page of a website is linked from other pages within the same website domain.

Search engines determine the relative importance of the current page on a website based on the site’s overall internal link structure. Pages that are heavily linked to internally (e.g., site-wide pages) are considered important by search engines, and therefore receive more frequent visits from spiders.


4. Sitemap and Robots.txt File

Creating a search engine sitemap for your site helps your site to get indexed more deeply as well as more frequently. A typical sitemap contains a list of URLs for crawler to retrieve.

If the sitemap is formatted in XML, you can specify extra information for crawlers, such as frequency of content change, last modification date, or relative importance of a page.

While sitemap informs crawlers which pages to retrieve, robots.txt does the opposite. i.e., robots.txt prevents spiders from retrieving all or part of your website, which otherwise is publicly accessible by human.

As webmasters become more SEO-savvy, they start to make use of robots.txt more actively (e.g., to eliminate duplicate content). But at the same time, it increases a chance for them to fumble robots.txt, and unwittingly block search engine spiders. In order to prevent any costly mistake, always arm yourself with the up-to-date syntax of robots.txt recommended by major search engines such as Google and Yahoo, and look out for Google’s crawl error reports.


5. Your Server Speed

The web server where your site is hosted should respond to a request in a reasonable time. Fast response time offers visitors good surfing experience.

The same logic applies to search engine robots as well. Given that the search engine’s primary role is to provide users good searching experience, having your website hosted on a fast web server helps your site indexed faster and updated more frequently by search engine.

Source : What Determines Search Engine’s Crawl Rate? - Search Engine Optimization Blog