Spiders a web page like Google, Yahoo or MSN would

General Tools
Duplicate Content Analysis
Check your pages for in-site duplicate content on a block level with a shingle-analysis.
To WWW or not? Domain check
Check your domain to determine the optimal domain name, 301's and canonicals.
Check 'IfModifiedSince' handling
Check to make sure your IfModifiedSince and Last-Modified handling is correct
Check server headers
Check your server's headers, check 404-handling and check for correct redirects
Page Title and Headers
Page titles and headers are very important elements for search engines.
Link Analysis
Find out about all links on a page, including hidden ones and nofollow-markers
Keyword frequency
Determine keyword frequency for 1,2,3-word phrases
Spider simulator
See a page as a search engine spider will see it, headers, text and links
Javascript content hiding
Hide content on your site from search engines and other crawlers / bots
The AOL search database
Currently completely online, available for your searching and browsing
Mobile browser tools
Website diagnosis tools for mobile browsers, phones, handhelds, etc.

 
Firefox Extension
Get our Firefox Extension to use these tools from your context-menu!
Search Engine Referrers
A statistic tool to find interesting information about your visitors.
Collected ramblings
Articles, tips, tricks and ideas for websites in general
Yahoo! Secret Site-Info
Yahoo!s information for your site regarding the number of links and deep links
Yahoo! Uniqueness Test
Testing the uniqueness of a web page or a text snippet with shingle analysis
Google datacenter tools
Google World-Wide
Try Google geolocation targeting in search from all over the world.
Google's results over DCs
Check your favorite query on a different datacenter (images, news, blogsearch, etc)
Indexed page count per DC
Find the number of known pages in Google per datacenter
Supplemental index count per DC
Find the number of pages in the supplimental index in Google per datacenter
Recognized links per DC
Find the number of known links in Google per datacenter
Cache-Date per DC
Check the date a URL was last cached by Google per datacenter
Google Pagerank in general
Check Googles PR value for a page
Google Pagerank per DC
Check Googles PR value over multiple datacenters
Give a DC a name!
So you like (or hate) a Google datacenter? Give it a name of your own!
Search with Pagerank
Do a normal Google search (requires free Google API-Key), see results with PR-number

Full URL:  
User-Agent:  

This tool spiders a page, groups the content by title and headers and displays all found links (as well as their relevant attributes). A search engine will try to extract content similar to how it is done here, determining context of text by location and headers.