Block level shingle analysis for determining duplicate content

General Tools
Duplicate Content Analysis
Check your pages for in-site duplicate content on a block level with a shingle-analysis.
To WWW or not? Domain check
Check your domain to determine the optimal domain name, 301's and canonicals.
Check 'IfModifiedSince' handling
Check to make sure your IfModifiedSince and Last-Modified handling is correct
Check server headers
Check your server's headers, check 404-handling and check for correct redirects
Page Title and Headers
Page titles and headers are very important elements for search engines.
Link Analysis
Find out about all links on a page, including hidden ones and nofollow-markers
Keyword frequency
Determine keyword frequency for 1,2,3-word phrases
Spider simulator
See a page as a search engine spider will see it, headers, text and links
Javascript content hiding
Hide content on your site from search engines and other crawlers / bots
The AOL search database
Currently completely online, available for your searching and browsing
Mobile browser tools
Website diagnosis tools for mobile browsers, phones, handhelds, etc.

 
Firefox Extension
Get our Firefox Extension to use these tools from your context-menu!
Search Engine Referrers
A statistic tool to find interesting information about your visitors.
Collected ramblings
Articles, tips, tricks and ideas for websites in general
Yahoo! Secret Site-Info
Yahoo!s information for your site regarding the number of links and deep links
Yahoo! Uniqueness Test
Testing the uniqueness of a web page or a text snippet with shingle analysis
Google datacenter tools
Google World-Wide
Try Google geolocation targeting in search from all over the world.
Google's results over DCs
Check your favorite query on a different datacenter (images, news, blogsearch, etc)
Indexed page count per DC
Find the number of known pages in Google per datacenter
Supplemental index count per DC
Find the number of pages in the supplimental index in Google per datacenter
Recognized links per DC
Find the number of known links in Google per datacenter
Cache-Date per DC
Check the date a URL was last cached by Google per datacenter
Google Pagerank in general
Check Googles PR value for a page
Google Pagerank per DC
Check Googles PR value over multiple datacenters
Give a DC a name!
So you like (or hate) a Google datacenter? Give it a name of your own!
Search with Pagerank
Do a normal Google search (requires free Google API-Key), see results with PR-number

Please enter up to 10 URLs (on separate lines):

Choose your desired shingle-size (number of words):

This tool spiders a page, extracts the indexable text on a block level, creates 'shingles' (groups of words) and compares those to the other URLs you have specified. This algorithm can be used to help determine how similar different pages are - or how unique the content on the page actually is.

It can make sense to run several URLs from the same site through this tool to determine the in-site duplicity. It can also make sense to run several related URLs through this tool to dermine how related their indexable content is.

The "k-shingle" algorithm is sometimes referenced in patents issued to search engines.

[ See latest shingle analysis reports | Discussion ]