Site Update Notification
The Site Update Notification project aims to deliver a mechanism for web masters to automatically publish information about changes to their website (changes in content and in presentation and layout) directly to those interested in keeping up-to-date with their publications. This could be Search Engines, Page Monitors, Bookmark Managers, Offline Browsers, Website Mirroring, Web Archives or others.
This is an alternative to the traditional crawling of sites that download pages and following hyperlinks to related pages (crawling the web) and distributes the burden from a centralised (albeit distributed) web spider and increases efficiency by reducing bandwidth requirements across the internet.
Keywords
crawler, spider, web crawler scalability, optimization, change notification, site update notification
Project Goals
The Site Update Notification System seeks to ...
- address the problem with crawler scalability
- reduce the crawler bandwidth and index load
- reduce the time that a web site’s change takes to be available in a search engine’s index
- address the problem crawlers currently face in accessing the Deep Web
- provide an infrastructure for agent data aggregation and delivery to subscribers that reduces the bandwidth of individual web sites
Home