Suggest a public RSS/Atom feed, sitemap, creator page, podcast, video channel, article, document or other online source for WorldIndex.
Submit a source
For now, email the public URL to info@worldindex.online. Include the source name, URL, type, and why it is useful.
Supported discovery model
WorldIndex can catalog RSS/Atom feeds, XML sitemaps and public URLs. A production crawler should normalize URLs, respect robots directives and rate limits, calculate content fingerprints, preserve source credits and send index records to the public site through a secure API.
Important hosting boundary
The continuous crawler should run on the Windows developer machine or VPS. The shared hosting site should receive and present indexed data rather than operate as a large-scale spider.