Web Scraping, Indexing, and Scanning Applications (Resource Overusage)
Print this Article
Last Updated:
May 26, 2015 12:58 PM
You cannot host websites or applications that that scan, scrape, or index external sites. Doing so can cause you to overuse your hosting resources.
Is this an issue for me?
Here are some common sources of excessive resource usage due to web scraping or indexing:
- Hosting applications intended to generate content by web scraping or web indexing
- Hosting applications that automatically generate content strictly for SEO purposes
- Hosting applications that duplicate external site content
How can I correct this issue?
You can try to correct resource usage caused by web scraping by following any and all of these steps:
- Remove any and all applications intended to generate web scraping content.
- Remove any automated content generation for strictly SEO purposes.
- Migrate to a VPS or dedicated server if the above steps are not an option.
Next Steps
If you are still experiencing issues with resource usage, review your hosting for other possible causes at Common causes for resource overusage (Linux).
You will want to make sure that you are correcting the underlying cause to any resource issue. Simply removing sites and/or files without correcting the underlying issue can cause the problem to recur.
