In recent weeks, openDemocracy’s website has been repeatedly knocked offline, as if under enemy attack. The partial cause is a gargantuan barrage of automated bots from AI companies.
As people increasingly turn to chatbots and other large language models (LLMs) rather than searching or visiting sites directly, AI companies are using large-scale scrapers to gather the data to power their services.
Many of these scrapers are so sophisticated that it is hard, or impossible, to detect them in action. They often ignore the websites’ programmatic pleas not to be scraped, and are known to hit the more fragile parts of a website repeatedly.
