Web scraping is something of a secret, says data security consultant Eran Halevy, and usually reserved for the largest companies, like Amazon and Walmart. In an article for Entrepreneur magazine, however, Halevy says that web scraping has become an unavoidable online arms race.
Data scraping is the process of rapidly gathering data from competitors’ websites, like their prices, and use it to remain competitive. This straightforward algorithm has now turned into efforts to sabotage such data collection and even engage in disinformation tactics.
Get the latest in cybersecurity and digital commerce at NJBIA’s The Internet of Things – Transforming Your Business
Some, for instance, will recognize when their website is being scraped and provide the data collector with false information, like higher prices than they actually charge. To avoid these tactics, companies have employed proxy networks—data-center-based routers through which they route to hide their identities. Companies have also turned to peer-to-peer networks where they use an individual consumer’s home computer, with their consent, and collect intelligence by looking at the web as the consumer sees it.
So is all of this digital cloak and dagger worth the effort?
“The bottom line is that web scraping is surreptitiously powering more online commerce than you realize,” Halevy said. “Fortune 500 companies remain competitive by algorithmically adjusting their prices in reference to the market, an impossible task without scraping.
“Having these data collection machines be misled by the target websites means pricing based on false information. This is a strong enough motivation for businesses to win this scraping battle,” he wrote.
