Andrius Palionis, VP of Enterprise Sales at Oxylabs
There’s no secret that financial services organizations thrive off of information. Almost all business models in the sector rely on beating out the competition by finding out the most profitable investment or spending opportunities. As these are limited in supply, getting to the insights second or third means losing out on returns.
As a result, financial services companies are always on the lookout for ways to gain that sought-after edge, which is often discovered through data collection and analysis. These organizations employ many different methods, however, it hasn’t been exactly clear which are the most effective ones.
So, Oxylabs took it upon themselves to find out if there are some up-and-coming data collection methods, which data sources are the most valuable, and how the industry is shaping up. In cooperation with Censuswide, we surveyed over 1000 key decision makers in various financial services companies.
Web scraping pushes the envelope
While web scraping (i.e., automated public web data collection) is relatively new, it has swiftly gained popularity, especially in ecommerce and finance industries. As the data collection method allows organizations to extract as much information as they need in real-time, it was seen as particularly attractive to these industries.
Exact data, however, was unavailable. Through our survey, we found out that web scraping has been strikingly successful. First, over a quarter (26%) of companies said that it has had the greatest impact on revenue out of all external data collection methods. Web scraping beat out:
- Third-party traditional data aggregators (12%).
- Manual data collection (12%).
- Third-party alternative data aggregators (11%).
Even internal data collection, a method that has existed for eternity and has been optimized to no end, barely came out ahead of web scraping with 28% of participants thinking it as having the greatest impact on revenue.
All of these numbers in combination lead to one conclusion – web scraping has entrenched itself within financial organizations. It’s staying for the longer-term as well, as our further data has shown.
Companies move towards web scraping
Unsurprisingly, there have been a decent number of companies that have yet to implement web scraping or those that are just at the beginning stages of the journey. Going by our previous data, one could make the conclusion that web scraping is seen as valuable by relatively few.
Other questions in the survey, however, illuminated web scraping’s place in the industry. For example, nearly half of our respondents (44%) plan to invest the most into web scraping in the coming years. Clearly, a much larger part of financial services companies sees the method as highly valuable.
Putting web scraping’s best foot forward, however, is the shifting focus in these organizations. According to our survey, 80% of all respondents think their focus will shift more towards web scraping in the next 12 months.
Therefore, it seems reasonable to conclude that those who have implemented and optimized web scraping pipelines have already seen significant benefits. Others, however, are catching up. Clearly, the benefits outweigh whatever the costs may be and by a large margin.
Geographical differences
There are, however, some wide ranging differences in the way web scraping is understood between the US and UK regions. While web scraping has been dominant in both, there are some differences in how it’s utilized.
For example, US companies thought that web scraping is the dominant driver of revenue out of all methods while ones in the UK thought internal data acquisition was more valuable. Additionally, US companies were significantly less likely to outsource anything web scraping or compliance related, focusing more on internal management.
Finally, what we found most interesting is that the US is somewhat more enthusiastic about web scraping. Nearly nine out of ten (86%) participants thought that their focus will shift more towards web scraping in the coming 12 months. In the UK, on the other hand, while still many, fewer decision makers thought this data acquisition method will be in focus (74%).
Conclusion
Financial services companies hold all acquisition and analysis methods as closely guarded secrets. While some of them were publicly known and commonly utilized, their status and importance in the industry was still shrouded in secrecy.
Our survey outlines a few important trends that might be reflecting global initiatives. First, web scraping is experiencing a surge of interest. While not all companies might have implemented it yet, those who have are reaping the benefits and the competitors have taken note.
Additionally, the ROI on web scraping is extremely enticing. We have seen that it beats out either all acquisition methods or only falls behind internal data. Most notably, web scraping is still maturing technology, but it has already reached incredible heights.
As a result, we will inevitably see nearly exponential growth in the field as the technology becomes more optimized and accessible. A lot of troubles today, as evidenced by the rest of our survey, are due to the still developing legal contexts and the seemingly complicated nature of the technology. As these resolve, the data acquisition method will become irreplaceable.
