Web Scraping
- 39% of respondents state the primary reason they have yet to implement web scraping are budget constraints
- 36% of respondents list high risks of legal complications as their primary concern
- 35% respondents claimed that their needs were already fully covered by other means of data collection, such as third-party databases
- 33% of respondents see automated web scraping technologies as too complicated
Data strategy for the upcoming 12 months
- 48% data collection and management infrastructure reliability
- 47% collecting more real-time signals for faster investment decisions
- 46% collecting and utilizing more of alternative data in data driven investment decisions
- 38% data quality assurance
Oxylabs white paper ‘The Growing Importance of Alternative Data in the Finance Industry’, reveals the reasons why some financial service organisations chose to adopt automated data acquisition while others are lagging behind.
The research highlighted that the key reason for avoiding automated data acquisition are budget constraints for a large number of financial services organisations (39%). However, while significant advancements are being made through the appearance of out-of-the-box-solutions, the upfront integration costs and lack of functionalities still remain above the bar for some.
According to Oxylabs, the landscape of data has changed. While previously traditional sources would have maintained their relevance, nowadays there are few businesses that rely exclusively on them. As external data acquisition becomes increasingly important, especially in the wake of the global pandemic, it is expected that businesses will increase their scraping-related budgets. While web scraping does retain some of its negative press, this is usually associated with its unclear legal status and high costs associated with data acquisition.
Additionally, the stigma that web scraping carries a high risk of legal complications remains with 36% of the financial sector listed as a major concern. The attempt to educate burgeoning data-driven companies about the legal intricacies of web scraping is ongoing, however, the message needs to be spread wider and with more clarity.
The report also went on to highlight that 35% claimed their needs were already fully covered by other means of data collection, such as third-party databases. However, the findings showed that as other companies expand upon their data acquisition efforts, the usefulness of third-party databases might decrease as in-house web scraping allows for greater quality control, data exclusivity, and goal adaptability.
Finally, a third (33%) viewed automated web scraping technologies as too complicated – this perception arising from attempts to build complete in-house scraping pipelines, which needs to be set from the ground up. This does however indicate an immense opportunity for services that create out-of-the-box solutions. Simplifying product architecture and improving ease of use could bring an incredible amount of new clients
Denas Grybauskas, Head of Legal at Oxylabs.io said: “Web scraping is definitely fraught with a lot of legal and technical intricacies and because there are currently no industry-wide regulations and ethics agreements, it’s easy to get into trouble. Nevertheless, one of the most important pieces of advice, outside of always consulting with a legal professional, is to only extract publicly available data. All other types of data are bound to be protected by a swath of laws and regulations and scraping them without infringing upon someone’s rights is nearly impossible.”
Data strategies beyond 2021
For the next 12 months, most financial services organisations stated that they have a dedicated data strategy. However, out of all aspects of data strategy, quality assurance was one of the goals that sits at the lowest rung of the ladder (38%) while others (e.g., infrastructure reliability, collecting more signals, and utilising more alternative data) are divided equally amongst themselves (46-48%).
It is clear to see that alternative data is playing and will continue to play an increasingly important role in financial decision making. Whilst there are numerous financial incentives to do so, due to the complexity of external data acquisition, some companies are yet to implement these processes.
Third-party web scraping services and tool providers have an incredible opportunity to enter the market and solve most of the quality assurance problems that are causing companies to avoid external data acquisition.
Julius Černiauskas Chief Executive Officer at Oxylabs.io said:
“Data has always been the primary driver of decision-making in the finance sector. However, these findings have shed light on the new developments in data collection strategies. I can see there are ample opportunities for everyone in this field – from legal professionals to data scientists to take advantage of the opportunities in web scraping. The industry is still bristling with challenges, but for those who can solve them, they are sitting on a gold mine.”
