Web scraping has been coming into the limelight in recent years due to the rising interest in data. Businesses across the globe have been eyeing automated data collection as a way to enhance their profitability and overall decision making.
We’ve sat down with the Lead of Commercial Product Owners at Oxylabs.io, Nedas Višniauskas, to talk about the future of web scraping. Few people have been as deeply involved with the industry as Nedas, which has allowed him to gain a unique perspective on how it has developed and how it will continue to do so.
What do you think has been the biggest change in web scraping over the last decade? How has Oxylabs participated in these changes?
There have been some interesting changes during the past few years. One of them, I think, has been the proliferation of increasingly sophisticated anti-bot systems. Scraping such websites at scale, in turn, becomes more difficult.
Scraping enthusiasts, of course, have their own answer to these issues, which is to develop dedicated data collection tools. These, while limiting the field of use, can bypass the anti-bot systems and they are constantly being updated for that purpose.
Headless ones, therefore, are a necessity. At the same time, that means infrastructure costs are rising as headless browsers take up much more computing power and traffic than simple HTTP requests.
Finally, ethics have been in the limelight. For example, residential proxy providers are looking for ways to inform and reward participants of the network. We ourselves took charge of building the framework for ethical acquisition, which, I believe, has played a part in the fact that there are less shady practices and more clarity among all industry participants.
To answer the second question, Oxylabs have reacted to these changes with the development of Scraper APIs. We created both dedicated and universal scrapers that can acquire publicly available data from nearly any website without issue. Additionally, all of our proxies are ethically sourced, giving our partners the much needed peace of mind when engaging in scraping.
Have you seen or noticed any particular trends in data acquisition or web scraping? Are specific data types becoming popular?
Off the cuff I’d say that the use of ecommerce and delivery data has been booming since the pandemic hit. Businesses want to (legally) spy on competitors and gain access to as much data as possible. Data types like pricing, products or delivery times are important to any competitor.
But these have always been important. Maybe I would say that external data in general has risen in importance. Outside of that, I don’t think there have been any particular trends in data types. There have been, however, changes in the entire supply chain. As I’ve mentioned, businesses only really need the data. Even then, the data is not the key – insights are.
As such, businesses at the tail-end of the chain have proliferated in recent years. Data-as-a-service aggregators, ones that collect information and sell sets of it, have been rising in popularity.
There are also some businesses that provide insights directly. While these are still few and far between, some of them have unique value propositions that I could see as worthwhile. Jungle Scout, for example, is a service that both scrapes external data and has large datasets from internal sources. As such, they can provide insights other businesses can’t.
What do you think are the biggest challenges the industry is facing currently? Are there any innovative solutions to these or other challenges on the horizon?
Bot protection has always been the greatest challenge. Scraping, you see, is a cat-and-mouse game. Websites attempt to implement anti-bot measures, such as the well-known CAPTCHA, while scraping companies attempt to continue evading them to retain access to data.
There have been great strides made in bot protection. TLS (Transport Layer Security) fingerprinting has been one such improvement. Sophisticated websites can use initial network handshakes to match them with headers. As many scraping tools manually modify the headers sent, TLS can often be mismatched, which would be a dead giveaway.
On the other hand, the deck is always slightly stacked in the favor of scraping. Most anti-bot protection features put a dent in the overall user experience. Filling in a CAPTCHA is something that detracts from that frictionless experience of the modern web we’re used to.
Some businesses use these techniques and see no issue. Others, ones highly concerned with delivering the best user experience possible, avoid using CAPTCHAs unless absolutely necessary. It’s always a tradeoff. More bot protection equals, almost always, worse UX, which leads to less revenue. But then less people are scraping your website.
Additionally, new pages with interesting data and content appear all the time. And you don’t start building a website from bot protection. It has to be functional first. So, the process of scraping is a lot easier than it could be for a long time.
Would you say that there are potential benefits in web scraping for academic research or policy-making? If so, why hasn’t the scientific or political community adopted the practice?
Academic research, quantitative in particular, is in large part based on data that doesn’t exist on the internet, yet. There could be studies, however, on internet behavior or something of the like where scraping could be immensely useful. Additionally, I think we’re not seeing such widespread adoption due to the previously mentioned barrier to entry.
Let’s imagine that there’s no previous scraping experience in some particular university. The researcher would have to build everything from the ground up, get all the deep knowledge, and the funding required just to start acquiring the data.
It doesn’t help that the research areas that benefit the most from scraping (like sociology, economics, psychology, etc.) are far removed from the coding, development, and IT in general. I think it’s more of an unfortunate, but temporary, circumstance, because web scraping providers will be able to reduce the barrier by a significant margin in the future.
When it comes to policy-making, I’m not so sure. I think that rather than making, it should be about enforcing. Governments are definitely knee-deep in web scraping for all kinds of security purposes. Businesses, on the other hand, have been using the same processes to protect themselves from counterfeits and copyright infringement. There’s an entire business vertical dedicated explicitly to brand protection.
Wealth Managers and the Future of Trust: Insights from CFA Institute’s 2022 Investor Trust Study
Author: Rhodri Preece, CFA, Senior Head of Research, CFA Institute
Corporate responsibility is more important than ever. Today, many investors expect more than just profit from their financial decisions; they want easy access to financial products and to be able to express personal values through their investments. Crucial to meeting these new investor expectations is trust in the financial services providers that enable investors to build wealth and realise personal goals. Trust is the bedrock of client relationships and investor confidence.
The 2022 CFA Institute Investor Trust Study – the fifth in a biennial series – found that trust levels in financial services among retail and institutional investors have reached an all-time high. Reflecting the views of 3,588 retail investors and 976 institutional investors across 15 markets globally, the report is a barometer of sentiment and an encouraging indicator of the trust gains in financial services.
Wealth managers may want to know how this trust can be cultivated, and how they can enhance it within their own organisations. I outline three key trends that will shape the future of client trust.
THE RISE OF ESG
ESG metrics have risen to prominence in recent years, as investors increasingly look at environmental, social and governance factors when assessing risks and opportunities. These metrics have an impact on investor confidence and their propensity to invest; we find that among retail investors, 31% expect ESG investing to result in higher risk-adjusted returns, while 44% are primarily motivated to invest in ESG strategies because they want to express personal values or invest in companies that have a positive impact on society or the environment.
The Trust Study shows us that ESG is stimulating confidence more broadly. Of those surveyed, 78% of institutional investors said the growth of ESG strategies had improved their trust in financial services. 100% of this group expressed an interest in ESG investing strategies, as did 77% of retail investors.
There are also different priorities within ESG strategies, and our study found a clear divide between which issues were top of mind for retail investors compared to institutional investors. Retail investors were more focused on investments that tackled climate change and clean energy use, while institutional investors placed a greater focus on data protection and privacy, and sustainable supply chain management.
What is clear is that the rise of ESG investing is building trust and creating opportunities for new products.
TECHNOLOGY MULTIPLIES TRUST
Technology has the power to democratise finance. In financial services, technological developments have lowered costs and increased access to markets, thereby levelling the playing field. Allowing easy monitoring of investments, digital platforms and apps are empowering more people than ever to engage in investing. For wealth managers, these digital advancements mean an opportunity for improved connection and communication with investors, a strategy that also enhances trust.
The study shows us that the benefits of technology are being felt, with 50% of retail investors and 87% of institutional investors expressing that increased use of technology increases trust in their financial advisers and asset managers, respectively. Technology is also leading to enhanced transparency, with the majority of retail and institutional investors believing that their adviser or investment firms are very transparent.
It’s worth acknowledging here that a taste for technology-based investing varies across age groups. More than 70% of millennials expressed a preference for technology tools to help navigate their investment strategy over a human advisor. Of the over-65s surveyed, however, just 30% expressed the same choice.
THE PULL OF PERSONALISATION
How does an investor’s personal connection to their investments manifest? There are two primary ways. The first is to have an adviser who understands you personally, the second is to have investments that achieve your personal objectives and resonate with what you value.
Among retail investors surveyed for the study, 78% expressed a desire for personalised products or services to help them meet their investing needs. Of these, 68% said they’d pay higher fees for this service.
So, what does personalisation actually look like? The study identifies the top three products of interest among retail investors. They are: direct indexing (investment indexes that are tailored to specific needs); impact funds (those that allow investors to pursue strategies designed to achieve specific real-world outcomes); and personalised research (customised for each investor).
When it comes to this last product, it’s worth noting that choosing advisors with shared values is also becoming more significant. Three-quarters of respondents to the survey said having an adviser that shares one’s values is at least somewhat important to them. Another way a personal connection with clients can be established is through a strong brand, and the proportion of retail investors favouring a brand they can trust over individuals they can count on continues to grow; it reached 55% in the 2022 survey, up from 51% in 2020 and 33% in 2016.
TRUST IN THE FUTURE
As the pressure on corporations to demonstrate their trustworthiness increases, investors will also look to financial services to bolster trust. Wealth managers that embrace ESG issues and preferences, enhanced technology tools, and personalisation, can demonstrate their value and build durable client relationships over market cycles.
5 tips to ensure CSR efforts come across as genuine
By Mick Clark, Managing Director, WePack Ltd
Corporate social responsibility – or CSR – is playing an increasingly pivotal role in the long-term success of modern-day companies.
The harsh reality is that only a paltry 46 percent of people trust the brands they buy from. And with more competition than ever in all walks of business, a positive brand reputation needs to be earned or customers will simply take their money elsewhere.
That’s why I share my insights on the importance of CSR in modern business and introduce an effective plan to avoid coming off as disingenuous to your employees and customer base.
The value of CSR
The needs of modern employees and consumers are changing. There is a higher emphasis placed on the ethics and morals of companies and their handling of hot button topics like the environment or social issues.
59 percent of UK workers believe their business should be investing in charitable initiatives. 67 percent of people aged 18-19 feel this way, showing a generational shift in favour of companies that support ethical, social, or environmental causes.
At WePack, we recognise the importance of this and make sure to regularly donate to a variety of charities including RRT (Rapid Relief Team), and donated £6,000 to the charity’s social causes last year.
An example of good CSR can be found in search engine giant, Google. It has had notable success with its CSR initiatives. Its flagship CSR campaign, Google Green, is a companywide commitment to using clean sources of energy, cutting down on its use of fossil fuels and drastically increasing energy efficiency as a direct response to the climate crisis.
It has been so successful that its data centres now require 50 percent less power to run than the average data centre and it’s poured over $1 billion into jumpstarting renewable energy projects.
Customer attitudes are fundamentally changing, and people are far more concerned about the values that their money could be indirectly supporting. In fact, 71 percent of customers prefer buying from businesses that align directly with their values.
In the modern-day, demonstrating high levels of CSR boosts brand perception. Businesses that make it a priority are more attractive – from an investment standpoint – to both customers and potential stakeholders.
For example, more than a third of consumers are also willing to pay more for a product or service if the business prioritises sustainability specifically – so it pays to be responsible.
Businesses with purpose-driven and ethical goals and proven commitments to CSR help retain employees. Millennials will make up 75 percent of the workforce by 2025, and it’s that cohort that is increasingly demanding socially responsible employers.
Those that fail to meet the needs will ultimately see their customers take their purchasing power elsewhere.
Addressing the challenges
As obvious as it may sound for a business to take on as much CSR as possible, many organisations face limitations.
Pressure from investors can disrupt the growth of CSR initiatives. Sometimes, the direction that stakeholders want to take the company doesn’t fully align with plans to target social or environmental issues.
Companies face becoming fixated on linking profitability with CSR programmes. It can be tough to present a genuine CSR programme without it coming across as a marketing ploy – presenting an extra hurdle for businesses to overcome.
Despite the challenges businesses face that are out of their control, many firms unwittingly make their own mistakes that cost them dearly.
For example, businesses can struggle to bolster their CSR programmes if they don’t consult their customers and staff first. A simple survey helps companies decide what issues to put as a priority and target to satisfy their customer base and employees.
Any attempt to create an effective CSR programme needs top-down support. Many businesses wrongly treat CSR as a separate entity, rather than fostering a companywide culture. This can lead any attempt to push back on global issues to appear disingenuous to those looking in.
Shifting the CSR approach
Because of the global shift in public needs and opinions in recent years, businesses need to better demonstrate their efforts to avoid having their campaigns labelled as a box-ticking exercise.
It’s no secret that consumers are doing more research and are becoming more switched on to spotting lacklustre approaches to CSR. Also, everyone can have their say online – it’s much easier to get exposed if your CSR campaign is nothing but an empty publicity stunt.
For example, Volkswagen’s reputation was left in tatters after its ‘greenwashing’ scandal promoted a newer, cleaner diesel vehicle that wasn’t any better for the environment than previous models. The company took it further by fitting a device that helped it cheat emissions tests – resulting in a $125 million fine.
For this reason, CSR campaigns need tangible results to be credible and trustworthy.
Sharing top tips
When it comes to structuring a strong CSR campaign, it’s critical to demonstrate several things to prove your strategy is effective in helping the chosen cause.
Firstly, evidence the fact that your efforts are helping wider communities. Whether it’s through statistics or showing proof of investment in social causes, tangible evidence goes a long way when legitimising your CSR campaign.
Secondly, balance your rhetoric. Effective communications are vital to the success of a campaign. However, it can damage a company’s image when done poorly. Businesses should speak about their chosen issues in their dialogue rather than spending too much time talking about the solutions the company has implemented. This stops them from becoming too self-promotional or sounding braggy.
To further avoid this, make sure you can directly tie your CSR campaign to corporate values and beliefs. As well as helping to strengthen your comms, it will also guarantee that company values are more than just surface-level – helping to facilitate tangible, long-term change.
Rivery Raises $30M B Round of Venture Funding from Tiger Global
With data needs growing and data talent scarcity, there is huge demand for Rivery’s 100% SaaS solution to create an...
Wealth Managers and the Future of Trust: Insights from CFA Institute’s 2022 Investor Trust Study
Author: Rhodri Preece, CFA, Senior Head of Research, CFA Institute Corporate responsibility is more important than ever. Today, many...
Q&A with Andréa Jacquemin, founder and CEO of Beamy
Beamy is a fast-growing scale-up that focuses on pioneering a new approach to SaaS management for large companies. Founded in...
How to reignite your store with streamlined operations and a distinctive customer experience
Colin Neil, MD, Adyen UK Retailers know that prioritising customer experience is vital to success today. This, amongst the...
5 tips to ensure CSR efforts come across as genuine
By Mick Clark, Managing Director, WePack Ltd Corporate social responsibility – or CSR – is playing an increasingly pivotal role...
How to Build Your Credit Up Safely
by Taylor McKnight, Author for Compare Credit What Is Credit? Credit is money owed by a person that allows...
PCI DSS Compliance in the Cloud – Everything you should know
Introduction PCI DSS 4.0 is the latest and updated version of PCI DSS that was introduced on March 31st, 2022....
2022 ESG Investment Trends
Jay Mukhey, Senior Director, ESG at Finastra Environmental, Social and Governance (ESG) themes have been front and center throughout...
PROTECT THE VALUE OF YOUR SAVINGS AND AVOID RISING INFLATION PRESSURE
Planning for the next financial year? Former Bank Manager and successful whisky investor, Roger Parfitt, tells us why cask ownership is...
UK Organisations turn to artificial intelligence to fight sophisticated cyberattacks
New research by cybersecurity expert Mimecast finds that email attacks are becoming more frequent and sophisticated More and more companies...
The power of diversity: The need for female role models in FinTech
By Isavella Frangou, VP of Sales and Marketing, payabl. As our world is constantly evolving, it’s easy to believe...
Securing BNPL Platforms for Merchants
By: James Hunt, Payments SME at Feedzai The buy now, pay later (BNPL) market has boomed because it offers...
Addressing the talent gap within cybersecurity
By Merlin Piscitelli, Chief Revenue Officer, EMEA at Datasite Rising geopolitical tensions and increasingly sophisticated cyberwarfare tactics have meant...
Biometric payment card FAQs with Michel Roig, Fingerprints’ President of Payments & Access
We sat down with Michel Roig to answer your frequently asked questions regarding biometric payment cards – their benefits, current...
Opportunities for UK Challenger Banks to address AML Compliance
Author: Gabriel Hopkins, Chief Product Officer, Ripjar UK challenger banks have revolutionised the banking sector with innovative products and...
HOW GOING DIGITAL COULD HELP CHARITIES OVERCOME THE CHALLENGES OF INFLATION
By Shaf Mansour, not for profit solutions specialist at The Access Group. The topic of inflation and its impact...
How to manage transformational change successfully
Adrian Odds, Marketing and Innovation Director, CDS 2020 accelerated change in the business landscape significantly. Many were already considering –...
Why the pandemic has put the pressure back on fintechs
Ben Walker, Partner & CTO, Airwalk Traditionally, the only genuine threats to the incumbent banking giants were macroeconomic instability and...
Neobank Fi launches new feature ‘Connected Accounts’ allowing users to sync multiple bank accounts on a single app.
Neobanking app Fi launched its ‘Connected Accounts’ feature to become one of the first fintechs to build a product on...
Accounts Payable fraud: Do you know who’s accessing your finances?
Mark Blakemore, CFO at Compleat Software The use of social engineering and phishing attacks on accounts payable (AP) departments...