When the Canadian restaurant chain Freshii started trading in early 2017, Founder and CEO Matthew Corrin was optimistic about their prospects: “Over the next few years we are going to double our restaurant count, and I believe we will continue to be one of the most compelling growth stories and stocks in North America.”

Meanwhile, Bogdan Tudose, an investment analyst sharpening his finance and programming skills at a Toronto-based hedge fund, was utilizing Python’s web-scraping tools on Freshii’s website to calculate how many locations the company was opening compared to expectations.

By September of that year, Freshii had altered its expectations to 90 to 95 net new openings for the 2017 calendar year, down from the previous target of 150 to 160. The stock tanked. Tudose and his firm had already figured out that the company would miss street expectations, and they shorted the stock before the earnings release.

Python in finance

Web scraping is one of the many applications of Python in finance that Tudose discussed as instructor of a two-day event hosted by CFA Society Toronto in April. During the Python 3: Web Scraping and Machine Learning course, participants worked with Python and several of its popular packages to create programs that would help them acquire, consolidate, analyze, and present large data sets.

The April sessions are part of an expanding collection of Python classes taught by the Marquee Group as interest in the programming language continues to grow. Python is an invaluable tool for processing huge amounts of data and is the foundation of data science and machine learning.

Given its usefulness in working with alternative data sets and its applications across many other areas of finance, the language is gaining wider acceptance in the investment community. In a recent global survey of CFA Institute members and candidates, 41 percent indicated that they were planning to pursue further education and training in one of the data analysis coding languages, with Python being the most popular choice.

Web scraping

Much of the Marquee Group’s Python sessions focused on web scraping, an automated method for extracting large amounts of data from websites. The process involves collecting unstructured data from websites and storing it in a structured form so it can be used for further analysis. Web scraping is used for a variety of reasons and has applications in many different industries.

Some common uses of web scraping include:

• Comparing prices
• Extracting product data for e-commerce activities
• Generating leads for marketing
• Collecting financial data from websites such as Yahoo Finance
• Conducting academic research
• Gathering social media data

There are different ways to scrape websites, as each website is unique. While there may be general structures that repeat themselves, each website will require a tailored approach to extract the relevant information.

Scraping can be done by opening a browser from within a programming language, navigating to a specific web page, and then downloading the data from that web page either directly into the programming language or as separate files. Sometimes, a browser does not even need to be opened, and the code can access the data directly from the server.

As highlighted during the Marquee Group’s presentation, web scraping can paradoxically be one of the easiest and one of the hardest things to code in a programming language due to the complexity and variability in how information is stored on websites. For more complex websites, prior knowledge of web design tools (e.g., HTML, CSS, JavaScript) may be helpful.

Why Python for web scraping?

While many software programs can be used for web scraping, Python has become a preferred choice. Python is one of the easiest programming languages to learn, and its syntax is simple to write and understand, which helps when sharing code with other users.

Python also comes with a large collection of libraries that help save time for developers who would otherwise build their projects from scratch. By incorporating Python libraries into their projects, developers can reduce time spent coding routine and common tasks. For instance, during the Python sessions, Tudose introduced students to Pandas, Beautiful Soup, and Selenium—common libraries associated with web scraping.

If developers get stuck along the way, they can always find help from a large, supportive user base. The Python community helps support beginners, experts, and all levels in between with online resources, forums, regional events, and local meetups.

Learning with examples

Along with the initial example of scraping restaurant store locations, Tudose provided several other real-life applications of Python’s web-scraping tools.

For example, a craft brewer hired the Marquee Group to provide competitive analysis. Part of that analysis required scraping the website of the Liquor Control Board of Ontario (LCBO) for data on available beers. Bogdan did a walk-through of the setup and steps required to write and execute the Python code that would automate this task.

The process involved filtering all the products listed in the beer category and collecting information such as product names, prices, packaging descriptions, regions, and image links. With almost 2,000 results spanning over 160 web pages, completing this task manually would have involved countless hours of boring, repetitive work. Using Python, however, the work was easily automated, saving time and money, enabling faster project completion, and reducing the potential for human error in the data entry process.

To extract data with Python, follow these general steps:

• Specify the URL that you want to scrape
• Inspect the web page and its elements to view the code and the makeup of the website
• Find the data you want to extract by exploring the site
• Write the code using Python’s libraries (e.g., Pandas, Beautiful Soup, Selenium)
• Run the code and extract the data
• Store the data in the required format, such as a CSV or Excel file

Depending on the website and how the data is stored, the second and third steps may require more time and effort. Through trial and error, and perhaps with some help from the incredibly supportive community of Python users, the data can be retrieved.

Once you determine how to scrape the first item, you can use conditional logic such as “if” statements and loops to repeat chunks of program code that run repeatedly until the loop is done and all data items on the website have been scraped.

Although it may be difficult at first, once you have practiced scraping a few sites, the process becomes much quicker and easier.

Next steps

As data continues to grow exponentially and more data becomes available on web pages, the advantages of learning Python for web scraping will grow, and there will be many learning options available.

Consider enrolling in some of the Python classes taught by the Marquee Group, and stay tuned for upcoming events offered through CFA Society Toronto or check out the Marquee Group website for online self-study options.

If you are completely new to programming and Python and would like to learn the basics, it may be helpful to refer to an earlier article published in the September 2020 issue of The Analyst, entitled “Python and its Tightening Grip in Finance”.