RPA and Web Scraping using Jupyter

Photo by Rock’n Roll Monkey on Unsplash

Overview

In my previous article, I walked through with you how to use Python + requests + lxml to scrape stock data. In this article let’s explore using Robotic Process Automation (RPA) in a Jupyter Notebook environment to perform web scraping. Personally, I find Jupyter Notebook + RPA a great combination as the interactive nature of Jupyter Notebook allows for quick iterations and trial-and-errors when developing robots. Also, another good thing is that all these tools are open source.

Python: Stock Data Scraping

I am going to usexeus-robot which is a Jupyter kernel for Robot Framework based on the native implementation of the Jupyter protocol xeus.

Setup

xeus-robot

I assumed you already have JupyterLab 3.0 and above installed. To install xeus-robot and its dependencies, just follow the instructions and run the following commands

$ conda install -c conda-forge xeus-robot

xeus-robot depends on Robot Framework which is a generic open-source automation framework for acceptance testing…