RPA and Web Scraping using Jupyter

Photo by Rock’n Roll Monkey on Unsplash

Overview

Setup

xeus-robot

$ conda install -c conda-forge xeus-robot

SeleniumLibrary

$ pip install --upgrade robotframework-seleniumlibrary

Browser Drivers

$ pip install webdrivermanager
$ webdrivermanager firefox chrome --linkpath /usr/local/bin
XRobot Notebook

Web Scraping using RPA

S&P 500 Stocks

Robot Framework in Jupyter Notebook

The Code

Settings, Variables and Keywords

*** Settings ***
Documentation Settings with reusable keywords and variables.
...
... The system specific keywords created here form our own
... domain specific language. They utilize keywords provided
... by the imported SeleniumLibrary.
Library SeleniumLibrary
Library OperatingSystem
Library String
*** Variables ***
${SERVER} https://www.investing.com
${BROWSER} Chrome
${STOCKS URL} ${SERVER}/equities/americas
${stocks_filter} xpath=//*[@id="stocksFilter"]
${stocks_to_grab} S&P 500
${stock_link} //tr[starts-with(@id,'pair')]/td/a
${link_count} 0
*** Keywords ***
Open Browser To Stocks Page
Open Browser ${STOCKS URL} ${BROWSER}
Save to File
[Arguments] ${value1} ${value2}
Append To File path=${EXECDIR}/stocks.txt content=${value1},${value2}

Variables

Keywords

Jupyter Interactive Environment

Test Cases

*** Test Cases ***
Get All Stocks
Open Browser to Stocks Page
Maximize Browser Window
Wait Until Element Is Visible ${stocks_filter}
Select From List By Label ${stocks_filter} ${stocks_to_grab}
Wait Until Element Is Visible xpath:${stock_link}
${link_count}= Get Element Count xpath:${stock_link}
Log Many link_count ${link_count}
Should Be True ${link_count} > 0

FOR ${index} IN RANGE 1 ${link_count}+1
${link_text}= Get Text xpath:(${stock_link})[${index}]
${link_url}= Get Element Attribute xpath=(${stock_link})[${index}] href
Log Many link_text ${link_text}
Log Many link_url ${link_url}
Save to File ${link_text} ${link_url}
END

Close All Browsers

Results

Log Report

Log Report for Test Cases
Log Report Screen

Scraped Stock Information

stocks.txt

Summary

References

Programmer and occasional blogger.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store