Trustpilot Reviews Scraping, Keyword Extraction and Analysis

Trustpilot contains reviews for multiple companies which allow us to look into issues going on with these companies. Such reviews are a great source of information if you can download this text and analyze them.

The process can be divided in 3 parts

1. Review scraping
2. Keyword Extraction
3. Analysis and visualization (Word Clouds, Race Bar Charts, etc.)

Part 1: Scraping Reviews from Trustpilot Website

1. Import required libraries
2. Create function to get overall number of reviews of the company
3. Create function to get the beautiful soup output of all pages of review of the company. This step takes care of errors due to Trustpilot blocking any requests from your system after a certain number of requests, by creating an infinite loop which keeps retrying after sleeping for short durations.
4. Select company and extract reviews into a dataframe. Here I have used “www.fitbit.com” as an example

Install BS4 before running — https://www.crummy.com/software/BeautifulSoup/