<div class="Objective_card card-content-body"><p>You will be creating a bash script that will scrape the URLs of tweets from Twitter Bookmarks and saves them in a markdown file.</p></div>

Objective

<div class="Project_Context_card card-content-body"><p>Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites. In general, web data extraction is used by people and businesses who want to make use of the vast amount of publicly available web data to make smarter decisions.
<br/>
<br/></p><p>In this project we will be scraping Twitter and collecting the URL of the tweets from the Bookmarks page and we will be doing this with simple command line tools such as cURL and jq.
<br/>
<br/></p><p>[Disclaimer: Web Scraping has to be used only for learning purposes. Any other attempts to use the data scraped might result in legal action or Your IP might get blocked.]</p></div>

Project Context

<div class="Project_Stages_card card-content-body"><p>The project consists of the following stages:</p><p><img alt="sequence_diagram" src="https://storage.googleapis.com/crio-content-container-assets/ME_ME_PROJECT_SCRAPING_TWITTER_MODULE_ME_PROJECT_SCRAPING_TWITTER_MODULE_SCRAPING_TWITTER_scraping_twitter_bookmarks_sequence_diagram.png" style="max-width:750px;"/></p></div>

Project Stages

<div class="High-Level_Approach_card card-content-body"><ul>
<li>Configure the required web driver to access the browser capabilities in testing mode.</li>
<li>Open up the Twitter login page in a browser session.</li>
<li>Sign in to a Twitter account using your credentials, passed on using appropriate cURL commands.</li>
<li>Scrape the URLs of tweets and redirect the same to a markdown file.</li>
</ul></div>

High-Level Approach

<div class="Creating_a_Browser_Session_card card-content-body"><p>First we need to start the Browser Session so that we can give commands to that Browser Session and then work with the output given by it.
First we need to download the ChromeDriver and then run the ChromeDriver Server as a background process so that we can give further commands to the shell on the same tab.</p></div>

Creating a Browser Session

<div class="Requirements_card card-content-body"><ul>
<li>You will need to have Chrome Browser as it is very reliable.</li>
<li>Download the appropriate ChromeDriver that is according to your Chrome Browser Version.</li>
<li>Run the ChromeDriver Server as a background process so that we can give further commands to the shell on the same tab.</li>
<li>Create a Browser Session using <code>cURL</code> command and save the sessionID in a variable as it will be needed in every command you will use.</li>
<li>Since the ChromeDriver Server gives output in JSON format you may need a <code>jq</code> command for parsing the JSON.</li>
</ul><div class="References"><h3>References</h3><ul>
<li><a href="https://www.crio.do/bytes/linux-basics-one/" target="_blank">Linux Basics 1 - Crio Byte</a></li>
<li><a href="https://www.crio.do/bytes/linux-basics-two/" target="_blank">Linux Basics 2 - Crio Byte</a></li>
<li><a href="https://chromedriver.chromium.org/downloads" target="_blank">ChromeDriver</a></li>
<li><a href="https://linuxize.com/post/how-to-run-linux-commands-in-background/" target="_blank">Run Linux Commands In Background</a></li>
<li><a href="https://www.tecmint.com/linux-curl-command-examples/" target="_blank">Learning cURL</a></li>
<li><a href="https://www.baeldung.com/linux/jq-command-json" target="_blank">jq command for processing JSON</a></li>
<li><a href="https://github.com/SeleniumHQ/selenium/wiki/JsonWireProtocol#session" target="_blank">JsonWireProtocol for communicating with Browser Session</a></li>
</ul></div></div>

Requirements

<div class="Bring_it_On!_card card-content-body"><ul>
<li>Can you open the Browser Session in headless mode? You can use this feature to do scraping without even the Browser popping in front of you.</li>
</ul></div>

Bring it On!

<div class="Expected_Outcome_card card-content-body"><p>You would be able to see the Chrome Browser(if headless is not passed as argument when creating a Browser Session) pop up in front of your screen and terminal waiting for the next command.
<br/></p><p>It would look like this:</p><p><img alt="Chrome-Browser-Window" src="https://storage.googleapis.com/crio-content-container-assets/ME_ME_PROJECT_SCRAPING_TWITTER_MODULE_ME_PROJECT_SCRAPING_TWITTER_MODULE_SCRAPING_TWITTER_1-sample-image.png" style="max-width:750px;"/>
<br/></p></div>

Expected Outcome

<div class="Signing_in_to_Twitter_Account_card card-content-body"><p>Before we go to the bookmarks section of Twitter and start Scraping we first need to sign in into our Twitter Account. For doing so first we need to go <a href="https://twitter.com/login" target="_blank">https://twitter.com/login</a> and then select the input fields and put appropriate values into that fields and then search and click the submit button.</p></div>

Signing in to Twitter Account

<div class="Requirements_card card-content-body"><ul>
<li>First you need to move to <a href="https://twitter.com/login" target="_blank">https://twitter.com/login</a> by sending a request to browser session using appropriate cURL command.</li>
<li>After that try to select the input fields and then send values to the field using the appropriate cURL commands.</li>
<li>Select the submit button and then click the button using appropriate command.</li>
</ul><div class="References"><h3>References</h3><ul>
<li><a href="https://github.com/SeleniumHQ/selenium/wiki/JsonWireProtocol#sessionsessionidurl" target="_blank">Moving to a particular webpage</a></li>
<li><a href="https://github.com/SeleniumHQ/selenium/wiki/JsonWireProtocol#sessionsessionidelementidelement" target="_blank">Searching an element on the webpage</a></li>
<li><a href="https://github.com/SeleniumHQ/selenium/wiki/JsonWireProtocol#sessionsessionidelementidvalue" target="_blank">Send a value to a field</a></li>
<li><a href="https://github.com/SeleniumHQ/selenium/wiki/JsonWireProtocol#sessionsessionidelementidclick" target="_blank">Clicking an element</a></li>
</ul></div><div class="Tip"><h3>Tip</h3><ul>
<li>Use xpath for searching the element.</li>
</ul></div></div>

<div class="Expected_Outcome_card card-content-body"><p>After completing the requirements, you would be signed in with your Twitter Account and can now move on to the next module where we do scraping.
<br/></p></div>

<div class="Scraping_URL_of_tweets_card card-content-body"><p>In this unit we will scrape the URL of the tweets present on <a href="https://twitter.com/i/bookmarks" target="_blank">https://twitter.com/i/bookmarks</a>. For this first we need to move to this page and then smartly locate the elements where URL of the tweets are present.</p></div>

Scraping URL of tweets

<div class="Requirements_card card-content-body"><ul>
<li>First move to <a href="https://twitter.com/i/bookmarks" target="_blank">https://twitter.com/i/bookmarks</a> using the command you used in the previous milestone.</li>
<li>Then try to search for all the elements on which URL of the tweets are present.</li>
</ul><div class="References"><h3>References</h3><ul>
<li><a href="https://github.com/SeleniumHQ/selenium/wiki/JsonWireProtocol#sessionsessionidurl" target="_blank">Moving to a particular webpage</a></li>
<li><a href="https://github.com/SeleniumHQ/selenium/wiki/JsonWireProtocol#sessionsessionidelementidelements" target="_blank">Searching an element on the webpage</a></li>
<li><a href="https://www.guru99.com/xpath-selenium.html" target="_blank">How to find XPATH</a></li>
</ul></div></div>

<div class="Bring_it_On!_card card-content-body"><ul>
<li>Try to locate and scrape the whole tweet i.e account name who posted the tweet, content of the tweet, date when posted, number of likes, number of comments and number of retweets for that tweet.</li>
</ul></div>

<div class="Expected_Outcome_card card-content-body"><p>After fulfilling all the requirements you would be able to grab URL of the tweets present on the screen.
<br/></p></div>

<div class="Moving_the_scroll_bar_slowly_for_grabbing_all_the_tweets_card card-content-body"><p>When we were grabbing the URL of tweets in the previous unit you would have noticed that we were not able to grab all the tweets. Since twitter loads it's tweet dynamically we need to wait for some time after scrolling the scroll bar and we need to do this until the scroll bar does not move even if we give it the command for it.</p></div>

Moving the scroll bar slowly for grabbing all the tweets

<div class="Requirements_card card-content-body"><ul>
<li>Execute a JS command in the browser to move the scroll bar downwards.</li>
<li>Compare the previous position of the scroll bar to the current position of the scroll bar to determine whether the scroll has moved or not. Keep in mind that internet might be slow on some systems so even if we have tweets left and we move the scroll bar it won't move.</li>
</ul><div class="References"><h3>References</h3><ul>
<li><a href="https://github.com/SeleniumHQ/selenium/wiki/JsonWireProtocol#sessionsessionidexecute" target="_blank">Execute a JS snippet</a></li>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Window/pageYOffset" target="_blank">Getting the position of Y scroll bar</a></li>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Window/scrollTo" target="_blank">Moving the scroll bar</a></li>
</ul></div></div>

<div class="Expected_Outcome_card card-content-body"><p>After fulfilling all the requirements you would be able to grab URLs of all the tweets that will be saved in a markdown file.
<br/></p></div>

<div class="Use_associative_array_to_store_the_URLs_&amp;_saving_URLs_in_a_markdown_file_card card-content-body"><p>In this section you will be using associative arrays since when you scroll down you get the same element twice or thrice depending upon how much you scroll down. So with the help of associative array (can be used as set data structure in Bash) we can keep only unique URLs in the array. After that we will save them in a markdown file.</p></div>

Use associative array to store the URLs & saving URLs in a markdown file

<div class="Requirements_card card-content-body"><ul>
<li>Use associative array to store unique URLs.</li>
<li>You can save the URLs in a file using redirection.</li>
</ul><div class="References"><h3>References</h3><ul>
<li><a href="https://linuxhint.com/associative_array_bash/" target="_blank">Associative Array in Bash</a></li>
<li><a href="https://opensource.com/article/20/6/redirection-bash" target="_blank">Redirection in Bash</a></li>
</ul></div></div>

<div class="Bring_it_On!_card card-content-body"><ul>
<li>Can you think of any other way of making a markdown file with unique URLs without using associative arrays.(HINT: You should look into <code>uniq</code> command of bash)</li>
</ul></div>

<div class="Expected_Outcome_card card-content-body"><p>After fulfilling all the requirements you would be able to save the URL of all the tweets in a markdown file.
<br/></p></div>

CONTAINER_REGULAR_TASK_ME_PROJECT_SCRAPING_TWITTER_MODULE_SCRAPING_TWITTER_TASK_1

Creating a Browser Session using cURL

CONTAINER_REGULAR_TASK_ME_PROJECT_SCRAPING_TWITTER_MODULE_SCRAPING_TWITTER_TASK_2

CONTAINER_REGULAR_TASK_ME_PROJECT_SCRAPING_TWITTER_MODULE_SCRAPING_TWITTER_TASK_3

CONTAINER_REGULAR_TASK_ME_PROJECT_SCRAPING_TWITTER_MODULE_SCRAPING_TWITTER_TASK_4

CONTAINER_REGULAR_TASK_ME_PROJECT_SCRAPING_TWITTER_MODULE_SCRAPING_TWITTER_TASK_5

Use associative array to store the URLs

Realtime Video Processing using ChromaKey (GreenScreen) Effect

Companion App: A mental health tracker built using Flutter and Firebase.

Exploratory Analysis of Geolocational Data

Learn to create apps on Kubernetes - for beginners.

OurApp - a social media web app in NodeJS

Visualising and forecasting stocks using Dash