R4CSS-Crawlers

Tutorials

Docs:

• R4DS-Ch2-5.pdf

Slides

• slide: What is json?
• slide: Crawler designs
• slide: XPath and CSS selector
• slide: HTML parser

Videos

Prior knowledge

1. What is json?
2. Reading json with R
3. Reading json (advanced)

Getting JSON from web

1. Finding data url
2. Scraping 104

Getting and Parsing HTML files

1. Understanding HTML
2. Parsing HTML files 1 -> Parsing HTML files 2
3. Scraping data with POST(): Scraping ibon address as an example
4. Scraping data with cookies: Scraping PTT Gosshiping as an example
5. Scraping PTT post content

Open data links

• Youbike: https://tcgbusfs.blob.core.windows.net/blobyoubike/YouBikeTP.gz

Inspecting data links in the following service

1. 空品資料中央監測資料https://airtw.epa.gov.tw/CHT/EnvMonitoring/Central/CentralMonitoring.aspx
2. Dcard感情版最新文章 https://www.dcard.tw/f/relationship?latest=true
3. 104 job search https://www.104.com.tw/ (Must solve cookies)
4. PTT boy-girl https://www.ptt.cc/bbs/Boy-Girl/index.html
5. ibon address https://www.ibon.com.tw/mobile/retail_inquiry.aspx#gsc.tab=0
6. ltn news search https://news.ltn.com.tw/search?keyword=%E8%82%BA%E7%82%8E
7. cnyes.com