v1.0.0

Author: Chip Nguyen

.gitignore

@@ -0,0 +1,5 @@
+__pycache__
+*/__pycache__
+build/*
+dist/*
+lazynlp.egg-info/*

MANIFEST.in

@@ -0,0 +1,4 @@
+include README.md MANIFEST.in
+include setup.py
+recursive-include . *.txt
+include lazynlp/*.txt

README.md

@@ -0,0 +1,221 @@
+# lazynlp
+
+A straightfoward library that allows you to crawl, clean up, and deduplicate webpages to create massive monolingual datasets. Using this library, you should be able to create datasets larger than the one used by OpenAI for GPT-2.
+
+## Setup
+This library uses Python 3.
+
+1. Clone this library and cd into the lazynlp folder:
+
+```
+git clone https://github.com/chiphuyen/lazynlp.git
+cd lazynlp
+```
+
+2. Install dependencies
+
+``
+pip3 install -r requirements.txt
+``
+
+3. Install the library
+``
+python3 setup.py install
+``
+
+If you want to uninstall the library, use:
+
+``
+pip3 uninstall lazynlp
+``
+
+## How to create a massive dataset using lazynlp:
+
+### Step 1. Obtain URLs of the webpages you want to crawl
+There are several major dumps of URLs available that you can use.
+
+#### Reddit URLs
+This is [the link to all submissions to Reddit by months](https://files.pushshift.io/reddit/submissions/)'s . You can download the raw dump and process to get the links. However, keep in mind that each of these dumps is huge (100MB - 1GB).
+
+[@jcpeterson](https://github.com/jcpeterson) is kind enough to provide a list of deduplicated links with at least 3 karmas that you can download [here](https://drive.google.com/file/d/1hRtA3zZ0K5UHKOQ0_8d0BIc_1VyxgY51/view?usp=sharing).
+
+There are about 23M URLs from between 2015-06 to 2018-10, of which around 40 - 60 \% are bad URLs (URLs no longer exist or aren't scraper-friendly).
+It means that after you've downloaded and cleaned all good URLs from this, you should have approx 10M webpages or 50GB of pure text.
+
+#### Gutenberg
+You can download the list of all URLs to US Gutenberg books [here](). There are 50K books, which convert to about 14GB of pure text.
+
+You can also run ``lazynlp.get_us_gutenberg_links()`` to get the same list. For example, if you want to get all the Gutenberg URLs and store it in the file ``us_gutenberg.urls``:
+
+``
+lazynlp.get_us_gutenberg_links('us_gutenberg.urls')
+``
+
+
+You can download the list of all URLs to Australian Gutenberg books [here](https://drive.google.com/file/d/1C5aSisXMC3S3OXBFbnETLeK3UTUXEXrC/view?usp=sharing). There are 4k books, which convert to about 1GB of pure text.
+
+You can also run ``lazynlp.get_aus_gutenberg_links()`` to get the same list. For example, if you want to get all the Gutenberg URLs and store it in the file ``aus_gutenberg.urls``:
+
+``
+lazynlp.get_aus_gutenberg_links('aus_gutenberg.urls')
+``
+
+#### Wikipedia
+You can download the Wikipedia dumps [here](https://dumps.wikimedia.org/).
+
+
+### Step 2. Deduplicate URLs
+You don't want to download the same URL multiple times. There are two functions that help you deduplicate all URLs:
+
+``
+lazynlp.dedup_lines(files, outfold)
+``
+
+This function takes in a list of files (in each file, each line is a URLs) and deduplicate each file against all previous files.
+Save all the deduplicated files in outfold.
+
+``
+lazynlp.dedup_lines_from_new_file(original_files, new_file, outfile)
+``
+
+This function allows you to deduplicate a new file against all previously deduplicated files (original_files)
+
+### Step 3. Download the URLs
+
+``
+lazynlp.download_pages(link_file, folder, timeout=30, default_skip=True, extensions=[], domains=[])
+``
+
+	
+	"""
+	
+	link_file:
+
+		file contains links to webpages to crawl. Each line contains one URL.
+	
+	folder:
+	
+		folder that you want to contain your downloaded pages.
+	
+	timeout:
+	
+		seconds to wait for a page to respond before abandoning it.
+	
+	default_skip:
+
+		set to True if you want to automatically skip all URLs that contain domains and extensions that are known to be scraper-unfriendly.
+	
+	You can see the list of excluded domains at lazynlp/exclude_domains.txt.
+
+	You can see the list of excluded extensions at lazynlp/exclude_extensions.txt
+	
+	You can also add your own domains and extensions to skip with domains and extensions and arguments.
+	
+	In the folder:
+
+		Each URL is downloaded into a file, indexed by the order in which it is downloaded. The first line of each file is the URL. The rest is the textual content of the page.
+	 	
+	 	index.urls contains all the URLs that have been successfully downloaded.
+		
+		bad.urls contains the URLs that are bad.
+		
+		connection.urls contains the URLs that haven't been downloaded because of connection issues.
+		
+		non_ascii.urls contains the URLs that haven't been downloaded because of bad encoding issues.
+		
+		empty.urls contains the URLs that have empty textual content.
+
+	"""
+
+If you have a lot of URLs, you can divide the list into multiple files and call this function separately. I was able to run 40 scripts in parallel.
+I guess I could have parallizing the code. I just found this to be easier.
+
+If you want to download each webpage separately, call:
+
+``
+lazynlp.download_page(link, ctx=None, timeout=None)
+``
+
+### Step 4. Clean the webpages
+You can get rid of all HTML tags, decode utf-8 into string, transliterate foreign characters, collapse white space, replace unprintable characters, unescape HTML, etc. using methods available in lazynlp/cleaner.py.
+
+You can also just call
+
+``
+lazynlp.clean_page(page)
+``
+
+to do most of it.
+
+Note:
+In this library, the function lazynlp.download_pages() does both the crawling and cleaning part, so the webpages you have are pure text, like this:
+
+```
+http://www.thecannabist.co/2017/03/02/jeff-sessions-russia-resign-democrats/74687/
+Attorney general nominee Sen. Jeff Sessions, R-Ala., testifies on Capitol Hill in Washington on Jan. 10, 2017, in the first day of his confirmation hearing before the Senate Judiciary Committee. Top Democrats now say that because he misled the committee about his visits to Russia, he should resign. (Andrew Harnik, The Associated Press)
+
+House Oversight and Government Reform Committee Chairman Jason Chaffetz, R-Utah, tweeted early Thursday that "AG Sessions should clarify his testimony and recuse himself."
+
+Later, Sen. Rob Portman, R-Ohio, said in a statement, "Jeff Sessions is a former colleague and a friend, but I think it would be best for him and for the country to recuse himself from the DOJ Russia probe."
+
+House Majority Leader Kevin McCarthy, R-Calif., also initially said during an appearance on MSNBC's "Morning Joe" that Sessions should bow out.
+
+Asked whether Sessions should recuse himself in this situation, McCarthy replied "I think the trust of the American people -- you recuse yourself in these situations, yes."
+
+McCarthy was pressed a second time about whether he was calling for Sessions to recuse himself and he confirmed that he believed the situation required a recusal.
+
+"I think it would be easier from that standpoint, yes," McCarthy said.
+
+But McCarthy later said his comment had been misinterpreted, telling Fox News' "Fox and Friends," "I'm not calling on him to recuse himself. I was asked on 'Morning Joe,' if he needs to recuse himself as going forward. As you just heard, Attorney General Sessions said he would recuse himself going forward -- appropriate, and that's all my answer was."
+
+The comments from prominent Republicans follow revelations that Sessions met with the Russian ambassador during election season. Under oath in front of the Senate Judiciary Committee for his confirmation hearing in January, Sessions had said that he had not met with any Russian officials.
+
+Senate Minority Leader Charles Schumer, D-N.Y., joined growing Democratic calls for Sessions to either resign or at least recuse himself from any investigations into Russia's meddling in U.S. elections.
+
+"Attorney General Sessions cannot possibly lead an investigation into Russian interference in our elections or come anywhere near it. With these revelations, he may indeed become the subject of it," Schumer told reporters. "Better for the country if he resigns, but let's get an investigation going."
+
+Because the Department of Justice should be above reproach, for the good of the country, the Attorney General should resign.
+```
+### Step 5. Remove duplicated webpages
+To avoid any piece of texts being over-represented, you want to only include pages that don't signicantly overlap with other pages.
+
+To estimate the amount of overlapping of target files with certain source files, use this function:
+
+``
+lazynlp.estimate_overlap(source_files, target_files, gran='word', n=8, capacity=10000, error_rate=1e-5, header=0, interval=100000)
+``
+
+``gran`` is the granulary of tokens: 'char' or 'word' level. 
+
+``n`` is the n-gram.
+
+``capacity`` and ``error_rate`` are for the BloomFilter used.
+
+``header``: number of lines of each file to skip. It's because in our format, the first line is the url
+
+To estimate the amount of overlapping of a target file with an existing BloomFilter, use this function:
+
+``
+lazynlp.estimate_overlap_bf(bf, target_file, gran='word', n=8, header=0)
+``
+
+If given a list of files, e.g. cleaned webpages, to filter out all the files that contain more than ``threshold`` overlapping with other files, use this function:
+
+``
+lazynlp.filter_files(files, threshold=0.5, gran='word', n=8, capacity=100000000, error_rate=1e-7, header=0, interval=1000000)
+``
+
+Names of all the files that are deemed duplicated are stored in ``dupped_files.list``
+
+Names of all the files used for the dataset are stored in ``clean_files.list``
+
+Some statistics to keep in mind:
+1. 1GB of text is about 1b characters. An English word has on average 4.5 characters, or 5.5 including whitespace.
+So 1GB of text is about 181M words.
+
+2. When I ran 30 scripts in parallel, it took 3 hours to download and clean 1GB of pure text. So it'd take 5 days to get 50GB of pure text.
+
+3. The OpenAI dataset has 40GB, which I estimate to contain about 7-8 billion words.
+If you download all the webpages from the good Reddit URLs and Gutenberg books, you should have a dataset bigger than OpenAI's WebText.
+
+4. OpenAI, in their paper for GPT-2, didn't include Wikipedia artciles for fear of overlapping. You can choose to include Wikipedia articles that have less than a certain amount of overlapping with the existing dataset using ``lazynlp.estimate_overlap_bf(bf, target_file, gran='word', n=8``.

lazynlp/__init__.py

@@ -0,0 +1,7 @@
+__VERSION__ = '0.0.1'
+
+from .analytics import *
+from .cleaner import *
+from .create import *
+from .crawl import *
+from .utils import *

lazynlp/analytics.py

@@ -0,0 +1,150 @@
+import os
+import random
+import time
+
+from pybloom import BloomFilter
+
+from lazynlp.cleaner import *
+from lazynlp.utils import *
+
+def build_ngram_from_tokens(tokens, n):
+	""" Create a dictionary of n-gram from the list of tokens
+	"""
+	count = {}
+	curr = tokens[:n]
+	count[' '.join(curr)] = 1
+	for token in tokens[n:]:
+		curr = curr[1:] + [token]
+		string = ' '.join(curr)
+		if not string in count:
+			count[string] = 0
+		count[string] += 1
+	return count
+
+def build_ngram(file, outfile=None, bf=None, gran='word', n=10, uncase=True, alphanumeric=True, interval=100000):
+	"""
+	gran: granularity of the token. It can be 'word' or 'char'
+	bf: BloomFilter to update the existence of n-grams. Use when the file is too large to store a dictionary count
+	alphanumeric: whether to keep only alphanumeric characters and space.
+	outfile: if outfile is specified, build dictionary of n-grams and write it to outfile
+	interval: how often to report the progress.
+	"""
+	if not gran in set(['word', 'char']):
+		raise ValueError("gran has to be 'word' or 'char'")
+	count = {}
+	f = open(file, 'r')
+	i = 1
+	line = f.readline()
+	start = time.time()
+	while line:
+		line = line.strip()
+		if line:
+			if uncase:
+				line = line.lower()
+			
+			if gran == 'word':
+				if alphanumeric:
+					line = remove_non_alphanumeric(line)
+			else:
+				line = remove_non_alpha(line)
+			line = collapse_white_spaces(line)
+			tokens = line.split()
+			line_count = build_ngram_from_tokens(tokens, n)
+
+			if outfile:
+				count.update()
+			
+			if not bf is None:
+				for key in line_count:
+					bf.add(key)
+
+			if interval > 0 and i % interval == 0:
+				print('Process line: {}. Time: {}'.format(i, time.time() - start))
+				start = time.time()
+
+			i += 1
+
+		line = f.readline()
+
+	f.close()
+
+	if outfile:
+		outfold = outfile[:outfile.rfind('/')]
+		os.makedirs(outfold, exist_ok=True)
+		dict_sorted_2_file(count, os.path.join(outfile.format(n)))
+
+	if bf:
+		return bf
+
+	return count
+
+def build_word_ngram(file, outfile, n=10, alphanumeric=True, norm=True, interval=100000):
+	""" Build word ngrams and store in outfile
+	n-grams in the format:
+	[n-gram][tab][count]
+
+	If alphanumeric, exclude all the words that contain non-alphanumeric characters
+	"""
+	return build_ngram(file, outfile=outfile, n=n, gran='word', alphanumeric=alphanumeric, norm=norm, interval=interval)
+
+def build_char_ngram(file, outfile, n=10, interval=100000):
+	"""
+	Build character n-grams and store in outfile
+	"""
+	return build_ngram(file, outfile=outfile, n=n, gran='char', interval=interval)
+
+def estimate_overlap(source_files, target_files, gran='word', n=8, capacity=10000, error_rate=1e-5, header=0, interval=100000):
+	""" Estimate overlapping of target_files with source_files using n-grams
+	gran: granularity of the token. It can be 'word' or 'char'
+	header: number of lines of each file to skip. It's because in our format, the first line is the url
+	"""
+	if not gran in set(['word', 'char']):
+		raise ValueError("gran has to be 'word' or 'char'")
+	if isinstance(source_files, str):
+		source_files = [source_files]
+	if isinstance(target_files, str):
+		target_files = [target_files]
+	
+	bf = BloomFilter(capacity=capacity, error_rate=error_rate)
+	for source_file in source_files:
+		bf = build_ngram(file=source_file, bf=bf, gran=gran, n=n, uncase=True, alphanumeric=True, interval=interval)
+
+	results = []
+	for file in target_files:
+		print(file)
+		results.append(estimate_overlap_bf(bf, file, gran=gran, n=8, header=header))
+	return results
+
+def estimate_overlap_bf(bf, target_file, gran='word', n=8, header=0):
+	""" Estimate overlapping of target_file with an existing bloomfilter
+	gran: granularity of the token. It can be 'word' or 'char'
+	"""
+	if not gran in set(['word', 'char']):
+		raise ValueError("gran has to be 'word' or 'char'")
+		
+	f = open(target_file, 'r')
+	for _ in range(header + 1):
+		line = f.readline()
+
+	total, seen = 0, 0
+	while line:
+		line = line.strip().lower()
+		
+		if gran == 'word':
+			line = remove_non_alphanumeric(line)
+		else:
+			line = remove_non_alpha(line)
+		line = collapse_white_spaces(line)
+		tokens = line.split()
+		line_count = build_ngram_from_tokens(tokens, n)
+
+		for key in line_count:
+			if key in bf:
+				seen += 1
+			total += 1
+
+		line = f.readline()
+	
+	result = seen / total
+	print('{} seen out of {}: {}'.format(seen, total, result))
+	return result