Releases: piskvorky/gensim
Docs 💬
3.5.0, 2018-07-06
This release comprises a glorious 38 pull requests from 28 contributors. Most of the effort went into improving the documentation—hence the release code name "Docs 💬"!
Apart from the massive overhaul of all Gensim documentation (including docstring style and examples—you asked for it), we also managed to sneak in some new functionality and a number of bug fixes. As usual, see the notes below for a complete list, with links to pull requests for more details.
Huge thanks to all contributors! Nobody loves working on documentation. 3.5.0 is a result of several months of laborious, unglamorous, and sometimes invisible work. Enjoy!
📚 Documentation improvements
- Overhaul documentation for
*2vecmodels (@steremma & @piskvorky & @menshikh-iv, #1944, #2087) - Fix documentation for LDA-related models (@steremma & @piskvorky & @menshikh-iv, #2026)
- Fix documentation for utils, corpora, inferfaces (@piskvorky & @menshikh-iv, #2096)
- Update non-API docs (about, intro, license etc) (@piskvorky & @menshikh-iv, #2101)
- Refactor documentation for
gensim.models.phrases(@CLearERR & @menshikh-iv, #1950) - Fix HashDictionary documentation (@piskvorky, #2073)
- Fix docstrings for
gensim.models.AuthorTopicModel(@souravsingh & @menshikh-iv, #1907) - Fix docstrings for HdpModel, lda_worker & lda_dispatcher (@gyanesh-m & @menshikh-iv, #1912)
- Fix format & links for
gensim.similarities.docsim(@CLearERR & @menshikh-iv, #2030) - Remove duplication of class documentation for
IndexedCorpus(@darindf, #2033) - Refactor documentation for
gensim.models.coherencemodel(@CLearERR & @menshikh-iv, #1933) - Fix docstrings for
gensim.sklearn_api(@steremma & @menshikh-iv, #1895) - Disable google-style docstring support (@menshikh-iv, #2106)
- Fix docstring of
gensim.models.KeyedVectors.similarity_matrix(@Witiko, #1971) - Consistently use
smart_open()instead ofopen()in notebooks (@sharanry, #1812)
🌟 New features:
- Add
add_entitymethod toKeyedVectorsto allow adding word vectors manually (@persiyanov, #1957) - Add inference for new unseen author to
AuthorTopicModel(@Stamenov, #1766) - Add
evaluate_word_analogies(will replaceaccuracy) method toKeyedVectors(@akutuzov, #1935) - Add Pivot Normalization to
TfidfModel(@markroxor, #1780)
👍 Improvements
- Allow initialization with
max_final_vocabin lieu ofmin_countinWord2Vec(@aneesh-joshi, #1915) - Add
dtypeargument forchunkize_serialinLdaModel(@darindf, #2027) - Increase performance in
Phrases.analyze_sentence(@JonathanHourany, #2070) - Add
ns_exponentparameter to control the negative sampling distribution for*2vecmodels (@fernandocamargoti, #2093)
🔴 Bug fixes:
- Fix
Doc2Vec.infer_vector+ notebook cleanup (@gojomo, #2103) - Fix linear decay for learning rate in
Doc2Vec.infer_vector(@umangv, #2063) - Fix negative sampling floating-point error for `gensim.models.Poincare (@jayantj, #1959)
- Fix loading
word2vecanddoc2vecmodels saved using old Gensim versions (@manneshiva, #2012) - Fix
SoftCosineSimilarity.get_similaritieson corpora ssues/1955) (@Witiko, #1972) - Fix return dtype for
matutils.unitvecaccording to input dtype (@o-P-o, #1992) - Fix passing empty dictionary to
gensim.corpora.WikiCorpus(@steremma, #2042) - Fix bug in
Similarity.query_shardsin multiprocessing case (@bohea, #2044) - Fix SMART from TfidfModel for case when
df == "n"(@PeteBleackley, #2021) - Fix OverflowError when loading a large term-document matrix in compiled MatrixMarket format (@arlenk, #2001)
- Update rules for removing table markup from Wikipedia dumps (@chaitaliSaini, #1954)
- Fix
_is_singlefromPhrasesfor case when corpus is a NumPy array (@rmalouf, #1987) - Fix tests for
EuclideanKeyedVectors.similarity_matrix(@Witiko, #1984) - Fix deprecated parameters in
D2VTransformerandW2VTransformer(@MritunjayMohitesh, #1945) - Fix
Doc2Vec.infer_vectorafter loading oldDoc2Vec(gensim<=3.2)(@manneshiva, #1974) - Fix inheritance chain for
load_word2vec_format(@DennisChen0307, #1968) - Update Keras version (avoid bug from
keras==2.1.5) (@menshikh-iv, #1963)
⚠️ Deprecations (will be removed in the next major release)
-
Remove
gensim.models.wrappers.fasttext(obsoleted by the new nativegensim.models.fasttextimplementation)gensim.examplesgensim.nosygensim.scripts.word2vec_standalonegensim.scripts.make_wiki_lemmagensim.scripts.make_wiki_onlinegensim.scripts.make_wiki_online_lemmagensim.scripts.make_wiki_online_nodebuggensim.scripts.make_wiki(all of these obsoleted by the new nativegensim.scripts.segment_wikiimplementation)- "deprecated" functions and attributes
-
Move
gensim.scripts.make_wikicorpus➡gensim.scripts.make_wiki.pygensim.summarization➡gensim.models.summarizationgensim.topic_coherence➡gensim.models._coherencegensim.utils➡gensim.utils.utils(old imports will continue to work)gensim.parsing.*➡gensim.utils.text_utils
3.4.0
3.4.0, 2018-03-01
🌟 New features:
-
Massive optimizations of
gensim.models.LdaModel: much faster training, using Cython. (@arlenk, #1767)-
Training benchmark 💥
dataset old LDA [sec] optimized LDA [sec] speed up nytimes 3473 1975 1.76x enron 774 437 1.77x -
This change affects all models that depend on
LdaModel, such asLdaMulticore,LdaSeqModel,AuthorTopicModel.
-
-
Huge speed-ups to corpus I/O with
MmCorpus(Cython) (@arlenk, #1825)-
File reading benchmark
dataset file compressed? old MmReader [sec] optimized MmReader [sec] speed up enron no 22.3 2.6 8.7x yes 37.3 14.4 2.6x nytimes no 419.3 49.2 8.5x yes 686.2 275.1 2.5x text8 no 25.4 2.5 10.1x yes 41.9 17.0 2.5x -
Overall, a 2.5x speedup for compressed
.mm.gzinput and 8.5x 🔥🔥🔥 for uncompressed plaintext.mm.
-
-
Performance and memory optimization to
gensim.models.FastText🚀 (@jbaiter, #1916)-
Benchmark (first 500,000 articles from English Wikipedia)
Metric old FastText optimized FastText improvement Training time (1 epoch) 4823.4s (80.38 minutes) 1873.6s (31.22 minutes) 2.57x Training time (full) 1h 26min 13s 36min 43s 2.35x Training words/sec 72,781 187,366 2.57x Training peak memory 5.2 GB 3.7 GB 1.4x -
Overall, a 2.5x speedup & memory usage reduced by 30%.
-
-
Implemented Soft Cosine Measure (@Witiko, #1827)
-
New method for assessing document similarity, a nice faster alternative to WMD, Word Mover's Distance
-
Benchmark
Technique MAP score Duration softcossim 45.99 1.24 sec wmd-relax 44.48 12.22 sec cossim 44.22 4.39 sec wmd-gensim 44.08 98.29 sec -
Soft Cosine notebook with detailed description, examples & benchmarks
-
Related papers:
-
👍 Improvements:
- New method to show the Gensim installation parameters:
python -m gensim.scripts.package_info --info. Use this when reporting problems, for easier debugging. Fix #1902 (@sharanry, #1903) - Added a flag to optionally skip network-related tests, to help maintainers avoid network issues with CI services (@menshikh-iv, #1930)
- Added
licensefield tosetup.py, allowing the use of tools likepip-licenses(@nils-werner, #1909)
🔴 Bug fixes:
- Fix Python 3 compatibility for
gensim.corpora.UciCorpus.save_corpus(@darindf, #1875) - Add
wvproperty to KeyedVectors for backward compatibility. Fix #1882 (@manneshiva, #1884) - Fix deprecation warning from
inspect.getargspec. Fix #1878 (@aneesh-joshi, #1887) - Add
LabeledSentencetogensim.models.doc2vecfor backward compatibility. Fix #1886 (@manneshiva, #1891) - Fix empty output bug in
Phrases(when usingmodel[tokens]twice). Fix #1401 (@sj29-innovate, #1853) - Fix type problems for
D2VTransformer.fit_transform. Fix #1834 (@Utkarsh-Mishra-CIC, #1845) - Fix
datatypeparameter forKeyedVectors.load_word2vec_format. Fix #1682 (@pushpankar, #1819) - Fix deprecated parameters in
doc2vec-leenotebook (@TheFlash10, #1918) - Fix file-like closing bug in
gensim.corpora.MmCorpus. Fix #1869 (@sj29-innovate, #1911) - Fix precision problem in
test_similarities.py, no more FP fails. (@menshikh-iv, #1928) - Fix encoding in Lee corpus reader. (@menshikh-iv, #1931)
- Fix OOV pairs counter in
WordEmbeddingsKeyedVectors.evaluate_word_pairs. (@akutuzov, #1934)
📚 Tutorial and doc improvements:
- Fix example block for
gensim.models.Word2Vec(@nzw0301, #1870) - Fix
doc2vec-leenotebook (@numericlee, #1870) - Store images from
README.mddirectly in repository. Fix #1849 (@ibrahimsharaf, #1861) - Add windows venv activate command to
CONTRIBUTING.md(@aneesh-joshi, #1880) - Add anaconda-cloud badge. Partial fix #1901 (@sharanry, #1905)
- Fix docstrings for lsi-related code (@steremma, #1892)
- Fix parameter description of
sgparameter forgensim.models.word2vec(@mdcclv, #1919) - Refactor documentation for
gensim.similarities.docsimandMmCorpus-related. (@CLearERR & @menshikh-iv, #1910) - Fix docstrings for
gensim.test.utils(@yurkai & @menshikh-iv, #1904) - Refactor docstrings for
gensim.scripts. Partial fix #1665 (@yurkai & @menshikh-iv, #1792) - Refactor API reference
gensim.corpora. Partial fix #1671 (@CLearERR & @menshikh-iv, #1835) - Fix documentation for
gensim.models.wrappers(@kakshay21 & @menshikh-iv, #1859) - Fix docstrings for
gensim.interfaces(@yurkai & @menshikh-iv, #1913)
⚠️ Deprecations (will be removed in the next major release)
-
Remove
gensim.models.wrappers.fasttext(obsoleted by the new nativegensim.models.fasttextimplementation)gensim.examplesgensim.nosygensim.scripts.word2vec_standalonegensim.scripts.make_wiki_lemmagensim.scripts.make_wiki_onlinegensim.scripts.make_wiki_online_lemmagensim.scripts.make_wiki_online_nodebuggensim.scripts.make_wiki(all of these obsoleted by the new nativegensim.scripts.segment_wikiimplementation)- "deprecated" functions and attributes
-
Move
- `gensim.scripts.make_wikicorpus...
3.3.0
3.3.0, 2018-02-02
🌟 New features:
-
Re-designed all "*2vec" implementations (@manneshiva, #1777)
- Modular organization of
Word2Vec,Doc2Vec,FastText, etc ..., making it easier to add new models in the future and re-use code - Fully backward compatible (even with loading models stored by a previous Gensim version)
- Detailed documentation for the *2vec refactoring project
- Modular organization of
-
Improve
gensim.scripts.segment_wikiby retaining interwiki links. Fix #1712
(@steremma, PR #1839)-
Optionally extract interlinks from Wikipedia pages (use the
--include-interlinksoption). This will output one additional JSON dict for each article:{ "interlinks": { "article title 1": "interlink text 1", "article title 2": "interlink text 2", ... } } -
Example: extract the Wikipedia graph with article links as edges, from a raw Wikipedia dump:
python -m gensim.scripts.segment_wiki --include-interlinks --file ~/Downloads/enwiki-latest-pages-articles.xml.bz2 --output ~/Desktop/enwiki-latest.jsonl.gz
- Read this field from the
segment_wikioutput:
import json from smart_open import smart_open with smart_open("enwiki-latest.jsonl.gz") as infile: for doc in infile: doc = json.loads(doc) src_node = doc['title'] dst_nodes = doc['interlinks'].keys() print(u"Source node: {}".format(src_node)) print(u"Destination nodes: {}".format(u", ".join(dst_nodes))) break """ OUTPUT: Source node: Anarchism Destination nodes: anarcha-feminist, Ivan Illich, Adolf Brand, Josiah Warren, will (philosophy), anarcha-feminism, Anarchism in Mexico, Lysander Spooner, English Civil War, G8, Sebastien Faure, Nihilist movement, Sébastien Faure, Left-wing politics, imamate, Pierre Joseph Proudhon, anarchist communism, Università popolare (Italian newspaper), 1848 Revolution, Synthesis anarchism, labour movement, anarchist communists, collectivist anarchism, polyamory, post-humanism, postcolonialism, anti war movement, State (polity), security culture, Catalan people, Stoicism, Progressive education, stateless society, Umberto I of Italy, German language, Anarchist schools of thought, NEFAC, Jacques Ellul, Spanish Communist Party, Crypto-anarchism, ruling class, non-violence, Platformist, The History of Sexuality, Revolutions of 1917–23, Federación Anarquista Ibérica, propaganda of the deed, William B. Greene, Platformism, mutually exclusive, Fraye Arbeter Shtime, Adolf Hitler, oxymoron, Paris Commune, Anarchism in Italy#Postwar years and today, Oranienburg, abstentionism, Free Society, Henry David Thoreau, privative alpha, George I of Greece, communards, Gustav Landauer, Lucifer the Lightbearer, Moses Harman, coercion, regicide, rationalist, Resistance during World War II, Christ (title), Bohemianism, individualism, Crass, black bloc, Spanish Revolution of 1936, Erich Mühsam, Empress Elisabeth of Austria, Free association (communism and anarchism), general strike, Francesc Ferrer i Guàrdia, Catalan anarchist pedagogue and free-thinker, veganarchism, Traditional knowledge, Japanese Anarchist Federation, Diogenes of Sinope, Hierarchy, sexual revolution, Naturism, Bavarian Soviet Republic, February Revolution, Eugene Varlin, Renaissance humanism, Mexican Liberal Party, Friedrich Engels, Fernando Tarrida del Mármol, Caliphate, Marxism, Jesus, John Cage, Umanita Nova, Anarcho-pacifism, Peter Kropotkin, Religious anarchism, Anselme Bellegarrigue, civilisation, moral obligation, hedonist, Free Territory (Ukraine), -ism, neo-liberalism, Austrian School, philosophy, freethought, Joseph Goebbels, Conservatism, anarchist economics, Cavalier, Maximilien de Robespierre, Comstockery, Dorothy Day, Anarchism in France, Fédération anarchiste, World Economic Forum, Amparo Poch y Gascón, Sex Pistols, women's rights, collectivisation, Taoism, common ownership, William Batchelder Greene, Collective farming, popular education, biphobia, targeted killings, Protestant Christianity, state socialism, Marie François Sadi Carnot, Stephen Pearl Andrews, World Trade Organization, Communist Party of Spain (main), Pluto Press, Levante, Spain, Alexander Berkman, Wilhelm Weitling, Kharijites, Bolshevik, Liberty (1881–1908), Anarchist Aragon, social democrats, Dielo Truda, Post-left anarchy, Age of Enlightenment, Blanquism, Walden, mutual aid (organization), Far-left politics, privative, revolutions of 1848, anarchism and nationalism, punk rock, Étienne de La Boétie, Max Stirner, Jacobin (politics), agriculture, anarchy, Confederacion General del Trabajo de España, toleration, reformism, International Anarchist Congress of Amsterdam, The Ego and Its Own, Ukraine, Civil Disobedience (Thoreau), Spanish Civil War, David Graeber, Anarchism and issues related to love and sex, James Guillaume, Insurrectionary anarchism, Political repression, International Workers' Association, Barcelona, Bulgaria, Voline, Zeno of Citium, anarcho-communists, organized religion, libertarianism, bisexuality, Ricardo Flores Magón, Henri Zisly, Eight-hour day, Freetown Christiania, heteronormativity, Mikhail Bakunin, Propagandaministerium, Ezra Heywood, individual reappropriation, Modern School (United States), archon, Confédération nationale du travail, socialist movement, History of Islam, Max Nettlau, Political Justice, Reichstag fire, Anti-Christianity, decentralised, Issues in anarchism#Communism, deschooling, Christian movement, squatter, Anarchism in Germany, Catalonia, Louise Michel, Solidarity Federation, What is Property?, European individualist anarchism, Pierre-Joseph Proudhon, Mexican Revolution, wikt:anarchism, Blackshirts, Jewish anarchism, Russian Civil War, property rights, anti-authoritarian, individual reclamation, propaganda by the deed, from each according to his ability, to each according to his need, Feminist movement, Confiscation, social anarchism, Anarchism in Russia, Daniel Guérin, Uruguayan Anarchist Federation, Anarcha-feminism, Enragés, Cynicism (philosophy), workers' council, The Word (free love), Allen Ginsberg, Campaign for Nuclear Disarmament, antimilitarism, Workers' self-management, Federación Obrera Regional Argentina, self-governance, free market, Carlos I of Portugal, Simon Critchley, Anti-clericalism, heterosexual, Layla AbdelRahim, Mexican Anarchist Federation, Anarchism and Marxism, October Revolution, Anti-nuclear movement, Joseph Déjacque, Bolsheviks, Luigi Fabbri, morality, Communist party, Sam Dolgoff, united front, Ammon Hennacy, social ecology, commune (intentional community), Oscar Wilde, French Revolution, egoist anarchism, Comintern, transphobia, anarchism without adjectives, social control, means of production, Michel Onfray, Anarchism in France#The Fourth Republic (1945–1958), syndicalism, Anarchism in Spain, Iberian Anarchist Federation, International of Anarchist Federations, Emma Goldman, Netherlands, anarchist free school, International Workingmen's Association, Queer anarchism, Cantonal Revolution, trade unionism, Karl Marx, LGBT community, humanism, Anti-fascism, Carrara, political philosophy, Anarcho-transhumanism, libertarian socialist, Russian Revolution (1917), Two Cheers for Anarchism: Six Easy Pieces on Autonomy, Dignity, and Meaningful Work and Play, Emile Armand, insurrectionary anarchism, individual, Zhuang Zhou, Free Territory, White movement, Greenwich Village, Virginia Bolten, transcendentalist, public choice theory, wikt:brigand, Issues in anarchism#Participation in statist democracy, free love, Mutualism (economic theory), Anarchist St. Imier International, censorship, federalist, 6 February 1934 crisis, biennio rosso, anti-clerical, centralism, Anarchism: A Documentary History of Libertarian Ideas, minarchism, James C. Scott, First International, homosexuality, political theology, spontaneous order, Oranienburg concentration camp, anarcho-communism, negative liberty, post-modernism, Anarchism in Italy, Leopold Kohr, union of egoists, counterculture, Miguel Gimenez Igualada, philosophical anarchism, International Libertarian Solidarity, homosexual, Counterculture of the 1960s, Errico Malatesta, strikebreaker, Workers' Party of Marxist Unification, Clifford Harper, Reification (fallacy), patriarchy, anarchist law, Apostle (Christian), market (economics), Summerhill School, positive liberty, socialism, feminism, Direct action, Melchor Rodríguez García, William Godwin, Nazi concentration camps, Synthesist anarchism, Margaret Anderson, Han Ryner, Federation of Organized Trades and Labor Unions, technology, Workers Solidarity Movement, Edmund Burke, Encyclopædia Britannica, state (polity), Herbert Read, Park Güell, utilitarian, far right leagues, Limited government, self-ownership, Pejorative, homophobia, Industrial Workers of the World, The Dispossessed, Hague Congress (1872), Stalinism, Reciprocity (cultural anthropology), Fernand Pelloutier, individualist anarchism in France, The False Principle of our Education, individualist anarchism, Pierre Monatte, Soviet Union, counter-economics, Rudolf Rocker, Anarchism and capitalism, Parma, Black Rose Books, lesbian, Arditi del Popolo, Emile Armand (1872–1962), who propounded the virtues of free love in the Parisian anarchist milieu of the early 20th century, collectivism, Development criticism, John Henry Mackay, Benoît Broutchoux, Illegalism, Laozi, feminist, Christiaan Cornelissen, Syndicalist Workers' Federation, anarcho-syndicalism, A...
- Read this field from the
-
Christmas Come Early
3.2.0, 2017-12-09
🌟 New features:
-
New download API for corpora and pre-trained models (@chaitaliSaini & @menshikh-iv, #1705 & #1632 & #1492)
- Download large NLP datasets in one line of Python, then use with memory-efficient data streaming:
import gensim.downloader as api for article in api.load("wiki-english-20171001"): print(article)
- Don’t waste time searching for good word embeddings, use the curated ones:
import gensim.downloader as api model = api.load("glove-twitter-25") model.most_similar("engineer") # [('specialist', 0.957542896270752), # ('developer', 0.9548177123069763), # ('administrator', 0.9432312846183777), # ('consultant', 0.93915855884552), # ('technician', 0.9368376135826111), # ('analyst', 0.9342101216316223), # ('architect', 0.9257484674453735), # ('engineering', 0.9159940481185913), # ('systems', 0.9123805165290833), # ('consulting', 0.9112802147865295)]
- Blog post introducing the API and design decisions.
- Jupyter notebook with examples
- Download large NLP datasets in one line of Python, then use with memory-efficient data streaming:
-
New model: Poincaré embeddings (@jayantj, #1696 & #1700 & #1757 & #1734)
- Embed a graph (taxonomy) in the same way as word2vec embeds words:
from gensim.models.poincare import PoincareRelations, PoincareModel from gensim.test.utils import datapath data = PoincareRelations(datapath('poincare_hypernyms.tsv')) model = PoincareModel(data) model.kv.most_similar("cat.n.01") # [('kangaroo.n.01', 0.010581353439700418), # ('gib.n.02', 0.011171531439892076), # ('striped_skunk.n.01', 0.012025106076442395), # ('metatherian.n.01', 0.01246679759214648), # ('mammal.n.01', 0.013281303506525968), # ('marsupial.n.01', 0.013941330203709653)]
- Tutorial on Poincaré embeddings (Jupyter notebook).
- Model introduction and the journey of its implementation (blog post).
- Original paper on arXiv.
- Embed a graph (taxonomy) in the same way as word2vec embeds words:
-
Optimized FastText (@manneshiva, #1742)
- New fast multithreaded implementation of FastText, natively in Python/Cython. Deprecates the existing wrapper for Facebook’s C++ implementation.
import gensim.downloader as api from gensim.models import FastText model = FastText(api.load("text8")) model.most_similar("cat") # [('catnip', 0.8538144826889038), # ('catwalk', 0.8136177062988281), # ('catchy', 0.7828493118286133), # ('caf', 0.7826495170593262), # ('bobcat', 0.7745151519775391), # ('tomcat', 0.7732658386230469), # ('moat', 0.7728310823440552), # ('caye', 0.7666271328926086), # ('catv', 0.7651021480560303), # ('caveat', 0.7643581628799438)]
- New fast multithreaded implementation of FastText, natively in Python/Cython. Deprecates the existing wrapper for Facebook’s C++ implementation.
-
Binary pre-compiled wheels for Windows, OSX and Linux (@menshikh-iv, MacPython/gensim-wheels/#7)
- Users no longer need to have a C compiler for using the fast (Cythonized) version of word2vec, doc2vec, fasttext etc.
- Faster Gensim pip installation
-
Added
DeprecationWarningsto deprecated methods and parameters, with a clear schedule for removal.
👍 Improvements:
- Add Montemurro and Zanette's entropy based keyword extraction algorithm. Fix #665 (@PeteBleackley, #1738)
- Fix flake8 E731, E402, refactor tests & sklearn API code. Partial fix #1644 (@horpto, #1689)
- Reduce distribution size. Fix #1698 (@menshikh-iv, #1699)
- Improve
scan_vocabspeed,build_vocab_from_freqmethod (@jodevak, #1695) - Improve
segment_wikiscript (@piskvorky, #1707) - Add custom
dtypesupport forLdaModel. Partially fix #1576 (@xelez, #1656) - Add
doc2idxmethod forgensim.corpora.Dictionary. Fix #1634 (@roopalgarg, #1720) - Add tox and pytest to gensim, integration with Travis and Appveyor. Fix #1613, #1644 (@menshikh-iv, #1721)
- Add flag for hiding outdated data for
gensim.downloader.info(@menshikh-iv, #1736) - Add reproducible order between Python versions for
gensim.corpora.Dictionary(@formi23, #1715) - Update
tox.ini,setup.cfg,README.md(@menshikh-iv, #1741) - Add optimized
logsumexpforLdaModel(@arlenk, #1745)
🔴 Bug fixes:
- Fix ranking formula in
gensim.summarization.bm25. Fix #1718 (@souravsingh, #1726) - Fixed incompatibility in persistence for
FastTextwrapper. Fix #1642 (@chinmayapancholi13, #1723) - Fix
gensim.sklearn_apibug withdocuments_columnsparameter. Fix #1676 (@chinmayapancholi13, #1704) - Fix slowdown of CI, remove pytest-cov (@menshikh-iv, #1728)
- Replace outdated packages in Dockerfile (@rbahumi, #1730)
- Replace
num_wordstotopninLdaMallet.show_topics. Fix #1747 (@apoorvaeternity, #1749) - Fix
os.renamefromgensim.downloaderwhen 'src' and 'dst' on different partitions (@anotherbugmaster, #1733) - Fix
DeprecationWarningfromlogsumexp(@dreamgonfly, #1703) - Fix backward compatibility problem in
Phrases.load. Fix #1751 (@alexgarel, #1758) - Fix
load_word2vec_formatfromFastText. Fix #1743 (@manneshiva, #1755) - Fix ipython kernel version in
Dockerfile. Fix #1762 (@rbahumi, #1764) - Fix writing in
segment_wiki(@horpto, #1763) - Fix write method of file requires byte-like object in
segment_wiki(@horpto, #1750) - Fix incorrect vectors learned during online training for
FastText. Fix #1752 (@manneshiva, #1756) - Fix
dtypeofmodel.wv.syn0_vocabon updatingvocabforFastText. Fix #1759 (@manneshiva, #1760) - Fix hashing-trick from
FastText.build_vocab. Fix #1765 (@manneshiva, #1768) - Add explicit
DeprecationWarningfor all outdated stuff. Fix #1753 (@menshikh-iv, #1769) - Fix epsilon according to
dtypeinLdaModel(@menshikh-iv, #1770)
📚 Tutorial and doc improvements:
- Update perf numbers of
segment_wiki(...
3.1.0
3.1.0, 2017-11-06
🌟 New features:
-
Massive optimizations to LSI model training (@isamaru, #1620 & #1622)
-
LSI model allows use of single precision (float32), to consume 40% less memory while being 40% faster.
-
LSI model can now also accept CSC matrix as input, for further memory and speed boost.
-
Overall, if your entire corpus fits in RAM: 3x faster LSI training (SVD) in 4x less memory!
# just an example; the corpus stream is up to you streaming_corpus = gensim.corpora.MmCorpus("my_tfidf_corpus.mm.gz") # convert your corpus to a CSC sparse matrix (assumes the entire corpus fits in RAM) in_memory_csc_matrix = gensim.matutils.corpus2csc(streaming_corpus, dtype=np.float32) # then pass the CSC to LsiModel directly model = LsiModel(corpus=in_memory_csc_matrix, num_topics=500, dtype=np.float32)
-
Even if you continue to use streaming corpora (your training dataset is too large for RAM), you should see significantly faster processing times and a lower memory footprint. In our experiments with a very large LSI model, we saw a drop from 29 GB peak RAM and 38 minutes (before) to 19 GB peak RAM and 26 minutes (now):
model = LsiModel(corpus=streaming_corpus, num_topics=500, dtype=np.float32)
-
-
Add common terms to Phrases. Fix #1258 (@alexgarel, #1568)
-
Phrases allows to use common terms in bigrams. Before, if you are searching to reveal ngrams like
car_with_driverandcar_without_driver, you can either remove stop words before processing, but you will only findcar_driver, or you won't find any of those forms (because they have three words, but also because high frequency of with will avoid them to be scored correctly), inspired by ES common grams token filter.phr_old = Phrases(corpus) phr_new = Phrases(corpus, common_terms=stopwords.words('en')) print(phr_old[["we", "provide", "car", "with", "driver"]]) # ["we", "provide", "car_with", "driver"] print(phr_new[["we", "provide", "car", "with", "driver"]]) # ["we", "provide", "car_with_driver"]
-
-
New segment_wiki.py script (@menshikh-iv, #1483 & #1694)
-
CLI script for processing a raw Wikipedia dump (the xml.bz2 format provided by MediaWiki) to extract its articles in a plain text format. It extracts each article's title, section names and section content and saves them as json-line:
python -m gensim.scripts.segment_wiki -f enwiki-latest-pages-articles.xml.bz2 | gzip > enwiki-latest-pages-articles.json.gz
Processing the entire English Wikipedia dump (13.5 GB, link here) takes about 2.5 hours (i7-6700HQ, SSD).
The output format is one article per line, serialized into JSON:
for line in smart_open('enwiki-latest-pages-articles.json.gz'): # read the file we just created article = json.loads(line) print("Article title: %s" % article['title']) for section_title, section_text in zip(article['section_titles'], article['section_texts']): print("Section title: %s" % section_title) print("Section text: %s" % section_text)
-
👍 Improvements:
- Speedup FastText tests (@horpto, #1686)
- Add optimization for
SlicedCorpus.__len__(@horpto, #1679) - Make
word_vecreturn immutable vector. Fix #1651 (@CLearERR, #1662) - Drop Win x32 support & add rolling builds (@menshikh-iv, #1652)
- Fix scoring function in Phrases. Fix #1533, #1635 (@michaelwsherman, #1573)
- Add configuration for flake8 to setup.cfg (@mcobzarenco, #1636)
- Add
build_vocab_from_freqto Word2Vec, speedup scan_vocab (@jodevak, #1599) - Add
most_similar_to_givenmethod for KeyedVectors (@TheMathMajor, #1582) - Add
__getitem__method to Sparse2Corpus to allow direct queries (@isamaru, #1621)
🔴 Bug fixes:
- Add single core mode to CoherenceModel. Fix #1683 (@horpto, #1685)
- Fix ResourceWarnings in tests. Partially fix #1519 (@horpto, #1660)
- Fix DeprecationWarnings generated by deprecated assertEquals. Partial fix #1519 (@poornagurram, #1658)
- Fix DeprecationWarnings for regex string literals. Fix #1646 (@franklsf95, #1649)
- Fix pagerank algorithm. Fix #805 (@xelez, #1653)
- Fix FastText inconsistent dtype. Fix #1637 (@mcobzarenco, #1638)
- Fix
test_filename_filteringtest (@nehaljwani, #1647)
📚 Tutorial and doc improvements:
- Fix code/docstring style (@menshikh-iv, #1650)
- Update error message for supervised FastText. Fix #1498 (@ElSaico, #1645)
- Add "DOI badge" to README. Fix #1610 (@dphov, #1639)
- Remove duplicate annoy notebook. Fix #1415 (@Karamax, #1640)
- Fix duplication and wrong markup in docs (@horpto, #1633)
- Refactor dendrogram & topic network notebooks (@parulsethi, #1571)
- Fix release badge (@menshikh-iv, #1631)
-
Remove
gensim.examplesgensim.nosygensim.scripts.word2vec_standalonegensim.scripts.make_wiki_lemmagensim.scripts.make_wiki_onlinegensim.scripts.make_wiki_online_lemmagensim.scripts.make_wiki_online_nodebuggensim.scripts.make_wiki
-
Move
gensim.scripts.make_wikicorpus➡gensim.scripts.make_wiki.pygensim.summarization➡gensim.models.summarizationgensim.topic_coherence➡gensim.models._coherencegensim.utils➡gensim.utils.utils(old imports will continue to work)gensim.parsing.*➡gensim.utils.text_utils
Also, we'll create experimental subpackage for unstable models. Specific lists will be available in the next major release.
3.0.1
3.0.1, 2017-10-12
🔴 Bug fixes:
- Fix Keras import, speedup importing time. Fix #1614 (@menshikh-v, #1615)
- Fix Sphinx warnings and retrieve all missing .rst (@anotherbugmaster and @menshikh-iv, #1612)
- Fix logger message in lsi_dispatcher (@lorosanu, #1603)
📚 Tutorial and doc improvements:
-
Remove
gensim.examplesgensim.nosygensim.scripts.word2vec_standalonegensim.scripts.make_wiki_lemmagensim.scripts.make_wiki_onlinegensim.scripts.make_wiki_online_lemmagensim.scripts.make_wiki_online_nodebuggensim.scripts.make_wiki
-
Move
gensim.scripts.make_wikicorpus➡gensim.scripts.make_wiki.pygensim.summarization➡gensim.models.summarizationgensim.topic_coherence➡gensim.models._coherencegensim.utils➡gensim.utils.utils(old imports will continue to work)gensim.parsing.*➡gensim.utils.text_utils
Also, we'll create experimental subpackage for unstable models. Specific lists will be available in the next release.
GSoC storm
3.0.0, 2017-09-27
🌟 New features:
- Add unsupervised FastText to Gensim (@chinmayapancholi13, #1525)
- Add sklearn API for gensim models (@chinmayapancholi13, #1462)
- Add callback metrics for LdaModel and integration with Visdom (@parulsethi, #1399)
- Add TranslationMatrix model (@robotcator, #1434)
- Add word2vec-based coherence. Fix #1380 (@macks22, #1530)
👍 Improvements:
- Add 'diagonal' parameter for LdaModel.diff (@parulsethi, #1448)
- Add 'score' function for SklLdaModel (@chinmayapancholi13, #1445)
- Update sklearn API for gensim models (@chinmayapancholi13, #1473) [:warning: breaks backward compatibility]
- Add CoherenceModel to LdaModel.top_topics. Fix #1128 (@macks22, #1427)
- Add dendrogram viz for topics and JS metric (@parulsethi, #1484)
- Add topic network viz (@parulsethi, #1536)
- Replace viewitems to iteritems. Fix #1495 (@HodorTheCoder, #1508)
- Fix Travis config and add style-checking for Ipython Notebooks. Fix #1518, #1520 (@menshikh-iv, #1522)
- Remove mutable args from definitions. Fix #1561 (@zsef123, #1562)
- Add Appveyour for all PRs. Fix #1565 (@menshikh-iv, #1565)
- Refactor code by PEP8. Partially fix #1521 (@zsef123, #1550)
- Refactor code by PEP8 with additional limitations. Fix #1521 (@menshikh-iv, #1569)
- Update FastTextKeyedVectors.__contains__ (@ELind77, #1499)
- Update WikiCorpus tokenization. Fix #1534 (@roopalgarg, #1537)
🔴 Bug fixes:
- Remove round in LdaSeqModel.print_topic. Fix #1480 (@menshikh-iv, #1547)
- Fix TextCorpus.samle_text (@menshikh-iv, #1548)
- Fix Mallet wrapper and tests for HDPTransform (@menshikh-iv, #1555)
- Fix incorrect initialization ShardedCorpus with a generator. Fix #1511 (@karkkainenk1, #1512)
- Add verification when summarize_corpus returns null. Fix #1531 (@fbarrios, #1570)
- Fix doctag unicode problem. Fix 1543 (@englhardt, #1544)
- Fix Translation Matrix (@robotcator, #1594)
- Add trainable flag to KeyedVectors.get_embedding_layer. Fix #1557 (@zsef123, #1558)
📚 Tutorial and doc improvements:
- Update exception text in TextCorpus.samle_text. Partial fix #308 (@vlejd, #1444)
- Remove extra filter_token from tutorial (@VorontsovIE, #1502)
- Update Doc2Vec-IMDB notebook (@pahdo, #1476)
- Add Google Tag Manager for site (@yardos, #1556)
- Update docstring explaining lack of multistream support in WikiCopus. Fix #1496 (@polm and @menshikh-iv, #1515)
- Fix PathLineSentences docstring (@gojomo)
- Fix typos from Translation Matrix notebook (@robotcator, #1598)
Docker image and integration with Sklearn
2.3.0, 2017-07-25
🌟 New features:
- Add Dockerfile for gensim with external wrappers (@parulsethi, #1368)
- Add sklearn wrapper for Word2Vec (@chinmayapancholi13, #1437)
- Add loss function for Word2Vec. Fix #999 (@chinmayapancholi13, #1201)
- Add sklearn wrapper for AuthorTopic model (@chinmayapancholi13, #1403)
👍 Improvements:
- Remove unittest2 (@souravsingh, #1490)
- Add multiple scoring methods for Phrases. Partial fix #1363 (@michaelwsherman, #1464)
- Add WordRank wrapper to Dockerfile (@parulsethi, #1460)
- Add PathLineSentences. Fix #1364 (@michaelwsherman, #1423)
- Add TextDirectoryCorpus and refactor TextCorpus. Fix #1387 (@macks22, #1459)
- Add sparse input support with topn parameter in any2sparse. Fix #1294 (@manneshiva, #1321)
- Add seed and length for sample_text. Partial fix #308 (@vlejd, #1422)
- Add word_ngram parameter to FastText (@fsonntag, #1432)
🔴 Bug fixes:
- Fix fastText loading from .bin file. Fix #1236 (@prakhar2b, #1341)
- Fix paths in WordRank and running gensim version in Dockerfile (@parulsethi, #1503)
- Fix commit version for gensim in Dockerfile (@parulsethi, #1491)
- Fix encoding problems with tests on windows. Fix #1441 (@menshikh-iv, #1469)
- Fix parameters in score_cbow_pair (@jmhessel, #1468)
- Fix parameters in score_sentence_cbow (@jmhessel, #1467)
- Fix TextDirectoryCorpus on windows (@macks22, #1463)
- Fix gensim version in Dockerfile (@parulsethi, #1456)
- Fix WordOccurenceAccumulator on windows. Fix #1441 (@macks22, #1449)
- Fix scipy/numpy requirements (downgrade). Fix #1450 (@menshikh-iv, #1450)
📚 Tutorial and doc improvements:
- Fix links and spaces in quick start guide (@iamsanten, #1500)
- Fix error of ConcatedDoc2Vec in doc2vec-imdb notebook (@robocator, #1377)
- Fix Sphinx warnings. Fix #1192 (@prerna135, #1442)
- Fix typo in LdaModel.diff method (@parulsethi, #1461)
- Add Tensorboard visualization for LDA (@parulsethi, #1396)
- Update old and add new notebook with CoherenceModel (@macks22, #1431)
Integration with Keras and Sklearn, LdaModel topic difference
2.2.0, 2017-06-21
🌟 New features:
- Add sklearn wrapper for RpModel (@chinmayapancholi13, #1395)
- Add sklearn wrappers for LdaModel and LsiModel (@chinmayapancholi13, #1398)
- Add sklearn wrapper for LdaSeq (@chinmayapancholi13, #1405)
- Add keras wrapper for Word2Vec model (@chinmayapancholi13, #1248)
- Add LdaModel.diff method (@menshikh-iv, #1334)
- Allow use of truncated Dictionary for coherence measures. Fix #1342 (@macks22, #1349)
👍 Improvements:
- Fix save_as_text/load_as_text for Dictionary (@vlejd, #1402)
- Add sampling support for corpus. Fix #308 (@vlejd, #1408)
- Add napoleon extension to sphinx (@rasto2211, #1411)
- Add KeyedVectors support to AnnoyIndexer (@Quole, #1318)
- Add BaseSklearnWrapper (@chinmayapancholi13, #1383)
- Replace num_words to topn in model for unification. Fix #1198 (@prakhar2b, #1200)
- Rename out_path to out_name & add logging for WordRank model. Fix #1310 (@parulsethi, #1332)
- Remove multiple iterations of corpus in p_boolean_document (@danielchamberlain, #1325)
- Fix codestyle in TfIdf (@piskvorky, #1313)
- Fix warnings from Sphinx. Partial fix #1192 (@souravsingh, #1330)
- Add test_env to setup.py (@menshikh-iv, #1336)
🔴 Bug fixes:
- Add cleanup in annoy test (@prakhar2b, #1420)
- Add cleanup in lda backprop test (@prakhar2b, #1417)
- Fix out-of-vocab in FastText (@jayantj, #1409)
- Add cleanup in WordRank test (@parulsethi, #1410)
- Fix rest requirements in Travis. Partial fix #1393 (@ibrahimsharaf, @menshikh-iv, #1400)
- Fix morfessor exception. Partial fix #1324 (@souravsingh, #1406)
- Fix test for FastText (@prakhar2b, #1371)
- Fix WikiCorpus (@alekol, #1333)
- Fix backward incompatibility for LdaModel (@chinmayapancholi13, #1327)
- Fix support for old and new FastText model format. Fix #1301 (@prakhar2b, #1319)
- Fix wrapper tests. Fix #1323 (@shubhamjain74, #1359)
- Update export_phrases method. Fix #794 (@toumorokoshi, #1362)
- Fix sklearn exception in test (@souravsingh, #1350)
📚 Tutorial and doc improvements:
- Fix incorrect link in tutorials (@aneesh-joshi, #1426)
- Add notebook with sklearn wrapper examples (@chinmayapancholi13, #1428)
- Replace absolute pathes to relative in notebooks (@vochicong, #1414)
- Fix code-style in keras notebook (@chinmayapancholi13, #1394)
- Replace absolute pathes to relative in notebooks (@vochicong, #1407)
- Fix typo in quickstart guide (@vochicong, #1404)
- Update docstring for WordRank. Fix #1384 (@parulsethi, #1378)
- Update docstring for SkLdaModel (@chinmayapancholi13, #1382)
- Update logic for updatetype in LdaModel (@chinmayapancholi13, #1389)
- Update docstring for Doc2Vec (@jstol, #1379)
- Fix docstring for KL-distance (@viciousstar, #1373)
- Update Corpora_and_Vector_Spaces tutorial (@charliejharrison, #1308)
- Add visualization for difference between LdaModel (@menshikh-iv, #1374)
- Fix punctuation & typo in changelog (@piskvorky, @menshikh-iv, #1366)
- Fix PEP8 & typo in several PRs (@menshikh-iv, #1369)
- Update docstrings connected with backward compability in for LdaModel (@chinmayapancholi13, #1365)
- Update Corpora_and_Vector_Spaces tutorial (@schuyler1d, #1360)
- Fix typo in Doc2Vec doctsring (@fujiyuu75, #1356)
- Update Annoy tutorial (@pmbaumgartner, #1355)
- Update temp folder in tutorials (@yl2526, #1352)
- Remove spaces after print in Topics_and_Transformation tutorial (@gsimore, #1354)
- Update Dictionary docstring (@oonska, #1347)
- Add section headings in word2vec notebook (@MikeTheReader, #1348)
- Fix broken urls in starter tutorials (@ka7eh, #1346)
- Update quick start notebook (@yardsale8, #1345)
- Fix typo in quick start notebook (@MikeTheReader, #1344)
- Fix docstring in keyedvectors (@chinmayapancholi13, #1337)
Doc2Vec visualisation
2.1.0, 2017-05-12
🌟 New features:
- Add modified save_word2vec_format for Doc2Vec, to save document vectors. (@parulsethi, #1256)
👍 Improvements:
- Add automatic code style check limited only to the code modified in PR (@tmylk, #1287)
- Replace
logger.warnbylogger.warning(@chinmayapancholi13, #1295) - Docs word2vec docstring improvement, deprecation labels (@shubhvachher, #1274)
- Stop passing 'sentences' as parameter to Doc2Vec. Fix #511 (@gogokaradjov, #1306)
🔴 Bug fixes:
- Allow indexing with np.int64 in doc2vec. Fix #1231 (@bogdanteleaga, #1254)
- Update Doc2Vec docstring. Fix #1302 (@datapythonista, #1307)
- Ignore rst and ipynb file in Travis flake8 validations (@datapythonista, #1309)
📚 Tutorial and doc improvements:
- Update Tensorboard Doc2Vec notebook (@parulsethi, #1286)
- Update Doc2Vec IMDB Notebook, replace codesc to smart_open (@robotcator, #1278)
- Add explanation of
sizeto Word2Vec Notebook (@jbcoe, #1305) - Add extra param to WordRank notebook. Fix #1276 (@parulsethi, #1300)
- Update warning message in WordRank (@parulsethi, #1299)