Scalable, fast, and lightweight system for large-scale topic modeling
Перейти к файлу
Fei Gao de9cca422c
Update README.md
2017-12-18 20:34:33 +08:00
example recovery example/nytimes.sh 2016-01-06 09:26:17 +08:00
inference Merge branch 'master' of https://github.com/Microsoft/lightlda into Microsoft-master 2016-01-06 09:15:14 +08:00
multiverso@9ed99cd2d3 update submodule 2016-04-14 19:17:11 +08:00
preprocess first commit 2015-09-12 14:45:19 +08:00
src Implement the windows part code for inference 2016-09-24 23:55:02 +08:00
windows add infer vs project file 2016-01-05 12:05:45 +08:00
.gitignore add submodule 2016-04-14 03:56:15 -07:00
.gitmodules update submodule 2016-04-14 19:17:11 +08:00
LICENSE first commit 2015-09-12 14:45:19 +08:00
Makefile Update Makefile 2016-01-04 21:19:57 +08:00
MakefileDocker fix the bug in MakefileDocker PROJECT := 2016-06-21 05:10:09 +08:00
README.md Update README.md 2017-12-18 20:34:33 +08:00
build.sh modify build script 2016-04-14 17:07:20 +08:00

README.md

LightLDA

LightLDA is a distributed system for large scale topic modeling. It implements a distributed sampler that enables very large data sizes and models. LightLDA improves sampling throughput and convergence speed via a fast O(1) metropolis-Hastings algorithm, and allows small cluster to tackle very large data and model sizes through model scheduling and data parallelism architecture. LightLDA is implemented with C++ for performance consideration.

We have sucessfully trained big topic models (with trillions of parameters) on big data (Top 10% PageRank values of Bing indexed page, containing billions of documents) in Microsoft. For more technical details, please refer to our WWW'15 paper.

For documents, please view our website http://www.dmtk.io.

Why LightLDA

The highlight features of LightLDA are

  • Scalable: LightLDA can train models with trillions of parameters on big data with billions of documents, a scale previous implementations cann't handle.
  • Fast: The sampler can sample millions of tokens per second per multi-core node.
  • Lightweight: Such big tasks can be trained with as few as tens of machines.

Quick Start

Run $ sh build.sh to build lightlda. Run $ sh example/nytimes.sh for a simple example.

Reference

Please cite LightLDA if it helps in your research:

@inproceedings{yuan2015lightlda,
  title={LightLDA: Big Topic Models on Modest Computer Clusters},
  author={Yuan, Jinhui and Gao, Fei and Ho, Qirong and Dai, Wei and Wei, Jinliang and Zheng, Xun and Xing, Eric Po and Liu, Tie-Yan and Ma, Wei-Ying},
  booktitle={Proceedings of the 24th International Conference on World Wide Web},
  pages={1351--1361},
  year={2015},
  organization={International World Wide Web Conferences Steering Committee}
}

Microsoft Open Source Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.