Mirror of Apache Kafka

Перейти к файлу

Neha Narkhede e27136d81a KAFKA-141 Follow up patch to fix the cpp files; patched by Lorenzo, nehanarkhede; reviewed by junrao, jjkoshy git-svn-id: https://svn.apache.org/repos/asf/incubator/kafka/trunk@1182158 13f79535-47bb-0310-9956-ffa450edef68		2011-10-12 01:05:28 +00:00
bin	KAFKA-141 Check and make sure that the files that have been donated have been updated to reflect the new ASF copyright;patched by nehanarkhede; reviewd by jjkoshy	2011-10-07 19:51:28 +00:00
clients	KAFKA-141 Follow up patch to fix the cpp files; patched by Lorenzo, nehanarkhede; reviewed by junrao, jjkoshy	2011-10-12 01:05:28 +00:00
config	KAFKA-141 Check and make sure that the files that have been donated have been updated to reflect the new ASF copyright;patched by nehanarkhede; reviewd by jjkoshy	2011-10-07 19:51:28 +00:00
contrib	KAFKA-143 Check and make sure that all source code distributed by the project is covered by one or more approved licenses; patched by nehanarkhede; reviewed by junrao	2011-10-11 19:00:28 +00:00
core	ZK consumer may lose a chunk worth of message during rebalance; patched by Jun Rao; reviewed by Neha Narkhede; KAFKA-154	2011-10-11 19:11:38 +00:00
examples	KAFKA-141 Check and make sure that the files that have been donated have been updated to reflect the new ASF copyright;patched by nehanarkhede; reviewd by jjkoshy	2011-10-07 19:51:28 +00:00
lib	reverting r1178669 to fix KAFKA-147 kafka integration tests fail on a fresh checkout; patched by nehanarkhede; reviewed by junrao	2011-10-06 01:57:38 +00:00
project	KAFKA-141 Check and make sure that the files that have been donated have been updated to reflect the new ASF copyright;patched by nehanarkhede; reviewd by jjkoshy	2011-10-07 19:51:28 +00:00
system_test	KAFKA-141 Check and make sure that the files that have been donated have been updated to reflect the new ASF copyright;patched by nehanarkhede; reviewd by jjkoshy	2011-10-07 19:51:28 +00:00
.gitignore	Restore .gitignore	2011-10-10 14:18:40 +00:00
KEYS	Updating the KEYS file with the correct public keys	2011-10-07 00:54:13 +00:00
LICENSE	KAFKA-93 \| add license to missed files and remove LinkedIn copyright line per ASF guideline. Thanks to Joel Koshy for pointing it out	2011-08-10 18:32:09 +00:00
NOTICE	Initial checkin of Kafka to Apache SVN. This corresponds to `709afe4ec7` except that git specific files have been removed and code has been put into trunk/branches/site/etc. This is just a copy of master, branches and history are not being converted since we can't find a good tool for it.	2011-08-01 23:41:24 +00:00
README.md	Update readme links.	2011-08-10 13:14:28 +00:00
sbt	reverting r1178669 to fix KAFKA-147 kafka integration tests fail on a fresh checkout; patched by nehanarkhede; reviewed by junrao	2011-10-06 01:57:38 +00:00

README.md

Kafka is a distributed publish/subscribe messaging system

It is designed to support the following

Persistent messaging with O(1) disk structures that provide constant time performance even with many TB of stored messages.
High-throughput: even with very modest hardware Kafka can support hundreds of thousands of messages per second.
Explicit support for partitioning messages over Kafka servers and distributing consumption over a cluster of consumer machines while maintaining per-partition ordering semantics.
Support for parallel data load into Hadoop.

Kafka is aimed at providing a publish-subscribe solution that can handle all activity stream data and processing on a consumer-scale web site. This kind of activity (page views, searches, and other user actions) are a key ingredient in many of the social feature on the modern web. This data is typically handled by "logging" and ad hoc log aggregation solutions due to the throughput requirements. This kind of ad hoc solution is a viable solution to providing logging data to an offline analysis system like Hadoop, but is very limiting for building real-time processing. Kafka aims to unify offline and online processing by providing a mechanism for parallel load into Hadoop as well as the ability to partition real-time consumption over a cluster of machines.

See our web site for more details on the project.

Contribution

Kafka is a new project, and we are interested in building the community; we would welcome any thoughts or patches. You can reach us on the Apache mailing lists.

The Kafka code is available from svn or a read only git mirror:

svn co http://svn.apache.org/repos/asf/incubator/kafka/trunk kafka
git clone git://git.apache.org/kafka.git

To build:

./sbt
update - This downloads all the dependencies for all sub projects
package - This will compile all sub projects and creates all the jars

Here are some useful sbt commands, to be executed at the sbt command prompt (./sbt) -

actions : Lists all the sbt commands and their descriptions

clean : Deletes all generated files (the target directory).

clean-cache : Deletes the cache of artifacts downloaded for automatically managed dependencies.

clean-lib : Deletes the managed library directory.

compile : Compile all the sub projects, but not create the jars

test : Run all unit tests in all sub projects

release-zip : Create all the jars, run unit tests and create a deployable release zip

package-all: Creates jars for src, test, docs etc

projects : List all the sub projects

project sub_project_name : Switch to a particular sub-project. For example, to switch to the core kafka code, use "project core-kafka"

Following commands can be run only on a particular sub project -

test-only package.test.TestName : Runs only the specified test in the current sub project

run : Provides options to run any of the classes that have a main method. For example, you can switch to project java-examples, and run the examples there by executing "project java-examples" followed by "run"