Skip to content

JosephP91/kweb

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

50 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

kweb

kweb is a Python application that exposes Kafka API via websocket. The application is based on Tornado asynchronous web framework and kafka-python library.

Donate

Help me to improve this project!

Donate button

Usage

kweb is very easy to use. You simply create a websocket connection and you can start issuing command to operate with kafka.

Start the service

Assuming you already have the source code, in order to start the service you can do:

cd kweb
python main.py

Now you have the service up and running.

Consumer API

In order to create a Kafka consumer you have to connect via websocket protocol to the following url:

ws://localhost:8000/kweb/consumer/ws/v1
  • Create consumer

In order to create a consumer, issue the following command on the websocket:

{
    "command_name": "create_consumer", 
    "parameters": {
        "bootstrap_servers": ["<kafka-host-1>:9092", "<kafka-host-2:9092"], 
        "group_id": "testing_group", 
        "enable_auto_commit": false
    }
}

Done. Consumer created. If you want to know which options you can specify, you can check here: Kafka Consumer

  • Subscribe

After you created a consumer, you should subscribe some topics. You can do that by issuing the following command:

{
    "command_name": "subscribe", 
    "parameters": {
        "topics": ["TOPIC1", "OTHER-TOPIC", "ANOTHER-TOPIC"]
    }
}

More details on this command here: Subscribe Command

  • Unsubscribe

If you want to unsubscribe the previously subcribed topics, you can issue the following command:

{
    "command_name": "unsubscribe", 
    "parameters": {}
}

More details on this command here: Unsubscribe Command

  • Seek to end

If you want to seek to end of topic-partition, you can issue the following command:

{
    "command_name": "seek_to_end", 
    "parameters": {
        "topic-partitions": [
            {
                "topic": "<topic-name>",
                "partition": 0
            }, 
            {
                "topic": "<another-assigned-topic>",
                "partition": 2
            }
        ]
    }
}

More details on this command here: Seek to End Command

  • Seek to beginning

If you want to seek to beginning of topic-partition, you can issue the following command:

{
    "command_name": "seek_to_beginning", 
    "parameters": {
        "topic-partitions": [
            {
                "topic": "<topic-name>",
                "partition": 0
            },
			{
                "topic": "<another-assigned-topic>",
                "partition": 2
            }
        ]
    }
}

More details on this command here: Seek to Beginning Command

  • Assign

If you want to assign a series of topic-partition to your consumer you can issue the following command:

{
    "command_name": "assign", 
    "parameters": {
        "topic-partitions": [
            {
                "topic": "<topic-name>",
                "partition": 0
            }, 
            {
                "topic": "<another-assigned-topic>",
                "partition": 2
            }
        ]
    }
}

More details on this command here: Assign Command

  • Commit Async

If you want to commit your offset in an asynchronous manner, you can issue the following command:

{
    "command_name": "commit_async", 
    "parameters": {
        "topic-offset-metadata": [
            {
                "key": {
                    "topic": "<topic-name>",
                    "partition": 0
                },
                "value": {
                    "offset": 1,
                    "metadata": "your metadata here"
                }
            }
        ]
    }
}

Note that topic-offset-metadata can also be an empty array. In this way all the offset will be committed. Note also that since this command is asynchronous, a callback will notify you on the same socket when the commit will be received by the cluster.

More details on this command here: Commit Async Command

  • Commit

If you want to commit your offset you can issue the following command:

{
    "command_name": "commit", 
    "parameters": {
        "topic-offset-metadata": [
            {
                "key": {
                    "topic": "<topic-name>",
                    "partition": 0
                },
                "value": {
                    "offset": 1,
                    "metadata": "your metadata here"
                }
            }
        ]
    }
}

Note that topic-offset-metadata can also be an empty array. In this way all the offset will be committed.

More details on this command here: Commit Command

  • Committed

If you want to know the last committed offset for a given topic-partition, you can issue the following command:

{
    "command_name": "committed", 
    "parameters": {
        "topic-partition": {
            "topic": "<topic-name>",
            "partition": 0
        },
        "metadata": false
    }
}

If you don't want only the last committed offset as an integer, you can set metadata to true. In this way more details will be returned.

More details on this command here: Committed

  • Partitions for topic

If you want to know the partitions available for a specific topic (assuming you are authorized to do so), you can issue the following command:

{
    "command_name": "partitions_for_topic", 
    "parameters": {
        "topic": "<topic-name>"
    }
}

More details on this command here: Partitions for Topic Command

  • Position

If you want to know the offset of the next record that will be fetched you can issue the following command:

{
    "command_name": "position", 
    "parameters": {
        "topic-partition": {
            "topic": "<topic-name>",
            "partition": 0
        }
    }
}

More details on this command here: Position Command

  • Highwater

If you want to know the last known highwater for a given topic-partition, you can issue the following command:

{
    "command_name": "highwater", 
    "parameters": {
        "topic-partition": {
            "topic": "<topic-name>",
            "partition": 0
        }
    }
}

If you want to know what's an highwater offset, or you want to have more details on this command, go here: Highwater Command

  • Pause

If you want to suspend fetching from a set of topic-partitions, you can issue the following command:

{
    "command_name": "pause", 
    "parameters": {
        "topic-partition": [
            {
                "topic": "<topic-name>", 
                "partition": 0
            }
        ]
    }
}

More details on this command here: Pause Command

  • Resume

If you want to resume fetching from a set of previously paused topic-partitions, you can issue the following command:

{
    "command_name": "resume", 
    "parameters": {
        "topic-partition": [
            {
                "topic": "<topic-name>", 
                "partition": 0
            }
        ]
    }
}

More details on this command here: Resume Command

  • Seek

If you want to seek on specifically topic-partion, you can issue the following command:

{
    "command_name": "seek", 
    "parameters": {
        "topic-partition": {
            "topic": "<topic-name>", 
            "partition": 0
        },
        "offset": 100
    }
}

More details on this command here: Seek Command

  • Beginning Offsets

If you want to get the initial offsets for the specified set of topic-partitions, you can issue the following command:

{
    "command_name": "beginning_offsets", 
    "parameters": {
        "topic-partition": [
            {
                "topic": "<topic-name>", 
                "partition": 0
            }
        ]
    }
}

More details on this command here: Beginning Offsets Command

  • End Offsets

If you want to get the end offsets for the specified set of topic-partitions, you can issue the following command:

{
    "command_name": "end_offsets", 
    "parameters": {
        "topic-partition": [
            {
                "topic": "<topic-name>", 
                "partition": 0
            }
        ]
    }
}

More details on this command here: End Offsets Command

About

Kafka API exposed over websocket

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published