Based on blog post, with the code here

Ray also provides a powerfull streaming library that simplifies building of streaming applications

We first show how to use Kafka with Python. Start from the installation of required libraries:

pip3 install -r /Users/boris/Projects/RayStreaming/install/requirements.txt

Our example uses JSON message to communicate between Kafka producers and consumers. The actual implementation is based on producer and consumer

Install Kafka locally (for Mac)

Start zookeeper and kafka

brew services start zookeeper
brew services start kafka

To create a topic with multiple partitions run kafka_management Kafka consumer is here and producer is here

Once done, stop kafka and zookeeper services:

brew services stop kafka
brew services stop zookeeper

We are using Ray 2.0. Install is done using:

pip3 install https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-2.0.0.dev0-cp37-cp37m-macosx_10_15_intel.whl

The image that we will use is:

docker pull rayproject/ray:880797-py37

Following Ray blog post both Kafka consumer and producer are implemented as Ray Actors

The code demonstrate how to integrate Ray with Kafka Producer/Consumer

See also this blog post on how to run it on the cluster and scale it up and down. This is done by Ray autoscaler based on the CPU/memory requirements We still run the fix number of listeners, but autoscaler will decide on the amount of nodes to deploy them on. So unlike K8 native autoscalers, for example, keda, which is trying to manage the amount of Kafka consumers based on the queue depth, A Ray approach is different. Instead of trying to add/remove kafka instances Ray based application starts the amount of Kafka listeners (actors) and manages the amount of Ray nodes, moving actors (transparently) to a new node, as the load on the actors increases.

Based on the example here