Apache Pulsar — Development Setup

Part 1 of the series on Apache Pulsar focusing on development setup and testing using Python and Rust.

Apache Pulsar Architecture

Overview

Apache Pulsar is a cloud-native, distributed messaging and streaming platform. In this article, I am going to focus on setting up a local development environment using Docker.

Client libraries for Apache Pulsar are available natively for Java, Go, Python, C++, Node.js, WebSocket, and C#. In this article, let’s try with Python and a 3rd party Rust library.

Pulsar and Pulsar Manager

Let’s get started by setting up Pulsar and Pulsar Manager.

Pulsar Manager is a web-based GUI management and monitoring tool that helps administrators and users manage and monitor tenants, namespaces, topics, subscriptions, brokers, clusters, and so on, and supports dynamic configuration of multiple environments.

docker-compose

Create the following docker-compose.yml file.

docker-compose.yml

Run the following commands to start the containers.

# mkdir -p data
# docker compose up
Pulsar and Pulsar Manager

Password Setup for Pulsar Manager

Run the following commands to set up the password for Pulsar Manager.

Set up the CRSF token,

# CSRF_TOKEN=$(curl http://localhost:7750/pulsar-manager/csrf-token)

Set up the admin user name to apachepulsar.

# curl \
-H "X-XSRF-TOKEN: $CSRF_TOKEN" \
-H "Cookie: XSRF-TOKEN=$CSRF_TOKEN;" \
-H 'Content-Type: application/json' \
-X PUT http://localhost:7750/pulsar-manager/users/superuser \
-d '{"name": "admin", "password": "apachepulsar", "description": "test", "email": "username@test.org"}'

Pulsar Manager UI

Open the URL http://localhost:9527/ from your browser and log in as admin. The password is apachepulsar if you use the above command.

Pulsar Manager Login Screen

Environment Setup

An environment represents a Pulsar instance or a group of clusters you want to manage. A Pulsar Manager is capable of managing multiple environments.

Create Pulsar Environment
  • Click “New Environment” button to add an environment.
  • Input the “Environment Name”. The environment name is used for identifying an environment.
  • Input the “Service URL”. The Service URL is the admin service URL of your Pulsar cluster. You need to make sure the service URL that Pulsar Manager is able to access. In this example, both Pulsar container and Pulsar Manager container are linked. So you can use Pulsar container name as the domain name of the Pulsar standalone cluster. Thus you can type http://pulsar:8080.
Pulsar Manager — Management Screen

Testing with Python

Let’s start by using the Python client library to perform testing.

$ pip install pulsar-client

Producing and Consuming Messages

This is a simple consumer code taken from the website. Run this code snippet and it shall loop and wait for messages from “my-topic”.

consumer.py

import pulsarclient = pulsar.Client('pulsar://localhost:6650')consumer = client.subscribe('my-topic', 'my-subscription')while True:
msg = consumer.receive()
try:
print("Received message '{}' id='{}'".format(msg.data(),
msg.message_id()))
# Acknowledge successful processing of the message
consumer.acknowledge(msg)
except:
# Message failed to be processed
consumer.negative_acknowledge(msg)
client.close()

Open another terminal and use the following code snippet to produce messages.

producer.py

import pulsarclient = pulsar.Client('pulsar://localhost:6650')producer = client.create_producer('my-topic')for i in range(10):
producer.send(('Hello-%d' % i).encode('utf-8'))
client.close()
Producer-Consumer Example

From Pulsar Manager, you can view the tenants, namespaces, topics, subscriptions, and more.

Pulsar Manager UI

Testing with CLI

For details on the CLI, refer to the documentation.

E.g. to consume messages from a topic,

$ pulsar-client consume my-topic -s my-subscription

To produce messages,

$ pulsar-client produce my-topic -m "hello pulsar"
Pulsar CLI

Testing with Rust

Now let’s use a Rust client library to produce and consume messages.

  • Install Rust.
  • Use Cargo to create a new project.
$ cargo new rust-client --bin
  • Add the dependencies to cargo.toml
[dependencies]
futures = "0.3"
pulsar = "2.0"
tokio = { version = "1.0", features = ["full"] }
serde = { version = "1.0", features = ["derive"] }
env_logger = "0.8.3"
serde_json = "1.0.64"
log = "0.4.14"
  • Refactor the project to have the consumer and producer separately by making these changes in cargo.toml
[[bin]]
name = "consumer"
path = "src/consumer/main.rs"
[[bin]]
name = "producer"
path = "src/producer/main.rs"

Test out with the producer and consumer sample code.

Producer
Consumer

For the producer, run. cargo run -bin producer

Rust Producer

For the consumer, run cargo run -bin consumer

Rust Consumer

Summary

At this stage, we have a basic development setup of Apache Pulsar. In future articles, we shall go deeper to explore more advanced features.

The source code for this article can be found in this repository.

You may also want to check out the following articles!

Programmer and occasional blogger.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store