Seamless Transition: Mastering Kafka Migration from 2.5.0 to 3.6.2

Introduction:
In the ever-evolving landscape of data processing, keeping up with the latest advancements is crucial for maintaining system efficiency and reliability. Kafka, being a pivotal component in many data architectures, demands careful attention when undergoing upgrades to ensure seamless transitions. In this article, we delve into the intricacies of upgrading Kafka, outlining key steps, potential challenges, and strategies for a successful upgrade process.

Understanding the Landscape: Test Scenarios and Probabilities

Before diving into the upgrade process, it’s imperative to anticipate potential scenarios and prepare accordingly. Here’s a rundown of the crucial aspects to consider:

Test Scenarios:

1. Initial Compatibility Testing: Before proceeding with the upgrade, thorough compatibility testing is essential to identify any potential conflicts or issues.
2. Broker Performance under Load: Post-upgrade, evaluating the performance of brokers under varying loads is critical to ensure optimal system functioning.
3. Consumer and Producer Compatibility: Compatibility between consumers, producers, and the upgraded Kafka version must be thoroughly assessed to prevent disruptions in data flow.
4. Rollback Procedures: Despite meticulous planning, unforeseen complications may arise. Establishing robust rollback procedures is imperative to mitigate risks and minimize downtime.

Possible Issues:

1. Downtime due to Misconfiguration: Misconfigurations during the upgrade process can lead to unexpected downtime, impacting system availability.
2. Client Incompatibility: Incompatibility issues between old and new clients could disrupt data transmission, requiring careful validation and adjustments.
3. Data Loss: The upgrade process itself poses the risk of data loss if not executed meticulously, highlighting the importance of thorough validation and backup measures.

Navigating the Upgrade: Key Steps

1. Initial Configuration Update: Update the server.properties file on all brokers, setting the inter.broker.protocol.version initially to 2.5.0 (As this was our current kafka version)to ensure backward compatibility with older clients.
   ex: inter.broker.protocol.version=2.5.0
2. Code Update for Kafka 3.4.1: Apply necessary code updates, utilizing scripts or automation tools, to ensure compatibility with Kafka version 3.4.1(As first steps was to upgrade till 3.4.1 and then 3.6).
   Ref: https://kafka.apache.org/36/documentation/streams/upgrade-guide
3. Broker Update: Implement the update across all existing brokers within the cluster to maintain uniformity and consistency.
4. Verification: Thoroughly verify the Kafka version on all brokers, meticulously monitoring their behavior for any anomalies or performance deviations.
5. Final Configuration Update: Once confident in the stability and performance post-upgrade, update the server.properties file to reflect the new version, adjusting the inter.broker.protocol.version accordingly.
   ex: inter.broker.protocol.version=3.4.1

Repeat the same steps for 3.4.1 to 3.6.2

A Brief of Infrastructure:

In addition to successfully upgrading Kafka from version 2.5.0 to 3.6 following the outlined steps, it’s noteworthy to highlight the scale and complexity of the environment in which this upgrade was executed.

1. Scale and Complexity of the Environment:
   Our infrastructure comprised four distinct clusters, each varying in size and serving diverse workloads. With over 2000 topics and more than 90,000 partitions distributed across these clusters, the magnitude of the upgrade undertaking was substantial. Despite the intricate topology and vast dataset, our approach remained steadfast and methodical.
2. Execution of the Upgrade Process:
   Drawing from the official Kafka documentation, we meticulously followed the prescribed steps, ensuring adherence to best practices and leveraging the inherent resilience of the Kafka ecosystem. The seamless execution of the upgrade process can be attributed to our commitment to simplicity and reliance on established procedures.
   Guided by the Kafka documentation, particularly the section on upgrading ZooKeeper-based clusters, we adopted a rolling upgrade approach to minimize disruptions and maintain continuous service availability. By diligently updating server properties, upgrading brokers incrementally, and verifying cluster behavior at each stage, we safeguarded against potential pitfalls and upheld system integrity.
3. Success Without Downtime:
   A notable achievement of this endeavor was the absence of downtime throughout the entire upgrade process. By adopting a systematic approach and leveraging the resilience of Kafka’s architecture, we ensured uninterrupted data flow and sustained operational efficiency. This accomplishment underscores our commitment to delivering seamless experiences to our users and stakeholders.

Steps to follow:

Now, let’s outline the pivotal steps involved in executing a Kafka upgrade with precision:

For a rolling upgrade

1. Update server.properties on all brokers and add the following properties. CURRENT_KAFKA_VERSION refers to the version you are upgrading from. CURRENT_MESSAGE_FORMAT_VERSION refers to the message format version currently in use. If you have previously overridden the message format version, you should keep its current value. Alternatively, if you are upgrading from a version prior to 0.11.0.x, then CURRENT_MESSAGE_FORMAT_VERSION should be set to match CURRENT_KAFKA_VERSION.If you are upgrading from version 0.11.0.x or above, and you have not overridden the message format, then you only need to override the inter-broker protocol version.
   - inter.broker.protocol.version=CURRENT_KAFKA_VERSION (e.g. 3.5, 3.4, etc.)
   - log.message.format.version=CURRENT_MESSAGE_FORMAT_VERSION
   (See potential performance impact following the upgrade for the details on what this configuration does.)
   If you haven’t made any changes to this default configuration, you can avoid this config i.e log.message.format.version
2. Upgrade the brokers one at a time: shut down the broker, update the code, and restart it. Once you have done so, the brokers will be running the latest version and you can verify that the cluster’s behavior and performance meets expectations. It is still possible to downgrade at this point if there are any problems.
   I used the shell script to execute this step given below.
3. Once the cluster’s behavior and performance has been verified, bump the protocol version by editing inter.broker.protocol.version and setting it to 3.4 or 3.6 [based on your version].
4. Restart the brokers one by one for the new protocol version to take effect. Once the brokers begin using the latest protocol version, it will no longer be possible to downgrade the cluster to an older version.
5. If you have overridden the message format version as instructed above, then you need to do one more rolling restart to upgrade it to its latest version. Once all (or most) consumers have been upgraded to 0.11.0 or later, change log.message.format.version to 3.6 on each broker and restart them one by one. Note that the older Scala clients, which are no longer maintained, do not support the message format introduced in 0.11, so to avoid conversion costs (or to take advantage of exactly once semantics), the newer Java clients must be used.

#!/bin/bash

NEW_KAFKA_VERSION="3.4.1"
# Download the specified Kafka version
wget https://archive.apache.org/dist/kafka/3.6.2/kafka_2.13-$NEW_KAFKA_VERSION.tgz

# Extract the downloaded archive
tar -xzf kafka_2.13-3.4.1.tgz

# Create a backup directory for Kafka
mkdir -p /etc/kafka_bkup

# Backup the current Kafka installation
cp -r /etc/kafka/* /etc/kafka_bkup/

# Stop the Kafka service
systemctl stop kafka

# Move the new version's files to the Kafka directory
mv kafka_2.13-$NEW_KAFKA_VERSION/bin /etc/kafka/
mv kafka_2.13-$NEW_KAFKA_VERSION/config /etc/kafka/
mv kafka_2.13-$NEW_KAFKA_VERSION/libs /etc/kafka/
mv kafka_2.13-$NEW_KAFKA_VERSION/NOTICE /etc/kafka/
mv kafka_2.13-$NEW_KAFKA_VERSION/site-docs /etc/kafka/
mv kafka_2.13-$NEW_KAFKA_VERSION/LICENSE /etc/kafka/
mv kafka_2.13-$NEW_KAFKA_VERSION/licenses /etc/kafka/

# Restart the Kafka service
systemctl start kafka

# Print the latest Kafka version
echo "Kafka has been upgraded to version:"
/etc/kafka/bin/kafka-topics.sh --version

Apache Kafka

Conclusion:

Upgrading Kafka is a critical endeavor that demands meticulous planning, thorough testing, and adept execution. By adhering to the outlined steps and being cognizant of potential challenges, organizations can navigate the upgrade process with confidence, ensuring uninterrupted data flow and system resilience in the face of evolving requirements.

In conclusion, embracing Kafka upgrades not only enhances system capabilities but also fortifies organizational agility in adapting to the ever-changing data landscape. With careful planning and diligent execution, the journey towards Kafka upgrade success is well within reach.

Keep Learning… Keep Sharing…

https://www.linkedin.com/in/rahulkumar25/