Release Notes - Kafka - Version 3.1.0

Below is a summary of the JIRA issues addressed in the 3.1.0 release of Kafka. For full documentation of the release, a guide to get started, and information about the project, see the Kafka project site.

Note about upgrades: Please carefully review the upgrade documentation for this release thoroughly before upgrading your cluster. The upgrade notes discuss any critical information about incompatibilities and breaking changes, performance changes, and any other changes that might impact your production deployment of Kafka.

The documentation for the most recent release can be found at https://kafka.apache.org/documentation.html.

New Feature

[KAFKA-13202] - KIP-768: Extend SASL/OAUTHBEARER with Support for OIDC

Improvement

[KAFKA-4064] - Add support for infinite endpoints for range queries in Kafka Streams KV stores
[KAFKA-8522] - Tombstones can survive forever
[KAFKA-8613] - Make Grace Period Mandatory for Windowed Operations in Streams
[KAFKA-10712] - Update release scripts to Python3
[KAFKA-10777] - Add additional configuration to control MirrorMaker 2 internal topics naming convention
[KAFKA-10847] - Avoid spurious left/outer join results in stream-stream join
[KAFKA-12486] - Utilize HighAvailabilityTaskAssignor to avoid downtime on corrupted task
[KAFKA-12766] - Consider Disabling WAL-related Options in RocksDB
[KAFKA-12840] - Removing `compact` cleaning on a topic should abort on-going compactions
[KAFKA-12882] - Add ActiveBrokerCount and FencedBrokerCount metrics (KIP-748)
[KAFKA-12887] - Do not trigger user-customized ExceptionalHandler for RTE
[KAFKA-12909] - Allow users to opt-into spurious left/outer stream-stream join improvement
[KAFKA-12963] - Improve error message for Class cast exception
[KAFKA-12994] - Migrate all Tests to New API and Remove Suppression for Deprecation Warnings related to KIP-633
[KAFKA-13071] - Deprecate and remove --authorizer option in kafka-acls.sh
[KAFKA-13075] - Consolidate RocksDBStoreTest and RocksDBKeyValueStoreTest
[KAFKA-13103] - Should group admin handlers consider REBALANCE_IN_PROGRESS and GROUP_AUTHORIZATION_FAILED as retryable errors?
[KAFKA-13111] - Re-evaluate Fetch Sessions when using topic IDs
[KAFKA-13134] - Heartbeat Request high lock contention
[KAFKA-13229] - KIP-761: implement a total blocked time metric in Kafka Streams
[KAFKA-13243] - Differentiate metric latency measured in millis and nanos
[KAFKA-13273] - Add support for Java 17
[KAFKA-13284] - Use sftp protocol in release.py to upload release candidate artifacts
[KAFKA-13301] - The relationship between request.timeout. ms and max.poll.interval.ms in the Consumer Configs is incorrect.
[KAFKA-13319] - Do not send AddOffsetsToTxn/TxnOffsetCommit if offsets map is empty
[KAFKA-13345] - WindowStore should use "delete" policy when duplicates are enabled
[KAFKA-13377] - Fix Resocue leak due to Files.list
[KAFKA-13381] - Wrap all uncaught exceptions as StreamsException with TaskId field
[KAFKA-13439] - Deprecate EAGER rebalancing in Kafka Streams
[KAFKA-13444] - Fix OAuthCompatibilityTool help and add SSL options
[KAFKA-13445] - Add ECDSA test for JWT validation

Bug

[KAFKA-9747] - No tasks created for a connector
[KAFKA-9887] - failed-task-count JMX metric not updated if task fails during startup
[KAFKA-12226] - High-throughput source tasks fail to commit offsets
[KAFKA-12257] - Consumer mishandles topics deleted and recreated with the same name
[KAFKA-12487] - Sink connectors do not work with the cooperative consumer rebalance protocol
[KAFKA-12762] - Use connection timeout when polling the network for new connections
[KAFKA-13017] - Excessive logging on sink task deserialization errors
[KAFKA-13049] - Log recovery threads use default thread pool naming
[KAFKA-13079] - Forgotten Topics in Fetch Requests may incorrectly use topic IDs
[KAFKA-13102] - Topic IDs not propagated to metadata cache quickly enough for Fetch path
[KAFKA-13122] - resource leak due to not close KeyValueIterator implemented instances
[KAFKA-13126] - Overflow in joinGroupTimeoutMs when max.poll.interval.ms is MAX_VALUE leads to missing rebalances
[KAFKA-13162] - ElectLeader API must be forwarded to Controller
[KAFKA-13168] - KRaft observers should not have a replica id
[KAFKA-13170] - Flaky Test InternalTopicManagerTest.shouldRetryDeleteTopicWhenTopicUnknown
[KAFKA-13175] - The topic is marked for deletion, create topic with the same name throw exception topic already exists.
[KAFKA-13194] - LogCleaner may clean past highwatermark
[KAFKA-13204] - wrong assignor selected if the assignor name is identical
[KAFKA-13210] - fetch/findSessions queries with open endpoints for SessionStore/WindowStore
[KAFKA-13216] - Streams left/outer joins cause new internal changelog topic to grow unbounded
[KAFKA-13225] - Controller skips sending UpdateMetadataRequest when shutting down broker doesnt host partitions
[KAFKA-13231] - `TransactionalMessageCopier.start_node` should wait until the process if fully started
[KAFKA-13256] - Possible NPE in ConfigDef when rendering (enriched) RST or HTML when documentation is not set/NULL
[KAFKA-13261] - KTable to KTable foreign key join loose events when using several partitions
[KAFKA-13264] - backwardFetch in InMemoryWindowStore doesn't return in reverse order
[KAFKA-13294] - Upgrade Netty to 4.1.68 for CVE fixes
[KAFKA-13305] - NullPointerException in LogCleanerManager "uncleanable-bytes" gauge
[KAFKA-13309] - InMemorySessionStore#fetch/backwardFetch doesn't return in correct order
[KAFKA-13315] - log layer exception during shutdown that caused an unclean shutdown
[KAFKA-13370] - Offset commit failure percentage metric is not computed correctly (regression)
[KAFKA-13391] - Failure on Windows due to AccessDeniedAcception when attempting to fsync the parent directory
[KAFKA-13394] - Topic IDs should be removed from PartitionFetchState if they are no longer sent by the controller
[KAFKA-13396] - kafka-topics.sh --create should not require the partitions/replication-factor argument
[KAFKA-13397] - MirrorMaker2 shouldn't mirror Connect's internal topics
[KAFKA-13406] - Cooperative sticky assignor got stuck due to assignment validation failed
[KAFKA-13430] - Remove broker-wide quota properties from the documentation
[KAFKA-13443] - Kafka broker exits when OAuth enabled and certain configuration not specified
[KAFKA-13446] - Remove JWT access token from logs
[KAFKA-13456] - Tighten KRaft config checks/constraints
[KAFKA-13461] - KafkaController stops functioning as active controller after ZooKeeperClient auth failure
[KAFKA-13469] - End-of-life offset commit for source task can take place before all records are flushed
[KAFKA-13472] - Connect can lose track of last committed offsets for topic partitions after partial consumer revocation
[KAFKA-13488] - Producer fails to recover if topic gets deleted (and gets auto-created)
[KAFKA-13490] - Fix some problems with KRaft createTopics and incrementalAlterConfigs
[KAFKA-13512] - topicIdsToNames and topicNamesToIds allocate unnecessary maps
[KAFKA-13544] - Deadlock during shutting down kafka broker because of connectivity problem with zookeeper
[KAFKA-13582] - `TestVerifiableProducer. test_multiple_kraft_security_protocols` consistently fails

Task

[KAFKA-13287] - Upgrade RocksDB to 6.22.1.1

Test

[KAFKA-12935] - Flaky Test RestoreIntegrationTest.shouldRecycleStateFromStandbyTaskPromotedToActiveTaskAndNotRestore
[KAFKA-13046] - Improve the test coverage for stickyAssignor

Sub-task

[KAFKA-9555] - Topic-based implementation for the RemoteLogMetadataManager
[KAFKA-10580] - Add topic ID support to Fetch request
[KAFKA-10774] - Support Describe topic using topic IDs
[KAFKA-12554] - Split Log layer into Log and LocalLog
[KAFKA-12802] - Add a file based cache for consumed remote log metadata for each partition to avoid consuming again incase of broker restarts.
[KAFKA-12988] - Change RLMM add/updateRemoteLogSegmentMetadata and putRemotePartitionDeleteMetadata APIS asynchronous.
[KAFKA-13066] - Replace EasyMock with Mockito for FileStreamSinkConnectorTest
[KAFKA-13068] - Rename Log to UnifiedLog
[KAFKA-13082] - Replace EasyMock with Mockito for ProcessorContextTest
[KAFKA-13088] - Replace EasyMock with Mockito for ForwardingDisabledProcessorContextTest
[KAFKA-13101] - Replace EasyMock and PowerMock with Mockito for RestServerTest
[KAFKA-13123] - Close KeyValueIterator implemented instance in example codes and some tests
[KAFKA-13124] - Close KeyValueIterator implemented instance in internal tests
[KAFKA-13125] - Close KeyValueIterator implemented instance in internal tests (cont.)
[KAFKA-13211] - fetch queries with open endpoints for WindowStore
[KAFKA-13212] - fetch/findSessions queries with open endpoints for SessionStore
[KAFKA-13237] - Add ActiveBrokerCount and FencedBrokerCount metrics to the ZK controller (KIP-748)
[KAFKA-13238] - Add ActiveBrokerCount and FencedBrokerCount metrics to the KRaft controller (KIP-748)