Skip to content

KAFKA-18004: Use 3.8 to run zk service for e2e #17790

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Nov 13, 2024

Conversation

FrankYang0529
Copy link
Member

We plan to remove all ZooKeeper-related code in version 4.0. However, some old brokers in the end-to-end tests still require ZooKeeper service, so we need to run the ZooKeeper service using the 3.x release instead of the dev branch.

Since version 3.9 is not available in the https://s3-us-west-2.amazonaws.com/kafka-packages repo, we can use version 3.8 for now.

Test with #17768. Failed case with version v2.1.1, v.2.2.2, and v2.3.1 will be fixed in https://issues.apache.org/jira/browse/KAFKA-17888.

> git fetch upstream pull/17768/head:PR-17768
> git merge PR-17768
> TC_PATHS="tests/kafkatest/tests/client/client_compatibility_produce_consume_test.py" /bin/bash tests/docker/run_tests.sh

================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.12.0
session_id:       2024-11-12--007
run time:         37 minutes 31.925 seconds
tests run:        18
passed:           15
flaky:            0
failed:           3
ignored:          0
================================================================================
test_id:    kafkatest.tests.client.client_compatibility_produce_consume_test.ClientCompatibilityProduceConsumeTest.test_produce_consume.broker_version=2.1.1
status:     FAIL
run time:   1 minute 26.081 seconds


    TimeoutError("Kafka server didn't finish startup in 60 seconds")
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py", line 351, in _do_run
    data = self.run_test()
  File "/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py", line 411, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.7/dist-packages/ducktape/mark/_mark.py", line 438, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/opt/kafka-dev/tests/kafkatest/tests/client/client_compatibility_produce_consume_test.py", line 89, in test_produce_consume
    self.kafka.start()
  File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 659, in start
    self.wait_for_start(node, monitor, timeout_sec)
  File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 909, in wait_for_start
    err_msg="Kafka server didn't finish startup in %d seconds" % timeout_sec)
  File "/usr/local/lib/python3.7/dist-packages/ducktape/cluster/remoteaccount.py", line 754, in wait_until
    allow_fail=True) == 0, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/ducktape/utils/util.py", line 58, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: Kafka server didn't finish startup in 60 seconds

--------------------------------------------------------------------------------
test_id:    kafkatest.tests.client.client_compatibility_produce_consume_test.ClientCompatibilityProduceConsumeTest.test_produce_consume.broker_version=2.2.2
status:     FAIL
run time:   1 minute 25.996 seconds


    TimeoutError("Kafka server didn't finish startup in 60 seconds")
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py", line 351, in _do_run
    data = self.run_test()
  File "/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py", line 411, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.7/dist-packages/ducktape/mark/_mark.py", line 438, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/opt/kafka-dev/tests/kafkatest/tests/client/client_compatibility_produce_consume_test.py", line 89, in test_produce_consume
    self.kafka.start()
  File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 659, in start
    self.wait_for_start(node, monitor, timeout_sec)
  File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 909, in wait_for_start
    err_msg="Kafka server didn't finish startup in %d seconds" % timeout_sec)
  File "/usr/local/lib/python3.7/dist-packages/ducktape/cluster/remoteaccount.py", line 754, in wait_until
    allow_fail=True) == 0, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/ducktape/utils/util.py", line 58, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: Kafka server didn't finish startup in 60 seconds

--------------------------------------------------------------------------------
test_id:    kafkatest.tests.client.client_compatibility_produce_consume_test.ClientCompatibilityProduceConsumeTest.test_produce_consume.broker_version=2.3.1
status:     FAIL
run time:   1 minute 25.947 seconds


    TimeoutError("Kafka server didn't finish startup in 60 seconds")
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py", line 351, in _do_run
    data = self.run_test()
  File "/usr/local/lib/python3.7/dist-packages/ducktape/tests/runner_client.py", line 411, in run_test
    return self.test_context.function(self.test)
  File "/usr/local/lib/python3.7/dist-packages/ducktape/mark/_mark.py", line 438, in wrapper
    return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
  File "/opt/kafka-dev/tests/kafkatest/tests/client/client_compatibility_produce_consume_test.py", line 89, in test_produce_consume
    self.kafka.start()
  File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 659, in start
    self.wait_for_start(node, monitor, timeout_sec)
  File "/opt/kafka-dev/tests/kafkatest/services/kafka/kafka.py", line 909, in wait_for_start
    err_msg="Kafka server didn't finish startup in %d seconds" % timeout_sec)
  File "/usr/local/lib/python3.7/dist-packages/ducktape/cluster/remoteaccount.py", line 754, in wait_until
    allow_fail=True) == 0, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/ducktape/utils/util.py", line 58, in wait_until
    raise TimeoutError(err_msg() if callable(err_msg) else err_msg) from last_exception
ducktape.errors.TimeoutError: Kafka server didn't finish startup in 60 seconds

--------------------------------------------------------------------------------
test_id:    kafkatest.tests.client.client_compatibility_produce_consume_test.ClientCompatibilityProduceConsumeTest.test_produce_consume.broker_version=2.4.1
status:     PASS
run time:   2 minutes 11.468 seconds
--------------------------------------------------------------------------------
test_id:    kafkatest.tests.client.client_compatibility_produce_consume_test.ClientCompatibilityProduceConsumeTest.test_produce_consume.broker_version=2.5.1
status:     PASS
run time:   2 minutes 9.542 seconds
--------------------------------------------------------------------------------
test_id:    kafkatest.tests.client.client_compatibility_produce_consume_test.ClientCompatibilityProduceConsumeTest.test_produce_consume.broker_version=2.6.3
status:     PASS
run time:   3 minutes 11.442 seconds
--------------------------------------------------------------------------------
test_id:    kafkatest.tests.client.client_compatibility_produce_consume_test.ClientCompatibilityProduceConsumeTest.test_produce_consume.broker_version=2.7.2
status:     PASS
run time:   2 minutes 11.408 seconds
--------------------------------------------------------------------------------
test_id:    kafkatest.tests.client.client_compatibility_produce_consume_test.ClientCompatibilityProduceConsumeTest.test_produce_consume.broker_version=2.8.2
status:     PASS
run time:   2 minutes 13.602 seconds
--------------------------------------------------------------------------------
test_id:    kafkatest.tests.client.client_compatibility_produce_consume_test.ClientCompatibilityProduceConsumeTest.test_produce_consume.broker_version=3.0.2
status:     PASS
run time:   2 minutes 13.112 seconds
--------------------------------------------------------------------------------
test_id:    kafkatest.tests.client.client_compatibility_produce_consume_test.ClientCompatibilityProduceConsumeTest.test_produce_consume.broker_version=3.1.2
status:     PASS
run time:   2 minutes 0.379 seconds
--------------------------------------------------------------------------------
test_id:    kafkatest.tests.client.client_compatibility_produce_consume_test.ClientCompatibilityProduceConsumeTest.test_produce_consume.broker_version=3.2.3
status:     PASS
run time:   1 minute 53.272 seconds
--------------------------------------------------------------------------------
test_id:    kafkatest.tests.client.client_compatibility_produce_consume_test.ClientCompatibilityProduceConsumeTest.test_produce_consume.broker_version=3.3.2
status:     PASS
run time:   1 minute 53.859 seconds
--------------------------------------------------------------------------------
test_id:    kafkatest.tests.client.client_compatibility_produce_consume_test.ClientCompatibilityProduceConsumeTest.test_produce_consume.broker_version=3.4.1
status:     PASS
run time:   1 minute 53.557 seconds
--------------------------------------------------------------------------------
test_id:    kafkatest.tests.client.client_compatibility_produce_consume_test.ClientCompatibilityProduceConsumeTest.test_produce_consume.broker_version=3.5.2
status:     PASS
run time:   1 minute 53.813 seconds
--------------------------------------------------------------------------------
test_id:    kafkatest.tests.client.client_compatibility_produce_consume_test.ClientCompatibilityProduceConsumeTest.test_produce_consume.broker_version=3.6.2
status:     PASS
run time:   1 minute 53.501 seconds
--------------------------------------------------------------------------------
test_id:    kafkatest.tests.client.client_compatibility_produce_consume_test.ClientCompatibilityProduceConsumeTest.test_produce_consume.broker_version=3.7.1
status:     PASS
run time:   1 minute 53.747 seconds
--------------------------------------------------------------------------------
test_id:    kafkatest.tests.client.client_compatibility_produce_consume_test.ClientCompatibilityProduceConsumeTest.test_produce_consume.broker_version=3.8.1
status:     PASS
run time:   2 minutes 53.991 seconds
--------------------------------------------------------------------------------
test_id:    kafkatest.tests.client.client_compatibility_produce_consume_test.ClientCompatibilityProduceConsumeTest.test_produce_consume.broker_version=dev.metadata_quorum=ISOLATED_KRAFT
status:     PASS
run time:   2 minutes 44.947 seconds
--------------------------------------------------------------------------------

Committer Checklist (excluded from commit message)

  • Verify design and implementation
  • Verify test coverage and CI build status
  • Verify documentation (including upgrade notes)

@github-actions github-actions bot added tests Test fixes (including flaky tests) small Small PRs labels Nov 13, 2024
@@ -44,7 +44,7 @@ class ZookeeperService(KafkaPathResolverMixin, Service):
}

def __init__(self, context, num_nodes, zk_sasl = False, zk_client_port = True, zk_client_secure_port = False,
zk_tls_encrypt_only = False, version=DEV_BRANCH):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to replace all dev by 3.8 for this file

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @chia7712, I replace all dev with 3.8 in this file and add a comment. Could you take a look again when you have time? Thank you.

@chia7712 chia7712 merged commit 1c3ee7f into apache:trunk Nov 13, 2024
8 checks passed
@FrankYang0529 FrankYang0529 deleted the KAFKA-18004 branch November 13, 2024 12:39
chiacyu pushed a commit to chiacyu/kafka that referenced this pull request Nov 30, 2024
…nd tests (apache#17790)

We plan to remove all ZooKeeper-related code in version 4.0. However, some old brokers in the end-to-end tests still require ZooKeeper service, so we need to run the ZooKeeper service using the 3.x release instead of the dev branch.

Since version 3.9 is not available in the https://s3-us-west-2.amazonaws.com/kafka-packages repo, we can use version 3.8 for now.

Reviewers: Chia-Ping Tsai <[email protected]>
tedyu pushed a commit to tedyu/kafka that referenced this pull request Jan 6, 2025
…nd tests (apache#17790)

We plan to remove all ZooKeeper-related code in version 4.0. However, some old brokers in the end-to-end tests still require ZooKeeper service, so we need to run the ZooKeeper service using the 3.x release instead of the dev branch.

Since version 3.9 is not available in the https://s3-us-west-2.amazonaws.com/kafka-packages repo, we can use version 3.8 for now.

Reviewers: Chia-Ping Tsai <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
small Small PRs tests Test fixes (including flaky tests)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants