[LockManager] Ensure index template are created #218901
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Pinging @elastic/obs-ai-assistant (Team:Obs AI Assistant) |
Contributor
🤖 GitHub commentsExpand to view the GitHub comments
Just comment with:
|
Contributor
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#8209[✅] x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.ai_assistant.serverless.config.ts: 50/50 tests passed. |
Contributor
Flaky Test Runner Stats🟠 Some tests failed. - kibana-flaky-test-suite-runner#8211[✅] x-pack/test/api_integration/deployment_agnostic/configs/serverless/oblt.ai_assistant.serverless.config.ts: 50/50 tests passed. |
viduni94
approved these changes
Apr 23, 2025
Contributor
viduni94
left a comment
There was a problem hiding this comment.
LGTM
Tested as follows:
- Started Kibana on
main - Lock index has incorrect mapping (see below)
- Checked out this PR
- Re-started Kibana
- Verified that the lock index mapping is the expected format (see below)
...servability_ai_assistant/server/service/distributed_lock_manager/setup_lock_manager_index.ts
Outdated
Show resolved
Hide resolved
…er/service/distributed_lock_manager/setup_lock_manager_index.ts Co-authored-by: Viduni Wickramarachchi <[email protected]>
Contributor
⏳ Build in-progress
History
|
Member
Author
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
sorenlouv
added a commit
to sorenlouv/kibana
that referenced
this pull request
Apr 30, 2025
Closes: elastic#218944 The index template for the Lock Manager was not created, causing index mappings and settings to be incorrect. Root cause: the function responsible for creating the index template (`ensureTemplatesAndIndexCreated`) was never invoked - only during automated testing 🤦 The mappings for the lock manager index (`.kibana_locks-000001`) should match this: ```ts { mappings: { dynamic: false, properties: { token: { type: 'keyword' }, metadata: { enabled: false }, createdAt: { type: 'date' }, expiresAt: { type: 'date' }, }, }, } ``` In this test we make sure that the LockManager library can recover and fix the mappings if the existing index has invalid mappings ``` DELETE .kibana_locks-000001 DELETE _index_template/.kibana_locks-index-template DELETE _component_template/.kibana_locks-component ``` correct mappings ``` PUT .kibana_locks-000001 ``` (Restart Kibana) ``` GET .kibana_locks-000001/_mapping ``` In this test we make sure that out of the box, the LockManager library creates an index with the correct mappings ``` DELETE .kibana_locks-000001 DELETE _index_template/.kibana_locks-index-template DELETE _component_template/.kibana_locks-component ``` (Restart Kibana) ``` GET .kibana_locks-000001/_mapping ``` Related: elastic#216916 elastic#216397 --------- Co-authored-by: Viduni Wickramarachchi <[email protected]> (cherry picked from commit f684ea4)
sorenlouv
added a commit
to sorenlouv/kibana
that referenced
this pull request
Apr 30, 2025
Closes: elastic#218944 The index template for the Lock Manager was not created, causing index mappings and settings to be incorrect. Root cause: the function responsible for creating the index template (`ensureTemplatesAndIndexCreated`) was never invoked - only during automated testing 🤦 The mappings for the lock manager index (`.kibana_locks-000001`) should match this: ```ts { mappings: { dynamic: false, properties: { token: { type: 'keyword' }, metadata: { enabled: false }, createdAt: { type: 'date' }, expiresAt: { type: 'date' }, }, }, } ``` In this test we make sure that the LockManager library can recover and fix the mappings if the existing index has invalid mappings ``` DELETE .kibana_locks-000001 DELETE _index_template/.kibana_locks-index-template DELETE _component_template/.kibana_locks-component ``` correct mappings ``` PUT .kibana_locks-000001 ``` (Restart Kibana) ``` GET .kibana_locks-000001/_mapping ``` In this test we make sure that out of the box, the LockManager library creates an index with the correct mappings ``` DELETE .kibana_locks-000001 DELETE _index_template/.kibana_locks-index-template DELETE _component_template/.kibana_locks-component ``` (Restart Kibana) ``` GET .kibana_locks-000001/_mapping ``` Related: elastic#216916 elastic#216397 --------- Co-authored-by: Viduni Wickramarachchi <[email protected]> (cherry picked from commit f684ea4)
sorenlouv
added a commit
that referenced
this pull request
May 1, 2025
# Backport This will backport the following commits from `main` to `8.19`: - [[Obs AI Assistant] Skip lock tests in MKI temporarily (#216753)](#216753) - [[Obs AI Assistant] Replace Task Manager usage with Lock Manager (#216916)](#216916) - [[LockManager] Ensure index template are created (#218901)](#218901) - [[LockManager] Expose as package (#219220)](#219220) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Søren Louv-Jansen","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-04-02T09:28:35Z","message":"[Obs AI Assistant] Skip lock tests in MKI temporarily (#216753)\n\nTests added in #216397 are failing\non MKI. Skipping temporarily in the affected environment\n\n### Error\n\n```\n └- ✖ fail: Serverless Observability - Deployment-agnostic API integration tests observability AI Assistant LockManager Basic lock operations acquires the lock when not held\n │ ResponseError: security_exception\n │ \tRoot causes:\n │ \t\tsecurity_exception: action [indices:admin/create] is unauthorized for user [testing-internal] with effective roles [superuser] on restricted indices [.kibana_locks-000001], this action is granted by the index privileges [create_index,manage,all]\n```\n\n### Root cause\n```ts\nconst es = getService('es');\nes.deleteByQuery({ index: '.kibana_locks-000001', query: { match_all: {} }});\n```","sha":"8bcce2e89b6be758d144b9b802eb9d2a09dd62cf","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:skip","v9.1.0","v8.19.0"],"title":"[Obs AI Assistant] Skip lock tests in MKI temporarily","number":216753,"url":"https://github.com/elastic/kibana/pull/216753","mergeCommit":{"message":"[Obs AI Assistant] Skip lock tests in MKI temporarily (#216753)\n\nTests added in #216397 are failing\non MKI. Skipping temporarily in the affected environment\n\n### Error\n\n```\n └- ✖ fail: Serverless Observability - Deployment-agnostic API integration tests observability AI Assistant LockManager Basic lock operations acquires the lock when not held\n │ ResponseError: security_exception\n │ \tRoot causes:\n │ \t\tsecurity_exception: action [indices:admin/create] is unauthorized for user [testing-internal] with effective roles [superuser] on restricted indices [.kibana_locks-000001], this action is granted by the index privileges [create_index,manage,all]\n```\n\n### Root cause\n```ts\nconst es = getService('es');\nes.deleteByQuery({ index: '.kibana_locks-000001', query: { match_all: {} }});\n```","sha":"8bcce2e89b6be758d144b9b802eb9d2a09dd62cf"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/216753","number":216753,"mergeCommit":{"message":"[Obs AI Assistant] Skip lock tests in MKI temporarily (#216753)\n\nTests added in #216397 are failing\non MKI. Skipping temporarily in the affected environment\n\n### Error\n\n```\n └- ✖ fail: Serverless Observability - Deployment-agnostic API integration tests observability AI Assistant LockManager Basic lock operations acquires the lock when not held\n │ ResponseError: security_exception\n │ \tRoot causes:\n │ \t\tsecurity_exception: action [indices:admin/create] is unauthorized for user [testing-internal] with effective roles [superuser] on restricted indices [.kibana_locks-000001], this action is granted by the index privileges [create_index,manage,all]\n```\n\n### Root cause\n```ts\nconst es = getService('es');\nes.deleteByQuery({ index: '.kibana_locks-000001', query: { match_all: {} }});\n```","sha":"8bcce2e89b6be758d144b9b802eb9d2a09dd62cf"}},{"branch":"8.x","label":"v8.19.0","branchLabelMappingKey":"^v8.19.0$","isSourceBranch":false,"state":"NOT_CREATED"}]},{"author":{"name":"Søren Louv-Jansen","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-04-08T08:13:00Z","message":"[Obs AI Assistant] Replace Task Manager usage with Lock Manager (#216916)\n\nThis PR replaces usage of the Kibana Task Manager with the [LockManager\nutility](#216397) that ensures\nstrict mutual exclusion when running operations such as re-indexing or\ndata migrations.\n\n---------\n\nCo-authored-by: Viduni Wickramarachchi <[email protected]>\nCo-authored-by: kibanamachine <[email protected]>","sha":"624410524f1a07dc72d92759464a3f4a85466c7a","branchLabelMapping":{"^v9.1.0$":"main","^v8.19.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:skip","Team:Obs AI Assistant","v9.1.0"],"title":"[Obs AI Assistant] Replace Task Manager usage with Lock Manager","number":216916,"url":"https://github.com/elastic/kibana/pull/216916","mergeCommit":{"message":"[Obs AI Assistant] Replace Task Manager usage with Lock Manager (#216916)\n\nThis PR replaces usage of the Kibana Task Manager with the [LockManager\nutility](#216397) that ensures\nstrict mutual exclusion when running operations such as re-indexing or\ndata migrations.\n\n---------\n\nCo-authored-by: Viduni Wickramarachchi <[email protected]>\nCo-authored-by: kibanamachine <[email protected]>","sha":"624410524f1a07dc72d92759464a3f4a85466c7a"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/216916","number":216916,"mergeCommit":{"message":"[Obs AI Assistant] Replace Task Manager usage with Lock Manager (#216916)\n\nThis PR replaces usage of the Kibana Task Manager with the [LockManager\nutility](#216397) that ensures\nstrict mutual exclusion when running operations such as re-indexing or\ndata migrations.\n\n---------\n\nCo-authored-by: Viduni Wickramarachchi <[email protected]>\nCo-authored-by: kibanamachine <[email protected]>","sha":"624410524f1a07dc72d92759464a3f4a85466c7a"}}]},{"author":{"name":"Søren Louv-Jansen","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-04-23T23:34:28Z","message":"[LockManager] Ensure index template are created (#218901)\n\nCloses: https://github.com/elastic/kibana/issues/218944\n\nThe index template for the Lock Manager was not created, causing index\nmappings and settings to be incorrect.\nRoot cause: the function responsible for creating the index template\n(`ensureTemplatesAndIndexCreated`) was never invoked - only during\nautomated testing 🤦\n\n## Manual testing steps\n\nThe mappings for the lock manager index (`.kibana_locks-000001`) should\nmatch this:\n```ts\n {\n mappings: {\n dynamic: false,\n properties: {\n token: { type: 'keyword' },\n metadata: { enabled: false },\n createdAt: { type: 'date' },\n expiresAt: { type: 'date' },\n },\n },\n}\n```\n\n### Test 1\n\nIn this test we make sure that the LockManager library can recover and\nfix the mappings if the existing index has invalid mappings\n\n#### Delete existing assets if they already exist\n```\nDELETE .kibana_locks-000001\nDELETE _index_template/.kibana_locks-index-template\nDELETE _component_template/.kibana_locks-component\n```\n\n#### Create lock index. This is invalid because it does not have the\ncorrect mappings\n```\nPUT .kibana_locks-000001\n```\n\n(Restart Kibana)\n\n#### Check mappings are correct\n```\nGET .kibana_locks-000001/_mapping\n```\n\n### Test 2\n\nIn this test we make sure that out of the box, the LockManager library\ncreates an index with the correct mappings\n\n```\nDELETE .kibana_locks-000001\nDELETE _index_template/.kibana_locks-index-template\nDELETE _component_template/.kibana_locks-component\n```\n\n(Restart Kibana)\n\n#### Check mappings are correct\n```\nGET .kibana_locks-000001/_mapping\n```\n\nRelated: https://github.com/elastic/kibana/pull/216916\nhttps://github.com/elastic/kibana/pull/216397\n\n---------\n\nCo-authored-by: Viduni Wickramarachchi <[email protected]>","sha":"f684ea4071dfb5e6acc4cb057b46acbf915943d7","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","backport:skip","Team:Obs AI Assistant","ci:project-deploy-observability","v9.1.0"],"title":"[LockManager] Ensure index template are created","number":218901,"url":"https://github.com/elastic/kibana/pull/218901","mergeCommit":{"message":"[LockManager] Ensure index template are created (#218901)\n\nCloses: https://github.com/elastic/kibana/issues/218944\n\nThe index template for the Lock Manager was not created, causing index\nmappings and settings to be incorrect.\nRoot cause: the function responsible for creating the index template\n(`ensureTemplatesAndIndexCreated`) was never invoked - only during\nautomated testing 🤦\n\n## Manual testing steps\n\nThe mappings for the lock manager index (`.kibana_locks-000001`) should\nmatch this:\n```ts\n {\n mappings: {\n dynamic: false,\n properties: {\n token: { type: 'keyword' },\n metadata: { enabled: false },\n createdAt: { type: 'date' },\n expiresAt: { type: 'date' },\n },\n },\n}\n```\n\n### Test 1\n\nIn this test we make sure that the LockManager library can recover and\nfix the mappings if the existing index has invalid mappings\n\n#### Delete existing assets if they already exist\n```\nDELETE .kibana_locks-000001\nDELETE _index_template/.kibana_locks-index-template\nDELETE _component_template/.kibana_locks-component\n```\n\n#### Create lock index. This is invalid because it does not have the\ncorrect mappings\n```\nPUT .kibana_locks-000001\n```\n\n(Restart Kibana)\n\n#### Check mappings are correct\n```\nGET .kibana_locks-000001/_mapping\n```\n\n### Test 2\n\nIn this test we make sure that out of the box, the LockManager library\ncreates an index with the correct mappings\n\n```\nDELETE .kibana_locks-000001\nDELETE _index_template/.kibana_locks-index-template\nDELETE _component_template/.kibana_locks-component\n```\n\n(Restart Kibana)\n\n#### Check mappings are correct\n```\nGET .kibana_locks-000001/_mapping\n```\n\nRelated: https://github.com/elastic/kibana/pull/216916\nhttps://github.com/elastic/kibana/pull/216397\n\n---------\n\nCo-authored-by: Viduni Wickramarachchi <[email protected]>","sha":"f684ea4071dfb5e6acc4cb057b46acbf915943d7"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/218901","number":218901,"mergeCommit":{"message":"[LockManager] Ensure index template are created (#218901)\n\nCloses: https://github.com/elastic/kibana/issues/218944\n\nThe index template for the Lock Manager was not created, causing index\nmappings and settings to be incorrect.\nRoot cause: the function responsible for creating the index template\n(`ensureTemplatesAndIndexCreated`) was never invoked - only during\nautomated testing 🤦\n\n## Manual testing steps\n\nThe mappings for the lock manager index (`.kibana_locks-000001`) should\nmatch this:\n```ts\n {\n mappings: {\n dynamic: false,\n properties: {\n token: { type: 'keyword' },\n metadata: { enabled: false },\n createdAt: { type: 'date' },\n expiresAt: { type: 'date' },\n },\n },\n}\n```\n\n### Test 1\n\nIn this test we make sure that the LockManager library can recover and\nfix the mappings if the existing index has invalid mappings\n\n#### Delete existing assets if they already exist\n```\nDELETE .kibana_locks-000001\nDELETE _index_template/.kibana_locks-index-template\nDELETE _component_template/.kibana_locks-component\n```\n\n#### Create lock index. This is invalid because it does not have the\ncorrect mappings\n```\nPUT .kibana_locks-000001\n```\n\n(Restart Kibana)\n\n#### Check mappings are correct\n```\nGET .kibana_locks-000001/_mapping\n```\n\n### Test 2\n\nIn this test we make sure that out of the box, the LockManager library\ncreates an index with the correct mappings\n\n```\nDELETE .kibana_locks-000001\nDELETE _index_template/.kibana_locks-index-template\nDELETE _component_template/.kibana_locks-component\n```\n\n(Restart Kibana)\n\n#### Check mappings are correct\n```\nGET .kibana_locks-000001/_mapping\n```\n\nRelated: https://github.com/elastic/kibana/pull/216916\nhttps://github.com/elastic/kibana/pull/216397\n\n---------\n\nCo-authored-by: Viduni Wickramarachchi <[email protected]>","sha":"f684ea4071dfb5e6acc4cb057b46acbf915943d7"}}]},{"author":{"name":"Søren Louv-Jansen","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-04-29T16:42:45Z","message":"[LockManager] Expose as package (#219220)\n\nExpose LockManager as package to make it easier to consume from other\nplugins\n\ncc @nchaulet\n\n---------\n\nCo-authored-by: kibanamachine <[email protected]>\nCo-authored-by: Viduni Wickramarachchi <[email protected]>","sha":"8b8d569986f216185755aa7ac98a2a3bbeb84f76","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Obs AI Assistant","ci:project-deploy-observability","backport:version","v9.1.0","v8.19.0"],"title":"[LockManager] Expose as package","number":219220,"url":"https://github.com/elastic/kibana/pull/219220","mergeCommit":{"message":"[LockManager] Expose as package (#219220)\n\nExpose LockManager as package to make it easier to consume from other\nplugins\n\ncc @nchaulet\n\n---------\n\nCo-authored-by: kibanamachine <[email protected]>\nCo-authored-by: Viduni Wickramarachchi <[email protected]>","sha":"8b8d569986f216185755aa7ac98a2a3bbeb84f76"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/219220","number":219220,"mergeCommit":{"message":"[LockManager] Expose as package (#219220)\n\nExpose LockManager as package to make it easier to consume from other\nplugins\n\ncc @nchaulet\n\n---------\n\nCo-authored-by: kibanamachine <[email protected]>\nCo-authored-by: Viduni Wickramarachchi <[email protected]>","sha":"8b8d569986f216185755aa7ac98a2a3bbeb84f76"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> --------- Co-authored-by: kibanamachine <[email protected]>
akowalska622
pushed a commit
to akowalska622/kibana
that referenced
this pull request
May 29, 2025
Closes: elastic#218944 The index template for the Lock Manager was not created, causing index mappings and settings to be incorrect. Root cause: the function responsible for creating the index template (`ensureTemplatesAndIndexCreated`) was never invoked - only during automated testing 🤦 ## Manual testing steps The mappings for the lock manager index (`.kibana_locks-000001`) should match this: ```ts { mappings: { dynamic: false, properties: { token: { type: 'keyword' }, metadata: { enabled: false }, createdAt: { type: 'date' }, expiresAt: { type: 'date' }, }, }, } ``` ### Test 1 In this test we make sure that the LockManager library can recover and fix the mappings if the existing index has invalid mappings #### Delete existing assets if they already exist ``` DELETE .kibana_locks-000001 DELETE _index_template/.kibana_locks-index-template DELETE _component_template/.kibana_locks-component ``` #### Create lock index. This is invalid because it does not have the correct mappings ``` PUT .kibana_locks-000001 ``` (Restart Kibana) #### Check mappings are correct ``` GET .kibana_locks-000001/_mapping ``` ### Test 2 In this test we make sure that out of the box, the LockManager library creates an index with the correct mappings ``` DELETE .kibana_locks-000001 DELETE _index_template/.kibana_locks-index-template DELETE _component_template/.kibana_locks-component ``` (Restart Kibana) #### Check mappings are correct ``` GET .kibana_locks-000001/_mapping ``` Related: elastic#216916 elastic#216397 --------- Co-authored-by: Viduni Wickramarachchi <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes: #218944
The index template for the Lock Manager was not created, causing index mappings and settings to be incorrect.
Root cause: the function responsible for creating the index template (
ensureTemplatesAndIndexCreated) was never invoked - only during automated testing 🤦Manual testing steps
The mappings for the lock manager index (
.kibana_locks-000001) should match this:Test 1
In this test we make sure that the LockManager library can recover and fix the mappings if the existing index has invalid mappings
Delete existing assets if they already exist
Create lock index. This is invalid because it does not have the correct mappings
(Restart Kibana)
Check mappings are correct
Test 2
In this test we make sure that out of the box, the LockManager library creates an index with the correct mappings
(Restart Kibana)
Check mappings are correct
Related: #216916 #216397