[Obs AI Assistant] Add query rewriting #224498
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sorenlouv
commented
Jun 19, 2025
x-pack/platform/plugins/shared/observability_ai_assistant/server/functions/context.ts
Outdated
Show resolved
Hide resolved
Contributor
|
Pinging @elastic/obs-ai-assistant (Team:Obs AI Assistant) |
Contributor
🤖 GitHub commentsExpand to view the GitHub comments
Just comment with:
|
54036a7 to
207a1bc
Compare
| ? [`--xpack.fleet.registryUrl=http://localhost:${dockerRegistryPort}`] | ||
| : []), | ||
| // @ts-expect-error | ||
| ...(options?.kbnTestServer?.serverArgs ?? []), |
Contributor
There was a problem hiding this comment.
Why are you doing this given the error we throw on line 44 if options.kbnServerArgs is set?
I think you can customize the kbnServer args in specific configs but not in this shared one if I remember correctly.
Member
Author
There was a problem hiding this comment.
Hey!
I want to enable debug logging in Kibana for the Obs AI Assistant plugin when running the Obs AI Assistant tests:
https://github.com/sorenlouv/kibana/blob/add-query-rewriting/x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.ai_assistant.stateful.config.ts
What's the right way to do this?
Why are you doing this given the error we throw on line 44 if options.kbnServerArgs is set?
btw yes, we throw an error for options.kbnServerArgs but not for options.kbnTestServer afaict.
Contributor
There was a problem hiding this comment.
I'm not sure, maybe someone from @elastic/appex-qa has more context on why we don't allow sending options.kbnServerArgs and if this usage of adding options?.kbnTestServer?.serverArgs is fine.
maybe @dmlemeshko
viduni94
approved these changes
Jun 23, 2025
Contributor
viduni94
left a comment
There was a problem hiding this comment.
Added a few comments.
I tested recalling and scoring in a normal conversation, contextual insights and via the evaluation framework scenario.
LGTM
...rm/plugins/shared/observability_ai_assistant/server/functions/context/utils/query_rewrite.ts
Outdated
Show resolved
Hide resolved
...rm/plugins/shared/observability_ai_assistant/server/functions/context/utils/query_rewrite.ts
Outdated
Show resolved
Hide resolved
...ation/deployment_agnostic/apis/observability/ai_assistant/complete/functions/context.spec.ts
Outdated
Show resolved
Hide resolved
...ation/deployment_agnostic/apis/observability/ai_assistant/complete/functions/context.spec.ts
Outdated
Show resolved
Hide resolved
...ation/deployment_agnostic/apis/observability/ai_assistant/complete/functions/context.spec.ts
Outdated
Show resolved
Hide resolved
...ation/deployment_agnostic/apis/observability/ai_assistant/complete/functions/context.spec.ts
Outdated
Show resolved
Hide resolved
x-pack/test/observability_ai_assistant_api_integration/common/create_llm_proxy.ts
Show resolved
Hide resolved
x-pack/test/observability_ai_assistant_api_integration/common/create_llm_proxy.ts
Outdated
Show resolved
Hide resolved
x-pack/test/observability_ai_assistant_api_integration/common/create_llm_proxy.ts
Outdated
Show resolved
Hide resolved
...rm/plugins/shared/observability_ai_assistant/server/functions/context/utils/query_rewrite.ts
Outdated
Show resolved
Hide resolved
dmlemeshko
approved these changes
Jun 24, 2025
Member
dmlemeshko
left a comment
There was a problem hiding this comment.
extended logger change LGTM
Co-authored-by: Viduni Wickramarachchi <[email protected]>
TattdCodeMonkey
approved these changes
Jun 24, 2025
Contributor
💛 Build succeeded, but was flaky
Failed CI StepsTest Failures
Metrics [docs]
History
|
Contributor
|
Starting backport for target branches: 8.19 |
Contributor
💔 All backports failed
Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
1 task
akowalska622
pushed a commit
to akowalska622/kibana
that referenced
this pull request
Jun 25, 2025
Closes elastic#224084 This improves the knowledge base retrieval by rewriting the user prompt before querying Elasticsearch. The LLM is asked to rewrite the prompt taking into account screen description and conversation history. We then use the LLM-generated prompt as the search query. Other changes: - Remove `screenContext` from being used verbatim as ES query. This was causing noise and leading to bad results - Take conversation history into account: with query rewriting, the LLM has access to the entire conversation history. This context will be embedded into the generated prompt along side screen context --------- Co-authored-by: Viduni Wickramarachchi <[email protected]>
Contributor
|
Friendly reminder: Looks like this PR hasn’t been backported yet. |
Contributor
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
viduni94
pushed a commit
to viduni94/kibana
that referenced
this pull request
Jun 26, 2025
Closes elastic#224084 This improves the knowledge base retrieval by rewriting the user prompt before querying Elasticsearch. The LLM is asked to rewrite the prompt taking into account screen description and conversation history. We then use the LLM-generated prompt as the search query. Other changes: - Remove `screenContext` from being used verbatim as ES query. This was causing noise and leading to bad results - Take conversation history into account: with query rewriting, the LLM has access to the entire conversation history. This context will be embedded into the generated prompt along side screen context --------- Co-authored-by: Viduni Wickramarachchi <[email protected]> (cherry picked from commit da41b47) # Conflicts: # x-pack/solutions/observability/plugins/observability_ai_assistant_app/scripts/evaluation/scenarios/kb/index.spec.ts # x-pack/test/api_integration/deployment_agnostic/apis/observability/ai_assistant/complete/functions/execute_query.spec.ts # x-pack/test/api_integration/deployment_agnostic/apis/observability/ai_assistant/complete/functions/get_alerts_dataset_info.spec.ts # x-pack/test/api_integration/deployment_agnostic/apis/observability/ai_assistant/complete/functions/get_dataset_info.spec.ts # x-pack/test/api_integration/deployment_agnostic/apis/observability/ai_assistant/complete/functions/retrieve_elastic_doc.spec.ts # x-pack/test/api_integration/deployment_agnostic/configs/stateful/oblt.ai_assistant.stateful.config.ts
viduni94
added a commit
that referenced
this pull request
Jun 27, 2025
# Backport This will backport the following commits from `main` to `8.19`: - [[Obs AI Assistant] Add query rewriting (#224498)](#224498) <!--- Backport version: 10.0.1 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Søren Louv-Jansen","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-06-24T16:19:15Z","message":"[Obs AI Assistant] Add query rewriting (#224498)\n\nCloses https://github.com/elastic/kibana/issues/224084\n\nThis improves the knowledge base retrieval by rewriting the user prompt\nbefore querying Elasticsearch. The LLM is asked to rewrite the prompt\ntaking into account screen description and conversation history. We then\nuse the LLM-generated prompt as the search query.\n\nOther changes:\n\n- Remove `screenContext` from being used verbatim as ES query. This was\ncausing noise and leading to bad results\n- Take conversation history into account: with query rewriting, the LLM\nhas access to the entire conversation history. This context will be\nembedded into the generated prompt along side screen context\n\n---------\n\nCo-authored-by: Viduni Wickramarachchi <[email protected]>","sha":"da41b47f1d2b297449377722ea0804880f0588ff","branchLabelMapping":{"^v9.1.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","backport missing","Team:Obs AI Assistant","ci:project-deploy-observability","backport:version","v9.1.0","v8.19.0"],"title":"[Obs AI Assistant] Add query rewriting","number":224498,"url":"https://github.com/elastic/kibana/pull/224498","mergeCommit":{"message":"[Obs AI Assistant] Add query rewriting (#224498)\n\nCloses https://github.com/elastic/kibana/issues/224084\n\nThis improves the knowledge base retrieval by rewriting the user prompt\nbefore querying Elasticsearch. The LLM is asked to rewrite the prompt\ntaking into account screen description and conversation history. We then\nuse the LLM-generated prompt as the search query.\n\nOther changes:\n\n- Remove `screenContext` from being used verbatim as ES query. This was\ncausing noise and leading to bad results\n- Take conversation history into account: with query rewriting, the LLM\nhas access to the entire conversation history. This context will be\nembedded into the generated prompt along side screen context\n\n---------\n\nCo-authored-by: Viduni Wickramarachchi <[email protected]>","sha":"da41b47f1d2b297449377722ea0804880f0588ff"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"main","label":"v9.1.0","branchLabelMappingKey":"^v9.1.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/224498","number":224498,"mergeCommit":{"message":"[Obs AI Assistant] Add query rewriting (#224498)\n\nCloses https://github.com/elastic/kibana/issues/224084\n\nThis improves the knowledge base retrieval by rewriting the user prompt\nbefore querying Elasticsearch. The LLM is asked to rewrite the prompt\ntaking into account screen description and conversation history. We then\nuse the LLM-generated prompt as the search query.\n\nOther changes:\n\n- Remove `screenContext` from being used verbatim as ES query. This was\ncausing noise and leading to bad results\n- Take conversation history into account: with query rewriting, the LLM\nhas access to the entire conversation history. This context will be\nembedded into the generated prompt along side screen context\n\n---------\n\nCo-authored-by: Viduni Wickramarachchi <[email protected]>","sha":"da41b47f1d2b297449377722ea0804880f0588ff"}},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> --------- Co-authored-by: Søren Louv-Jansen <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes #224084
This improves the knowledge base retrieval by rewriting the user prompt before querying Elasticsearch. The LLM is asked to rewrite the prompt taking into account screen description and conversation history. We then use the LLM-generated prompt as the search query.
Other changes:
screenContextfrom being used verbatim as ES query. This was causing noise and leading to bad results