Let's Chat to Find the APIs: Connecting Human, LLM and Knowledge Graph through AI Chain——KPL.pptx

Editor's Notes

• #2: Hi everyone. My name is Zhenyu Wan, a graduate student from the Jiangxi Normal University in China.Today, the paper I am going to present is called "Let's Chat to Find the APIs: Connecting Human, LLM and Knowledge Graph through AI Chain".OK, First, I will introduce the ISSUE in the paper.
• #3: API recommendation tools recommend APIs by obtaining user query, which can help developers quickly find APIs that meet their query.   However, these API recommendation tools cannot accurately recommend due to short user query, so tools to extend user query have been developed.   There are two ways to extend tools: direct extension and clarification extension.   Direct extension method (b) cannot accurately extend user query and does not understand user intent.   Clarification extension methods (c) and (d), while capable of capturing user intent during interactions with the user, may have knowledge base limitations or no optimal clarification paths.
• #4: When one gets user intent using clarification extension method,   If the clarification knowledge base is limited, it is difficult to accurately clarify user intent.   If there is no optimal clarification path, more clarification rounds are needed to clarify user intent.
• #5: In order to accurately recommend APIs, existing works [1][2] ask clarification questions to obtain user intent.   These methods clarify user query based on a limited knowledge base, which can lead to OOV issues.   On the other hand, in order to solve the OOV problem, we can use LLM to ask clarification questions, but LLM may ask random questions about user query.   Therefore, we use the optimal clarification paths in external API knowledge base to guide LLM in asking clarification questions.
• #6: Our approach is based on large language models, such as gpt-3.5, which have rich API knowledge/can solve OOV problems. We also utilize KG as an external API knowledge base to guide LLM to ask clarification questions with optimal clarification paths. The whole method revolves around the idea of ai-chain and builds a complete set of problem clarification chains.   The red part in the figure is the clarification question and option generation part. The user gives query, and KG matches the optimal path according to the query, and guides LLM to raise clarification questions as examples. LLM then gives possible options through query and problems for users to choose.   As shown in the black part, the user makes a choice and gives an answer.   In the blue part of the figure, LLM combines the query and clarification records to extend the query in a targeted manner, and finally recommends APIs that meet the user intent based on the extended query.
• #7: This table shows the best clarification path in the form of domain knowledge we obtain from API KG.   For example, the best questioning aspect in round 1 is "event", ask the question accordingly, and answer it. At this time, the best clarification aspect in the next round is "purpose". Round2 asks questions and answers with the aspect of "purpose". At this time, the best clarification aspect in the next round is "status". Round3 asks questions with the "status" aspect, gives answers, and finally recommends APIs.   The above is an example of an optimal path to clarification.
• #8: Here are the prompts used by 5 AI units in our method.   (a) The best question aspect generation unit, which gives the task query and the best question aspect examples matched from KG, so that LLM can generate the best question aspect according to user query. (b) The clarification question generation unit provides background settings and limits the question aspects, allowing LLM to raise clarification questions based on known information. (c) The alternatice option generation unit provides possible answer options based on user needs and clarification questions raised. (d) query extension unit, which extends query in a targeted manner based on query and clarification records. (e) API recommendation unit, which recommends APIs according to the extended query.
• #9: We set up three RQs to demonstrate the effectiveness of our approach.   RQ1: What is the effectiveness of each AI unit?   RQ2: How well does KPL perform in extending query statements across two scenarios, namely, query statement covered in API KG and those not covered in API KG?   RQ3: How effective is the knowledge-guided pathfinding strategy employed in our approach?
• #10: In the experiment, our data comes from the Stack Overflow data set of Huang et al. 60 in their dataset were processed at a more fine-grained level.   In addition, in order to exclude the influence of KG, we followed the method of Huang et al. and collected 60 pieces of data from Stack Overflow that were not in the API KG.
• #11: In this experiment, we demonstrated the effectiveness of each AI unit we set up through user experience.   We invited three master students with more than three years of programming experience who were not involved in our project to experience our approach and fill out a questionnaire to score each AI unit.   In the five-point scoring system, each AI unit received a high score of about 4.5, and the kappa coefficient was 0.6213, ensuring the consistency of the scoring.   This shows that the 5 AI units set by our method are reasonable and effective.
• #12: In this experiment we not only test the effectiveness of our method in KG data, but also the effectiveness of our method for query that are not in KG.   We compared the two methods ZaCQ and KAHAID from the four metrics of MRR, MAP, Precision and Recall respectively.   Overall, our method performs better than the other two methods at every round of clarification, whether using data from KG or not.   Our method shows good accuracy in both data, which proves that the knowledge in KG does not directly affect LLM decision-making, but guides LLM with "experience".
• #13: In this experiment, we compare our method with method without KG guidance and method with KG but without extracting the optimal path strategy.   the results demonstrate the effectiveness of KG guidance and optimal path extraction strategy.   Among the four evaluation metrics, our method performs 15%-22% better than the other two methods in each round of clarification.   In addition, methods without KG are sometimes better than methods with KG but without extracting the optimal path strategy. This is because the optimal path is not extracted, causing knowledge to incorrectly guide LLM.
• #14: In general, we propose a new knowledge-guided clarification method for API recommendation using KG to guide LLM, which not only solves the oov problem but also prevents LLM from asking random questions.   We leverage the structured API knowledge and entity relations in KG to filter noise and transfer the optimal clarification path from KG to LLM, thus improving the efficiency of the clarification process.   Our method is decomposed into an AI chain consisting of five steps, each of which is handled by a separate LLM call, thereby improving the accuracy, efficiency, and fluency of query clarification in API recommendation.   Our approach bridges the gap between KG and LLM, combining the strengths of both and compensating for the weaknesses of both.
• #15: That's all my report, thank you.