Location>code7788 >text

Direct, unprompted interaction with LLMs in the IDE is a promising future direction for tool builders to explore

Popularity:833 ℃/2024-08-13 11:13:53

This idea is presented in a collaborative article by researchers at Carnegie Mellon University and Google

《Using an LLM to Help With Code Understanding》

Presented in.

Paper Address:/doi/abs/10.1145/3597503.3639187

summaries

Understanding code can be very challenging, especially in new and complex development environments. Code comments and documentation can help, but are often under-resourced or difficult to navigate. Large Language Models (LLMs) are revolutionizing the process of writing code. Can they be equally helpful in understanding code?

This study provides the first study of an LLM-based conversational user interface that is directly embedded in an IDE (Integrated Development Environment) and is designed to facilitate code understanding.

Our IDE plugin uses OpenAI's GPT-3.5-turbo model to query four high-level requests, which can be accomplished without the user having to write explicit prompts: explaining the highlighted snippet, providing details about the API used in the code, explaining domain-specific terminology, and providing an example of the API's use.

The plugin also supports open-ended hints that automatically contextualize LLM according to the program being edited.

We evaluated this system in a user study among 32-bit participants, and the results showed that using our plug-in could facilitate task completion more than web search.

We also analyzed in detail how developers use our system and the ways in which they find it useful. Among other things, it was found that students and professionals use the system and derive different benefits from it.

We conclude that direct, unprompted interaction with large language models in IDEs is a promising future direction for tool builders to explore.

present (sb for a job etc)

Building and maintaining software systems requires a deep understanding of the code base. As a result, developers spend a great deal of time searching and gathering the information they need, and organizing and digesting the information they find that they need.

Understanding code is a daunting task, and developers need to absorb a great deal of information about the semantics of the code, the nuances of the APIs used, and the related domain expertise.

This information is often scattered across multiple sources and it becomes very difficult for developers, especially beginners or those working with unfamiliar APIs, to find the information they need. In addition, most of the relevant information is largely under-documented or scattered in different formats and media, ultimately leading to outdated information.

With the rise of large-scale language model (LLM)-driven code generation tools, it is clear that the need for informational support for code comprehension is also increasing.

These tools are able to generate code automatically, giving effective results even to developers who have limited programming skills or little knowledge of a particular domain. However, this convenience often comes with a price - developers may be left with code they don't understand at all.

Indeed, early studies of LLM code generation tools found that developers had trouble debugging code generated by LLM and were easily frustrated.

Fortunately, LLMs (Large Language Models) also offer an opportunity in this area to help developers deal with problems they encounter with unfamiliar code by providing out-of-the-box, code-generation-based information support for them.

LLM (Large Language Model) hints allow developers to provide more contextual information than a typical web search query. This actually allows them to access information that is more precisely tailored to their specific needs. Using LLM hints can be a much more economical time investment than taking a large amount of information from the web and then sifting through it to find content that meets a specific need.

Developers did start using web-based session LLM tools like ChatGPT at scale to get programming support, but this setup required them to then copy the relevant context from the IDE to the chat system for support when switching contexts.

In order to explore the potential for generative information support to be applied directly in a developer's programming environment, we have developed a prototype tool called GILT (Generation-based Information-support with LLM Technology).

GILT is capable of generating personalized, on-the-fly information and integrating it with the user's local coding environment. We have made this approach to interacting with LLMs more natural by incorporating this information in the prompts provided to the backend language model (LLM). This approach also introduces a new way of interacting without having to provide prompts. This design is intended to alleviate the cognitive burden of writing hints for developers who have limited domain-specific or programming knowledge.

Given the insufficient knowledge of best practices for using large-scale language models (LLMs) to provide information support (as opposed to just code generation), we conducted an exploratory user study with 32 participants assigned the task: the challenging task of understanding and extending unfamiliar code involving new domain concepts as well as Python data visualization and 3D rendering APIs. Our study quantitatively compares the effects of using a code editor with an LLM helper inside it, the effects of searching web pages directly in a web browser, and measuring task completion rates and code comprehension between the two. In addition, we delve deeper by qualitatively examining how participants used the two tools and their overall satisfaction with this novel interaction mode.

Specifically, we answered the following three research questions:

  1. How much of an impact does GILT have on developer understanding, task completion time, and task completion rates when faced with unfamiliar code?
  2. How do developers interact with GILT and what are the differences between different participants?
  3. How do developers see the usefulness of GILT?

Our results confirm that we do achieve a statistically significant advantage when using GILT (Generation-based Information-support with LLM Technology) to complete the task compared to performing a web search, demonstrating the utility of this information support. However, we did not find the same improvement in time efficiency and depth of understanding, which leaves room for future improvement. We also found that the magnitude of this benefit varied between students and professionals and explored this.

reach a verdict

We conducted a user study to investigate the effectiveness of generative information support using Large Language Models (LLMs) to help developers understand code. With our embedded IDE prototyping tool, GILT, we show that this approach significantly enhances developers' ability to accomplish their tasks, compared to traditional search-based information acquisition. At the same time, we also find that the degree of benefit developers can derive from the tool varies between students and professionals, and that they interact with the tool differently based on their learning style and familiarity with other AI tools.