Machine learning is revolutionizing industries, but with that power comes the potential for errors and complexities, often leaving users perplexed about when to rely on a model’s predictions. To tackle this issue, scientists have pioneered methods to explain these predictions better, making the system more user-friendly.
Despite their value, these explanations can sometimes be intricate, featuring detailed data on numerous model characteristics. Additionally, they are frequently displayed as intricate visualizations that may confound those without expertise in machine learning.
To bridge this gap, researchers from MIT have harnessed the capabilities of large language models (LLMs) to convert complex graphical explanations into straightforward, comprehensible language.
They designed a two-phase system that translates a machine-learning explanation into a clear paragraph and then assesses the quality of the written summary, enabling users to discern whether to trust the information provided.
By presenting the system with a few examples, the researchers can tailor narrative descriptions to align with the preferences of users or the demands of specific applications. In the long term, they aspire to enhance this approach, allowing users to ask follow-up questions regarding the model’s predictions in real-world contexts.
“Our primary objective was to enable users to engage in meaningful conversations with machine-learning models about their predictions, thus improving decision-making on whether to heed the model’s advice,” shares Alexandra Zytek, a graduate student in electrical engineering and computer science (EECS) and the lead author of the paper detailing this advancement.
Zytek’s team includes Sara Pido, a postdoctoral fellow at MIT; Sarah Alnegheimish, another EECS graduate student; Laure Berti-Équille, a research director at the French National Research Institute for Sustainable Development; and Kalyan Veeramachaneni, a principal research scientist at the Laboratory for Information and Decision Systems, who is the senior author of the paper. This research will be showcased at the IEEE Big Data Conference.
Understanding Explanations
The team specifically investigated a widely used machine-learning explanation method known as SHAP (SHapley Additive exPlanations). SHAP assigns a value to each feature utilized in a model’s prediction. For example, in predicting house prices, a key feature might be the home’s location, which would be linked to a positive or negative value indicating its impact on the final price prediction.
SHAP explanations frequently manifest as bar charts highlighting the significance of various features. However, this becomes cumbersome for models with an extensive array of features, often exceeding 100.
“As researchers, we often grapple with decisions about visual presentations. Opting to display only the top few features can lead to questions about those that are excluded. Relying on natural language alleviates these dilemmas,” remarks Veeramachaneni.
Instead of solely generating explanations through a large language model, the researchers utilize the LLM to convert existing SHAP explanations into a narrative format, ensuring a more accurate representation.
Their dual-component system, named EXPLINGO, comprises two interconnected elements.
The first part, NARRATOR, employs an LLM to craft readable descriptions of SHAP explanations tailored to user preferences. By supplying the NARRATOR with anywhere from three to five written examples, the LLM is able to mimic the desired narrative style during generation.
“Rather than forcing users to define the kind of explanation they seek, it’s simpler for them to articulate what they want to see,” notes Zytek. This makes NARRATOR adaptable for various scenarios by presenting it with alternate manually crafted examples.
Post-NARRATOR’s creation of an explanation, the second element, GRADER, assesses the narrative across four essential metrics: conciseness, accuracy, completeness, and fluency. GRADER prompts the LLM using both the text from NARRATOR and the SHAP explanation it accompanies.
“We realized that even if an LLM stumbles on task execution, it tends to excel in validation, ensuring high standards,” she adds. Users can also modify the importance assigned to each evaluation metric. For instance, in critical scenarios, users might prioritize accuracy and completeness over fluency.
Evaluating Narratives
A significant challenge for Zytek and her colleagues was fine-tuning the LLM to generate narratives that sound natural. Interestingly, the addition of more directives to shape the style often led to increased inaccuracies within the explanations.
“A lot of prompt adjustment was necessary to identify and rectify mistakes methodically,” she explains.
To test their innovative system, the researchers analyzed nine machine-learning datasets, tasked with generating narratives for each dataset through different user interactions. This provided valuable insight into NARRATOR’s capability to adopt unique styles. GRADER was employed to evaluate each generated narrative based on all four metrics.
The results revealed that their system could effectively produce high-quality narrative explanations while adeptly reflecting various writing styles. Their findings confirm that providing just a handful of manually written examples significantly enhances narrative quality. That said, careful crafting of these examples is essential—using comparative language can mislead GRADER into marking accurate explanations as flawed.
Building on these promising results, the researchers aim to investigate improved techniques for managing comparative terms in explanations. They also plan to broaden EXPLINGO’s capabilities to incorporate rationale into its narratives.
Ultimately, they envision developing an interactive system that allows users to inquire about particular predictions enabling better decision-making processes. Zytek envisions a scenario: “If users question a model’s prediction, we want to facilitate a swift method for verifying whether their instincts align with the model’s and identifying the source of any discrepancies.”
Photo credit & article inspired by: Massachusetts Institute of Technology
