Jie He

Large language models (LLMs) have demonstrated impressive capabilities in various reasoning tasks but face significant challenges with complex, knowledge-intensive multi-hop queries, particularly those involving new or long-tail knowledge. Existing benchmarks often fail to fully address these challenges. To bridge this gap, we introduce MINTQA (Multi-hop Question Answering on New and Tail Knowledge), a comprehensive benchmark to evaluate LLMs' capabilities in multi-hop reasoning across four critical dimensions: question handling strategy, sub-question generation, retrieval-augmented generation, and iterative or dynamic decomposition and retrieval. MINTQA comprises 10,479 question-answer pairs for evaluating new knowledge and 17,887 pairs for assessing long-tail knowledge, with each question equipped with corresponding sub-questions and answers. Our systematic evaluation of 22 state-of-the-art LLMs on MINTQA reveals significant limitations in their ability to handle complex knowledge base queries, particularly in handling new or unpopular knowledge. Our findings highlight critical challenges and offer insights for advancing multi-hop reasoning capabilities.

Large Language Models (LLMs) can enhance their capabilities as AI assistants by integrating external tools, allowing them to access a wider range of information. We present GenTool, a novel training framework addressing two dimensions: Zero-to-One Generalization and Weak-to-Strong Generalization. Through extensive experiments across four generalization scenarios, our method significantly enhances tool-usage capabilities of LLMs ranging from 1B to 8B parameters, achieving performance that surpasses GPT-4o.

We conduct a comprehensive evaluation of prompting open-source LLMs on graph-to-text generation tasks. We propose a novel diversity-difficulty-based few-shot sample selection method and introduce PlanGTG, a new graph-to-text dataset annotated with reordering and attribution sub-tasks. Extensive evaluations demonstrate significant improvements in generated text quality from both few-shot and fine-tuning perspectives.

We propose MiCEval, a framework to assess the correctness of reasoning chains in Multimodal Chain of Thought (MCoT) by evaluating the quality of both the description and each reasoning step. MiCEval is built upon a fine-grained dataset with annotations rating each step according to correctness, relevance, and informativeness. Experiments on four MLLMs show step-wise evaluations align more closely with human judgments.

We study the robustness of retrieval-augmented ICL against adversarial attacks. Retrieval-augmented models enhance robustness against test sample attacks (4.87% ASR reduction) but exhibit overconfidence in demonstrations (2% ASR increase). We introduce DARD, a training-free adversarial defence method, achieving a 15% reduction in ASR over baselines.

We study Parameter-Efficient Fine-Tuning (PEFT) methods for MLLMs, evaluating four PEFT methods on seven datasets. We show that the adapter is the best-performing PEFT method, and fine-tuning connector layers leads to improved performance in most MLLMs.

We propose to transform the downstream multiple choice QA task into a simpler binary classification task by ranking all candidate answers according to their reasonableness. We convert knowledge graph triples into reasonable and unreasonable texts for training. Extensive results show effectiveness on various QA benchmarks, with our approach being less data hungry than existing methods.

We propose TGEA, using carefully selected prompts to guide GPT-2 to generate candidate sentences, selecting 47K for error annotation. We create an error taxonomy covering 24 types of errors and propose a series of automatic diagnosis tasks including error detection, error type classification, associated span detection, and error rationale generation.

We present a test suite to evaluate the commonsense reasoning capability of neural machine translation, covering lexical and syntactic ambiguity that requires commonsense knowledge to resolve. We manually create 1,200 triples and demonstrate that neural machine translation performs poorly on commonsense reasoning.