Structuring Low-Rank Adaptation with Semantic Guidance for Model Fine-Tuning
Hongye Zheng, Yumeng Ma, Yichen Wang, Guiran Liu, Zhen Qi, Xu Yan
Abstract
This paper addresses the challenges of fine-tuning efficiency and semantic adaptation in large language models for question answering tasks. It proposes a low-rank parameter adaptation method that incorporates semantic representations. While keeping the main model parameters frozen, the method introduces a semantic guidance function to improve traditional low-rank tuning strategies. This allows the parameter update process to dynamically align with input semantics, enhancing the model’s ability to perceive complex semantic structures. The method embeds a semantic-aware module into the attention layers of the Transformer architecture. It uses representation vectors generated by a semantic encoder to guide the construction of low-rank matrices. In addition, a semantic similarity regularization term is applied to enforce consistency in the model’s responses to semantically similar inputs. The method was evaluated across multiple experimental settings. These include comparisons with existing mainstream parameter-efficient fine-tuning approaches, analysis of adaptability to different QA types, and robustness under semantic perturbation. In all cases, the proposed method demonstrates strong accuracy, stability, and generalization ability. Furthermore, training loss curves show that the method achieves good convergence speed and training stability during optimization. Overall, the results indicate that the semantically guided low-rank adaptation strategy enhances the semantic understanding of QA systems while significantly reducing computational and storage costs during fine-tuning. This provides a simple yet robust solution for building efficient intelligent QA models.