Steering LVLMs via Sparse Autoencoder for Hallucination Mitigation

To investigate whether there exist directions in LVLMs that are highly correlated with hallucinatory and faithful semantics, we randomly sampled $4,000$ image-text pairs from the MSCOCO dataset Lin et al. (2014). Using the LLaVA-Next-8b model Liu et al. (2024d) for the image captioning task, we extract the residual stream representations from the $25^{\text{th}}$ layer when the model generated object tokens classified as either hallucinatory or faithful.

Given that a small proportion of object words are tokenized into multiple subword units, we exclude these cases to simplify the analysis. Furthermore, because each sample exhibited an imbalance between the counts of hallucinatory and faithful object terms, we enforce class balance by sampling an equal number of residual vectors from each category per image–text pair. See Appendix A for a description of the process. Finally, we construct a balanced dataset containing $1,784$ samples and divide it into a training set and a test set in a $9:1$ ratio, for direction mining and direction effectiveness validation, respectively.

Inspired by Meng et al. (2022); Ferrando et al. (2025), we leverage SAE to identify latent directions aligned with hallucinatory and faithful semantics. Specifically, each residual stream sample from the training set is passed through the SAE, and we record the activation frequency of each latent activation across hallucinatory samples $\mathcal{X_{\text{hall}}}$ and faithful samples $\mathcal{X_{\text{faithful}}}$. For a given latent activation $j$, its activation frequencies on hallucinatory samples $f_{j}^{\text{hall}}$ and faithful samples $f_{j}^{\text{faithful}}$ are computed as:

where $N_{\text{hall}}$ and $N_{\text{faithful}}$ represent the number of hallucinatory and faithful samples, respectively. To quantify the semantic relevance of each latent activation, we compute the difference in activation frequencies as follows:

These values reflect the relevance of latent dimension $j$ to hallucinatory and faithful semantics, respectively. Finally, we identify the latent activation with the highest $s_{j}^{\text{hall}}$ as the hallucination semantic direction (hereafter referred to as the hall latent), and the one with the highest $s_{j}^{\text{faithful}}$ as the faithful semantic direction (hereafter referred to as the faithful latent).

We begin by analyzing the distributional differences of the hallucinatory latent and faithful latent activations across both sample types in the test set. These distributions are visualized using kernel density estimation (KDE) plots as shown in Figure 2. We further quantify the separation using independent two-sample $t$-tests and compute Cohen’s $d$ to assess effect sizes. Both latent activations exhibit statistically significant distributional shifts, with substantial effect sizes, confirming their discriminative power.

To probe the semantic alignment of the activations, we further conduct Spearman rank correlation analysis between the activation values of hallucinatory samples and their associated hallucinatory object terms. The hallucinatory latent correlates positively with hallucinatory objects (Spearman’s $\rho=0.42$, $p=9.95\times 10^{-9}$), whereas the faithful latent correlates negatively ($\rho=-0.44$, $p=9.85\times 10^{-10}$). Given the binary nature of the labels and the symmetry of rank correlation, we infer a positive correlation between the faithful latent and samples.

Finally, to quantitatively evaluate the predictive power of SAE-derived directions in distinguishing hallucinatory from faithful samples, we design a set of classification experiments based on logistic regression. The model takes the latent activation values—either individually or in combination—as input features for a binary hallucination classification task. As illustrated in Figure 4, the results demonstrate that the latent semantic directions extracted by the SAE are discriminative (see Appendix B for more experimental details). Furthermore, combining the hall latent and faithful latent as input features yields further performance improvements. A more detailed discussion on clarifying the boundary between hallucinatory and faithful semantic directions can be found in Appendix C.

A core architectural mechanism involves multi-layer residual connections that progressively model input information. At each layer, semantic representations are passed through residual flows, which can be divided according to the input sequence into four main components: system token, prompt tokens, visual tokens, and output tokens. Among these, visual tokens interact with language tokens to guide the model’s understanding of image content. Injecting faithful direction at this position enables the model to increase visual faithfulness. Output tokens represent the model’s autoregressive language generations, influenced by both prompts and visual inputs, thereby reflecting the model’s semantic behavior. Suppressing hallucinatory directions during this stage helps reduce hallucination tendencies and enhances factual consistency in language generation.

Following the method described in Section 2, we identify two semantic direction vectors: the hallucinatory direction $d_{\text{hall}}$ and the faithful direction $d_{\text{faithful}}$. During the visual feature fusion stage, we incorporate $d_{\text{faithful}}$ to improve the faithfulness of visual understanding. In the subsequent language generation stage, we suppress activations along $d_{\text{hall}}$ to reduce the risk of hallucinatory outputs. Semantic steering at layer $l$ is defined as follows:

where $\alpha$ is a tunable hyperparameter controlling the strength of semantic steering.

The setting of the steering strength $\alpha$ plays a crucial role in determining the effectiveness of semantic intervention. Traditional steering approaches often rely on a fixed hyperparameter $\alpha$ to linearly combine the steering vector with the residual representations. However, this fixed strategy can result in unstable or suboptimal performance, as the magnitude of residual vectors can vary across model layers and token positions. In such cases, a change that is too small may fail to induce meaningful guidance, while an excessively large change may cause semantic distortion or instability.

To address this limitation, we propose an adaptive feature steering mechanism, which dynamically adjusts the steering strength based on the norm of the residual vector at each token at a given layer. This approach ensures more stable and context-aware intervention across varying model states. Specifically, the adaptive steering strength $\alpha$ is computed as:

where $\gamma$ is a scaling factor, $x_{\text{residual}}$ denotes the residual vector, $d_{\text{steer}}$ is the steering direction, and $\epsilon$ is a small constant to avoid numerical instability. An overview of the proposed SSL method is presented in Figure 3. The complete procedure of SSL is provided in Algorithm 1.

We evaluate object hallucination using the Caption Hallucination Assessment with Image Relevance (CHAIR) metric Rohrbach et al. (2018), which compares generated image captions against ground-truth annotations to detect hallucinatory objects mentioned in the captions but absent from the image. CHAIR includes two metrics at both captions level ($\text{CHAIR}_{S}$) and object level ($\text{CHAIR}_{I}$):

We randomly sample $500$ images from the COCO 2014 validation set Lin et al. (2014) and conduct five runs with different random seeds. For all LVLMs, captions are generated using the prompt “Please describe this image in detail.” We report the mean and standard deviation for each metric.

We further evaluate object hallucination using the POPE benchmark Li et al. (2023b), a question-answering dataset designed to assess the factual consistency of generated image descriptions. POPE contains $500$ images from the MSCOCO dataset Lin et al. (2014), each paired with binary questions of the form: “Is there a <object> in the image?” The dataset comprises three subsets—random, popular, and adversarial—which differ in their object sampling strategies. Model performance is measured using standard classification metrics: Accuracy, Precision, Recall, and F1 score. To provide an overall assessment, we report the average results across all three subsets.

We evaluate LVLM performance using the LLaVA-Bench (In-the-Wild) benchmark Liu et al. (2024c), a comprehensive set designed to assess models across diverse and challenging visual scenarios. The benchmark includes $24$ images from varied real-world contexts, such as indoor scenes, outdoor environments, and internet memes, paired with $60$ carefully curated questions spanning open-ended QA, fine-grained descriptions, and complex reasoning. We prompt the GPT-4o model to evaluate the LVLMs’ outputs along two dimensions: factual accuracy and response detail.

To validate the necessity of adaptively adjusting the steering strength, we conduct an ablation study by replacing ASP with a fixed $\alpha$ steering parameter as shown in Equation 4 equal to the $\gamma$ value. As shown in Table 4, removing the adaptive strategy ASP results in consistent performance drops across all three model architectures, highlighting the importance of ASP in effectively mitigating hallucinations.

We conduct an ablation study on LLaVA-NeXT-8b to examine the impact of applying guidance at different layers. Figure 5 shows that the choice of guidance layer significantly affects model performance, setting $\gamma$ to $0.8$. For LLaVA-NeXT-8b, applying SSL at middle layers yields more effective mitigation of hallucinations, with layer $15$ achieving the best performance. For further analysis on steering layer selection and scaling factor $\gamma$ settings across different models, please refer to Appendix D and E.

To further validate the effectiveness of the semantic directions we identified, we compare the CHAIR benchmark across three model architectures under three settings: original model state, SSL, and Reverse-SSL (see Appendix F for details on Reverse-SSL). As shown in Figure 6, across all three model architectures, applying Reverse-SSL significantly increases hallucinations, while standard SSL guidance consistently reduces hallucinations. To further clearly illustrate the effectiveness of SSL, we present several qualitative examples in Appendix G.

During each generation step, SSL dynamically adjusts the steering strength through a single scaling and weighting operation, introducing negligible computational overhead. Compared to the overall generation process, the additional latency introduced by SSL is minimal. A comparison of inference time between SSL and other baselines is shown in Figure 7.

To further analyze the impact of model size on our method, we conducted additional experiments on the 7B, 8B, and 11B models, evaluating SSL performance on both the CHAIR and POPE benchmarks, with results reported in Table 5 and Table 6. Specifically, for the 11B model, we use Llama-3.2-11b-Vision-Instruct released by Meta Llama, where SSL is applied with $\gamma=0.4$ at the $32^{\text{th}}$ layer. The results demonstrate that SSL effectively mitigates hallucinations across models of different sizes and consistently outperforms other baseline methods.