Bao Gia Doan^{1}^{∗}, Afshar Shamsi^{2}^{∗}, Xiao-Yu Guo^{1}, Arash Mohammadi^{2}

Hamid Alinejad-Rokny^{3} Dino Sejdinovic^{1}, Damith C. Ranasinghe^{1}, Ehsan Abbasnejad^{1}^{1}The University of Adelaide, Australia ^{2}Concordia University, Canada ^{3} UNSW Sydney, Australia

###### Abstract

Computational complexity of Bayesian learning is impeding its adoption in practical, large-scale tasks. Despite demonstrations of significant merits such as improved robustness and resilience to unseen or out-of-distribution inputs over their non-Bayesian counterparts, their practical use has faded to near insignificance. In this study, we introduce an innovative framework to mitigate the computational burden of Bayesian neural networks (BNNs).Our approach follows the principle of Bayesian techniques based on deep ensembles, but significantly reduces their cost via multiple low-rank perturbations of parameters arising from a pre-trained neural network. Both vanilla version of ensembles as well as more sophisticated schemes such as Bayesian learning with Stein Variational Gradient Descent (SVGD), previously deemed impractical for large models, can be seamlessly implemented within the proposed framework, called Bayesian Low-Rank LeArning (Bella).In a nutshell, i)Bella achieves a dramatic reduction in the number of trainable parameters required to approximate a Bayesian posterior; and ii) it not only maintains, but in some instances, surpasses the performance of conventional Bayesian learning methods and non-Bayesian baselines. Our results with large-scale tasks such as ImageNet, CAMELYON17, DomainNet, VQA with CLIP, LLaVA demonstrate the effectiveness and versatility of Bella in building highly scalable and practical Bayesian deep models for real-world applications.

## 1 Introduction

Bayesian deep learning[1] provides mechanisms for building more robust predictive models—i)robustness to adversarial attacks; ii)unseen or out-of-distribution data—and iii)a theoretical framework for estimating model uncertainty[2, 3, 4, 5, 6]. In particular, quantifying uncertainty enables more reliable decision-making and facilitates identifying potential model vulnerabilities in uncertain regions of the input space. Consequently, embracing Bayesian deep neural networks (BNNs) represents a significant stride towards building more reliable and trustworthy AI systems for various real-world applications (e.g., autonomous driving, medical image analysis, etc.). Unfortunately, their practical use is encumbered by their computational complexity.

Unlike traditional alternatives with point estimates—a single set of model parameters mapping inputs to outputs— BNNs learn the distribution of model parameters to offer a distribution over possible predictions.Consider a neural network $f(\mathbf{x},\boldsymbol{\theta})$ with input $\mathbf{x}$ and weights $\boldsymbol{\theta}$ and a prior distribution on network weights $p(\boldsymbol{\theta})$. The likelihood function $p(\mathcal{D}|\boldsymbol{\theta})$ determines how well the network with weights $\boldsymbol{\theta}$ fits the data $\mathcal{D}$ [7]. Bayesian inference integrates the likelihood and the prior using Bayes’ theorem to derive a posterior distribution over the space of weights, given by$p\left(\boldsymbol{\theta}|\mathcal{D}\right)=\frac{p\left(\mathcal{D}|%\boldsymbol{\theta}\right)p\left(\boldsymbol{\theta}\right)}{p(\mathcal{D})}$and seek to compute the predictive distribution

$\displaystyle p(y|\mathbf{x},\mathcal{D})=\int p(y\mid\mathbf{x},\boldsymbol{%\theta})p(\boldsymbol{\theta}\mid\mathcal{D})d\boldsymbol{\theta}.\$ | (1) |

Despite evidence to support the adoption of Bayesian learning, their widespread use is hindered by several challenges; a primary issue is the intractability of the posterior distribution $p(\boldsymbol{\theta}|\mathcal{D})$ in practical learning tasks.The exact solution for the posterior, even for networks of moderate size, is impractical. Because of network complexity and the high-dimensional integral of the resulting denominator.This necessitates the use of approximations.

In this paper, we consider approximations of the posterior as ensemble or Stein Variational Gradient Descent (SVGD) but provide a practical method to learn BNNs for large-scale, learning tasks.

Effectively, SVGD employs a mechanism to separate the parameters of the ensembles (i.e. particles), ensuring the representation of alternative modes in the posterior distribution. But computational demands of the approximation, especially the memory demands for storing all parameters, hinder adoption in large-scale models such as CLIP[8].However, a recent conjecture[9] suggests that the permutation of optimal model parameters can lead to models that are not functionally different. Further, the path between modes may be linear. Therefore, we consider spawning parameter particles (models) by adding a linear interpolation to a pre-trained model’s parameters. But, it would still require updating a large number of model parameters and we are still confronted by computation complexity. The recent conjecture in[10] suggests modes may yet exist in the constrained region in the parameter space. Therefore, we propose only adding a linear interpolation to a portion of the model parameters and, subsequently, only learning them as illustrated in Figure1 for the CAMELYON17 task—a benchmark task for out-of-distribution evaluations. We summarise our key contributions as:

- •
We propose a new Bayesian learning framework, Bella, for SVGD approximation of the posterior—spawning particles or models, by exploiting the availability of pre-trained models, with linear interpolation of a constrained set of model parameters for learning with SVGD to approximate the posterior.

- •
Our approach more efficiently captures the complexity and multi-modality of the solution space compared to current SVGD butat a fraction of the cost—we observe on-par performance in terms of uncertainty estimation, performance improvement, and robustness, updating only 0.3% or less parameters.

- •
We demonstrate Bella to perform on par or better than baselines—ensembles or current SVGD implementations. Bella consistently outperforms the non-BNN counterparts on 8 datasets;image classification (ImageNet, CIFAR 10/100), Out-of-distribution (DomainNet, Camleyon17

^{1}^{1}1Notably, our approach sets a new state-of-the-art for the CAMELYON17 (seeTable6 for more details) dataset challenge according to the leaderboard[11].[12], CIFAR-10-C), and VQA as well as adversarial robustness.Notably, our approach, on a multi-modal architecture such as LlaVa[13], leads to improved performance and uncertainty estimation highly correlated with human confidence.

## 2 Related Work

Increasing network complexity has led to advances in parameter-efficient fine tuning and recent advances in exploring these methods for Bayesian learning, predominantly for language models. We discuss these in the context of our work.

Parameter-Efficient Fine Tuning.In contrast to fine-tuning all parameters, recent research proposed inserting adapters in between existing neural layers to reduce the number of trainable parameters and, subsequently, the compute (GPU consumption)[14, 15, 16].Hu et al.[17] use a bottleneck structure to impose a low-rank constraint on the weight updates, named LoRA. The key functional difference is that LoRA can be merged with the main weights during inference, thus avoiding the introduction of any latency whilst significantly reducing the number of parameters.

Fine-Tuning Approaches and Bayesian Deep Learning.Previous research investigating the application of fine-tuning approaches for BNNs have predominantly focused on large language models (LLMs), e.g. [18, 19, 20]. Notably, marking a departure from the conventional methods relying primarily on tuning the network’s parameters, these studies chose to define priors and approximate posterior over low rank attention weights. Concurrently, Yang et al.[20] introduced the concept of Laplace LoRA to incorporate Bayesian concepts to enhance the calibration of fine-tuned LLMs. However, Laplace’s method relies on a Gaussian approximation of the posterior distribution. Whilst this can be effective for unimodal and symmetric distributions, the approach does not fully encapsulate the intricacies of more complex posteriors, particularly in neural networks where the posterior has multimodality and asymmetry[21]. Deep ensembles[22] typically perform better in practice compared with variational and Laplace methods, due to their ability to capture multiple modes. When employing fine-tuning, a direct application of ensembling for LoRAs was considered in [23]. Some interpretations of deep ensembles suggest that they approximate gradient flows in function spaces and that building desirable properties into an ensemble (such as repulsive behavior), is possible[24]. SVGD[25], can be viewed in a similar vein. However, while these more sophisticated, repulsive, ensembling approaches are highly impractical in the pre-training phase, we argue that their expressivity can be brought to bear precisely in tandem with low-rank fine-tuning, which is the viewpoint we adopt in this contribution.

Building upon these foundations, our research represents a pioneering effort to apply the principles of repulsive ensemble-based low-rank fine-tuning to computer vision. In particular, we bridge pre-training and fine-tuning phases with recent conjectures on mode connectivity[9]. Our methodology not only capitalizes on the efficiency of fine-tuning techniques, e.g. [26, 27, 28] but also innovatively addresses the scalability challenges inherent to BNNs. Overall, our work sets a new precedent in applying Bayesian approaches to computer vision tasks by offering a scalable and efficient framework for enhancing model performance.

## 3 Background on Stein Variational Gradient Descent (SVGD) Approximations for a Bayesian Posterior

Here, we provide: i)a brief overview of methods to address the intractability of the posterior distribution $p(\boldsymbol{\theta}|\mathcal{D})$ in Bayesian learning tasks; and ii)we elaborate on our motivation for seeking a practical method to learn BNNs for large-scale tasks with SVGD.

Bayesian inference techniques have been integral to the development of neural networks, with a rich history underscored by previous works [29, 1, 30, 31, 32]. These methodologies provide a structured framework for generating reliable and interpretable estimates of uncertainty. However, the focus of prior research on Bayesian neural networks has predominantly been on the initial pretraining stages rather than on subsequent fine-tuning processes [33, 34]. This emphasis on pretraining presents significant challenges in terms of scalability and resource requirements, limiting the application of Bayesian neural networks to smaller datasets and network architectures with a minimal number of parameter particles. Consequently, the scalability issues hinder the extension of Bayesian neural networks to larger datasets, such as ImageNet[35], and more complex network architectures, like CLIP[8], which involve millions of parameters.

Variational Inference (VI) [36, 32] and Markov Chain Monte Carlo (MCMC)[29, 31] are two primary approximate Bayesian inference frameworks. The former substitutes the true posterior with a tractable alternative while the latter involves sampling.However, accurately computing the posterior with either MCMC or VI becomes computationally infeasible when dealing with large-scale networks containing millions of parameters. Although approximations can be obtained more efficiently with VI[36], VI is also demonstrably too restrictive to resemble the multi-modality of the true posterior and suffers from mode collapse[21].

Stein Variational Gradient Descent (SVGD)[25] is an alternative approximate Bayesian technique which combines the strengths of MCMC and VI by transporting a set of parameter *particles* to fit the true posterior distribution, while encouraging diversity among the particles, by incorporating a repulsive term in the parameter updates.This diversity prevents the mode collapse and enables learning multiple models to represent various patterns in the data.Using $n$ samples from the posterior (*i.e*. parameter particles), SVGD modifies the gradient descent as:

$\displaystyle\boldsymbol{\theta}_{i}=\boldsymbol{\theta}_{i}$ | $\displaystyle-\epsilon_{i}\hat{\boldsymbol{\phi}}{}^{*}(\boldsymbol{\theta}_{i%})\text{\quad with}\quad\hat{\boldsymbol{\phi}}{}^{*}(\boldsymbol{\theta})=%\sum_{j=1}^{n}\big{[}k(\boldsymbol{\theta}_{j},\boldsymbol{\theta})\nabla_{%\boldsymbol{\theta}_{j}}\log p(\boldsymbol{\theta}_{j}|\mathcal{D})-\frac{%\gamma}{n}\nabla_{\boldsymbol{\theta}_{j}}k(\boldsymbol{\theta}_{j},%\boldsymbol{\theta})\big{]}\,.$ | (2) |

Here, $\boldsymbol{\theta}_{i}$ is the $i$th particle, $k(\cdot,\cdot)$ is a kernel function that measures the similarity between particles and $\gamma$ is a hyper-parameter. Notably, the kernel function encourages the particles to be dissimilar in order to capture more diverse samples from the posterior and $\gamma$ controls the trade-off between the diversity of the samples versus the minimization of the loss.

## 4 Bayesian Low-Rank Learning (Bella) for SVGD

The problem with the current SVGD approach in large deep neural networks is its huge computational cost. This renders it infeasible to train efficiently and to scale to a sufficient number of parameter particles for accurately approximating the posterior distribution, which currently remains coarse.In this work, we propose to capitalize on the low-rank representations of fine-tuning in order to construct a practical and scalable variant of SVGD. Consider any dense layer, for which there is a fixed pre-trained weight matrix $\boldsymbol{\theta}_{0}\in\mathbb{R}^{d_{1}\times d_{2}}$ with $d_{1},d_{2}$ the corresponding numbers of hidden units. We consider $n$ low-rank perturbations of $\boldsymbol{\theta}_{0}$ as

$\boldsymbol{\theta}_{i}=\boldsymbol{\theta}_{0}+\Delta\boldsymbol{\theta}_{i}=%\boldsymbol{\theta}_{0}+\mathbf{B}_{i}\mathbf{A}_{i}\,,\quad i=1,\ldots,n.$ | (3) |

where $\mathbf{B}_{i}\in\mathbb{R}^{d_{1}\times r}$, $\mathbf{A}_{i}\in\mathbb{R}^{r\times d_{2}}$ are the low-dimensional update parameters, and $r\ll d_{1},d_{2}$ is the rank of the update.Now Bella proceeds as the joint SVGD on $(\mathbf{A}_{i},\mathbf{B}_{i})$, with updates

$\displaystyle\mathbf{A}_{i}=\mathbf{A}_{i}-\epsilon_{i}\sum_{j=1}^{n}\hat{%\boldsymbol{\phi}}{}_{j}^{*}(\mathbf{A}_{i}),\quad\mathbf{B}_{i}=\mathbf{B}_{i%}-\epsilon_{i}\sum_{j=1}^{n}\hat{\boldsymbol{\phi}}{}_{j}^{*}(\mathbf{B}_{i})%\text{\quad with}$ | (4) | |||

$\displaystyle\hat{\boldsymbol{\phi}}{}_{j}^{*}(\mathbf{B}_{i})$ | $\displaystyle=k\left(\mathbf{B}_{j}\mathbf{A}_{j},\mathbf{B}_{i}\mathbf{A}_{i}%\right)\!\nabla_{\mathbf{B}_{i}}\ell\left(f(\mathbf{x};\boldsymbol{\theta}_{0}%+\mathbf{B}_{i}\mathbf{A}_{i}),y\right)-\frac{\gamma}{n}\nabla_{\mathbf{B}_{i}%}k\left(\mathbf{B}_{j}\mathbf{A}_{j},\mathbf{B}_{i}\mathbf{A}_{i}\right),$ | |||

$\displaystyle\hat{\boldsymbol{\phi}}{}_{j}^{*}(\mathbf{A}_{i})$ | $\displaystyle=k\left(\mathbf{B}_{j}\mathbf{A}_{j},\mathbf{B}_{i}\mathbf{A}_{i}%\right)\!\nabla_{\mathbf{A}_{i}}\ell\left(f(\mathbf{x};\boldsymbol{\theta}_{0}%+\mathbf{B}_{i}\mathbf{A}_{i}),y\right)-\frac{\gamma}{n}\nabla_{\mathbf{A}_{i}%}k\left(\mathbf{B}_{j}\mathbf{A}_{j},\mathbf{B}_{i}\mathbf{A}_{i}\right)\,.$ |

Here, we have placed the (improper) uniform prior on $(\mathbf{A}_{i},\mathbf{B}_{i})$, but other choices are possible. Uniform prior implies that we only require the gradients of the log-likelihood computed on a mini-batch of data, as well as the kernel function to encourage particle diversity.Note that the kernel function on $(\mathbf{A}_{i},\mathbf{B}_{i})$ is given by $k\left(\boldsymbol{\theta}_{0}+\mathbf{B}_{j}\mathbf{A}_{j},\boldsymbol{\theta%}_{0}+\mathbf{B}_{i}\mathbf{A}_{i}\right)$, which ensures that the similarity is computed on the original parameter space, and in the commonly used case of shift-invariant kernels, this simplifies to $k\left(\mathbf{B}_{j}\mathbf{A}_{j},\mathbf{B}_{i}\mathbf{A}_{i}\right)$. Further simplifications are obtained for specific kernel functions – in particular, in the case of Gaussian RBF, while the naive implementation would require the cost of $O(rd_{1}d_{2})$ for a single kernel evaluation, we can bring it down to $O(r^{2}(d_{1}+d_{2}))$ using standard trace manipulation, as described in the appendix. This procedure can be repeated across all dense layers.

Bella introduces a significant improvement in the efficiency of model training and execution. By utilizing the same pre-trained weights $\boldsymbol{\theta}_{0}$ across all parameter particles but allowing for individual low-rank adaptations $\Delta\boldsymbol{\theta}_{i}$, we achieve a balance between parameter sharing and the diversity necessary for effective learning.Bella significantly reduces the parameter space from the full matrix’s $d_{1}d_{2}$ to just $r(d_{1}+d_{2})$, thereby enhancing both efficiency and scalability.This setup not only reduces the computational burden during training but also streamlines the process at inference time. The heavy lifting is done once by loading the large base model $\boldsymbol{\theta}_{0}$, and the lightweight low-rank adapters $\Delta\boldsymbol{\theta}_{i}$ can be dynamically applied with minimal overhead in order to approximate the posterior predictive distribution as $p(y^{*}\mid\mathbf{x}^{*},\mathcal{D})\approx\frac{1}{n}\sum_{i=1}^{n}p(y^{*}%\mid\mathbf{x},\boldsymbol{\theta}_{0}+\Delta\boldsymbol{\theta}_{i})$.This approach is particularly advantageous in large-scale models, where the weight matrices $\boldsymbol{\theta}_{0}$ are of substantial dimensions.

## 5 Empirical Experiments and Results

In this section, we provide an in-depth overview of our experimental setup, detailing the methodology, equipment, and procedures employed in the implementation of our Bella. We meticulously outline the configuration settings, dataset, and the criteria used for evaluating outcomes, ensuring a transparent and reproducible framework. Following this, we present the experimental results, offering a thorough analysis of the data obtained, including statistical evaluations and interpretations of the findings in the context of our research objectives. This comprehensive approach ensures a clear understanding of the experiment’s execution and its contributions to the field.

### 5.1 Experimental Set-up

Datasets.In this research, we have employed a variety of datasets, each selected for its relevance and contribution to the field of image recognition and computer vision. These datasets include CIFAR-10, CIFAR-100[37], CIFAR-10-C[38], STL-10[39], CAMELYON17[40], ImageNet[35], and DomainNet[41].We also employ VQA v2 dataset containing billions of questions answers given hundred thousands of images[42], which is utilized in Visual Question Answering (VQA) task.Detailed information about datasets is presented inAppendixH andAppendixF.

Networks.In our experiments, we employed the CLIP ViT-B/32 model[43], a pre-trained variant utilizing contrastive supervision from image-text pairs, as initially introduced in the seminal CLIP research[44]. We conducted end-to-end fine-tuning, adjusting all model parameters, a strategy typically yielding higher accuracy compared to training only the final linear layer.In our methodology, we selectively fine-tune only the image encoder of the CLIP model while maintaining the text encoder frozen as it is common[45]. To elaborate, both during training and inference phases, the final output is derived through an inner product between the output produced by the image encoder and the embedding generated by the frozen text encoder. For both our ensemble-based methods which include both traditional ensemble[22] and the Stein Variational Gradient Descent (SVGD) approach[25], we employed the technique of utilizing the logits (unnormalized outputs) of the models, as described in[46]. This strategy has been consistently applied in our ensemble configurations, encompassing traditional ensembles as well as SVGD.We also employ Ensemble networks as it was shown to be a good and efficient way to approximate the posterior distribution to be baselines[21].For VQA task, we employ the SoTA LLaVA-1.5 7B with billions of parameters to showcase the effectiveness of our Bella on large-scale network architecture.Details regarding hyper-parameters are inAppendixH.

### 5.2 Benchmark Task Performance and Cost Efficiency

The performance comparison of our Bella models with their respective base models, as well as with Vision Bayesian Lora (VBL)—a derived Laplacian sampling method from[20] for vision tasks—is delineated inTable1. The respective costs comparison on training (in terms of trainable parameters) is shown inTable2—notably, the grey columns are the ones used for generating the results in Table1.

Datasets | Bella Models | Baseline Models (Base) | ||||

Ensemble(n=5) | SVGD(n=5) | Single | VBL | Ensemble(n=5) | SVGD(n=5) | |

CIFAR10 | 97.57 | 97.95 | 97.20 | 94.41 | 97.54 | 97.87 |

CIFAR100 | 86.50 | 88.09 | 86.17 | 86.22 | 86.89 | 87.32 |

CAMELYON17 | 94.47 | 94.89 | 92.38 | 92.96 | 94.23 | 95.21 |

DomainNet | 82.70 | 84.47 | 74.61 | 82.97 | 85.07 | 85.42 |

ImageNet | 77.29 | 78.24 | 76.87 | - | 78.93 | 79.36 |

Models | SVGD base | SVGD Bella | Single | ||||||

$n=3$ | $n=5$ | $n=20$ | $n=40$ | $n=3$ | $n=5$ | $n=20$ | $n=100$ | $n=1$ | |

Trainable Parameters | 340M | 567M | 1.76B | 3.51B | 1.10M | 1.84M | 7.37M | 36.86M | 113M |

Memory Consumption (RAM in GB) | 6.71 | 8.35 | 26.08 | 48.45 | 4.48 | 4.50 | 4.63 | 5.19 | 5.05 |

Storage Consumption (MB) | 1321 | 2222 | 8868 | 17735 | 436 | 439 | 460 | 572 | 433 |

Impressively, across a spectrum of benchmark computer vision datasets such as CIFAR-10, CIFAR-100, Camelyon17, and ImageNet, the Bella models demonstrate superior performance. This is achieved with only a fraction of cost (trainable parameters) as shown inTable2, underscoring the models’ proficiency in parameter efficiency without compromising on accuracy. Further, the Bella models surpass the performance of Single models, while employing a comparable level of computational resources—see the Memory Consumption of models used, in Table2 reporting approximately 4.5GB for Bella compared to 5.05GB for Single. The results across the benchmarks attest to the efficacy of our methodology.

Notably, in Table1 with Bella SVGD, with 1.6% of the trainable parameters in comparison to the Single baseline, leads to approximately 2% and 10% increase in performance on the OOD tasks of DomainNet and CAMELYON17, respectively; whilst achieving comparable performance with the current SVGD implementation (SVGD baseline model).

Along with Table1, Table2 demonstrates the primary benefit of our approach—the significant reduction in memory and storage needs. For example, Table2,we obtain approximately a 5$\times$ reduction in model size—that is from 2,222MB for a 5 particle SVGD baseline model to just 440MB for a Bella SVGD model with 5 particles. This efficiency not only reduces the demands on GPUs but also minimizes potential I/O bottlenecks.

Moreover, the reduced GPU demand, as shown inTable2, facilitates larger mini-batch sizes during training to speedup the training process. More crucially, it enables the enhancement of the Bayesian posterior’s parameter particles to over 100, a feat previously unattainable with current SVGD implementations. Significantly, constructing a 100 parameter Bayesian approximation consumes only 5.19GB memory compared to current SVGD implementations for Bayesian models (SVGD base) needing over 6GB for even a 3 particle approximation.

Interestingly, our on par results of Bella models with current SVGD and ensemble approximations of the posterior provide empirical evidence that with constrained model parameters, it is still possible to reach the diverse modes of the posterior. Further, the results support recent conjectures on mode connectivity[10].

Models | DomainNet Dataset | Average | ||||

Real | Clip-Art | Infograph | Paint | Sketch | ||

Single base | 74.61 | 55.29 | 31.81 | 53.87 | 43.84 | 51.89 |

VBL | 82.97 | 59.94 | 26.56 | 52.81 | 46.96 | 53.85 |

Ensemble Bella | 82.70 | 61.22 | 28.15 | 55.32 | 51.58 | 55.60 |

Ensemble base | 85.07 | 65.40 | 36.07 | 57.90 | 54.22 | 59.73 |

SVGD Bella | 84.47 | 63.67 | 32.22 | 56.83 | 54.86 | 58.41 |

SVGD base | 85.42 | 65.53 | 36.58 | 58.18 | 55.47 | 60.24 |

### 5.3 Performance on Out-Of-Disrtibution (OOD) Datasets

Assessing the robustness of machine learning systems to unseen conditions is crucial, especially their ability to generalize to out-of-distribution (OOD) data. We evaluate robutness to ODD using multiple OOD benchmarks.

First, we use the DomainNet dataset, one of the most diverse domain adaptation datasets, covering a wide range of visual styles from real images to abstract art. This variety provides a challenging test bed for algorithms aiming to bridge different visual domains. Our study involves training the CLIP network on DomainNet’s ‘Real’ subset and testing its generalization across various domains (see Table3). Second, CIFAR-10-C, a corrupted version of CIFAR-10, helps assess model generalization and robustness (see Table4). Third, the STL-10 dataset, with significant label overlap with CIFAR-10, serves as a relevant OOD test case (seeAppendixB).

Bella models, with significantly better efficiency, demonstrate competitive performance against more resource-intensive implementations with Ensemble and SVGD baselines across both DomainNet and CIFA-10-C benchmarks. All of the Bayesian approximations, including our scalable and efficient method, outperform the Single model baseline.

Models | CIFAR-10-C | Average | |||||||

Contrast | GaussianBlur | GaussianNoise | ImpulseNoise | Pixelate | ShotNoise | Spatter | SpeckleNoise | ||

Single base | 91.78 | 90.58 | 62.96 | 66.48 | 77.94 | 71.56 | 92.14 | 73.87 | 78.53 |

VBL | 83.77 | 89.34 | 51.10 | 52.08 | 76.43 | 60.94 | 89.21 | 63.19 | 70.76 |

Ensemble Bella | 93.30 | 93.30 | 64.28 | 74.36 | 85.86 | 75.73 | 94.83 | 75.24 | 82.24 |

Ensemble base | 92.69 | 91.92 | 72.90 | 79.08 | 86.49 | 78.83 | 93.45 | 80.49 | 84.61 |

SVGD Bella | 94.69 | 94.05 | 67.89 | 75.86 | 88.70 | 76.49 | 94.85 | 77.61 | 83.77 |

SVGD base | 97.16 | 93.02 | 75.65 | 81.49 | 86.75 | 81.56 | 95.19 | 82.89 | 86.84 |

### 5.4 Comparing Uncertainty Estimations

Bayesian models capable of providing a theoretical basis for measuring model uncertainty. Also known as epistemic uncertainty, refers to uncertainty stemming from limitations in our knowledge or understanding of the underlying data generating process or the model itself. One of the ways to quantify model uncertainty is through mutual information estimates, following [47].

Mutual Information (MI).This is the mutual information between the output prediction and the posterior over model parameters $\boldsymbol{\theta}$, and can be used as a measure of epistemic (model) uncertainty.It can be expressed as:$\text{MI}(\boldsymbol{\theta},y\mid\mathcal{D},\mathbf{x})=H[p(y\mid\mathcal{D%},\mathbf{x})]-\mathbb{E}_{p(\boldsymbol{\theta}\mid\mathcal{D})}H[p(y\mid%\boldsymbol{\theta},\mathbf{x})]$If the parameters at input are well defined ( e.g., data seen during training), then we would gain little information from the obtaining label, or the MI measured will be low.

We employ MI to measure uncertainty to investigate whether the Bella approximations of the posterior leads to uncertainty estimates commensurate with those obtained from SVGD baselines. This provides empirical evidence of a functional equivalence of the Bella approximations of the posterior to that obtained from the current computationally intensive implementation of SVGD.

Datasets.We utilize the CIFAR-10-C task, featuring corrupted images, to examine the uncertainty of model predictions trained on the standard CIFAR-10 dataset. Additionally, we assess the uncertainty measures on the CAMELYON17 dataset, which is characterized by inherent dataset shifts within itself.

Results.Figure2 demonstrates the effectiveness of our approach to estimate uncertainty. Our Bella perform similarly to the SVGD base model, with a slightly better uncertainty on misclassified images of CAMELYON17 and corrupted CIFAR-10-C datasets (under brightness corruption with the maximum intensity), see details and other corruption types inAppendixC.

### 5.5 Robustness against Adversarial Examples

In this section, we examine the resilience of our proposed Bella against adversarial attacks, specifically employing the $L_{\infty}$ Fast Gradient Sign Method (FGSM) across various attack budgets as detailed inFigure3. This analysis aims to benchmark the robustness of our method in comparison to traditional models under adversarial conditions. We employ the robustness benchmark[48] to deploy the attack on CIFAR-10 test set and report results inFigure3.

The findings presented inFigure3 reveal that conventional models such as SVGD and Ensemble exhibit just slightly greater resistance to adversarial attacks. We attribute this enhanced robustness to the broader diversity in model parameters, which stems from their capacity to adjust the entire network’s parameters, unlike the Bella models.

Significantly, despite operating within the same computational constraints as a singular network model, our Bella demonstrates enhanced efficacy in mitigating adversarial attacks, thereby bolstering its robustness.

### 5.6 Ablation Studies

This section undertakes a series of ablation studies to examine the effects of various components within Bella.Our analysis includes an exploration of the training costs associated with different ranks and their consequent influence on model performance. Given that Bella incorporates multiple parameter particles, we also delve into how varying the number of these particles affects Bella’s efficacy.Additionally, we explore the application of low-rank adapters across different layers and assess their impact. Further details on other studies are inAppendixD. We show in Table6 that we achieve state-of-the-art performance on CAMELYON17.

Ablations on rank $r$.As outlined inSection4, we substitute the network’s extensive full matrix with low-rank matrices, which are defined by the ‘rank’ parameter ($r$). This section aims to assess how this parameter influences Bella’s performance.

To comprehensively demonstrate the rank’s impact, we perform our analysis on the demanding large-scale CAMELYON17 dataset. The findings, illustrated inFigure4(a), shed light on how the number of ranks affects performance. Utilizing a small rank significantly reduces the parameter space, but this constriction limits Bella’s ability to learn effectively and achieve optimal performance. Conversely, enhancing the rank to 4 markedly boosts Bella’s efficiency. It’s observed that performance plateaus at a rank of 16, indicating a saturation point.

Ablations on the number of particles $n$.In this section, we delve into the influence of the quantity of parameter particles on the performance of Bella. The findings, depicted inFigure4(b), reveal an improvement in Bella’s performance with an increase in the number of parameter particles.

This outcome is both intuitive and insightful, as a larger ensemble of parameter particles enhances the approximation of the Bayesian posterior more effectively.

Ablations on the number of trainable parameters.We will compare the number of trainable parameters between Bella and full SVGD in model performance. Critically, higher number of trainable parameters means higher cost to train. To show generalization, we also employ another large-scale challenging dataset Imagenet in this experiment.

The starting point of the plot inFigure5 associated with the CAMELYON17 dataset is for Bella with $r=2$ (consisting of $5$ particles), and the end point is associated with full SVGD (SVGD base) with the same number of particles. As $r$ increases, the accuracy of Bellas improves and after a while becomes plateau around $95.08$, comparable to the accuracy of full SVGD ($95.21$). This underscores the efficiency and advantages of our proposed Bella. Despite utilizing a substantially smaller pool of trainable parameters (around 0.3% for $r=4$) compared to full SVGD models, Bella demonstrates performance on par with these more parameter-intensive alternatives.

### 5.7 Generalization to a Visual Question Answer (VQA) Task

In this section, we extend the application of our Bella to another challenging vision task, Visual Question Answering (VQA), as detailed in[49]. We leverage the state-of-the-art, pre-trained, large multi-modal model LlaVA [13, 50] for this purpose. Utilizing LlaVA transforms VQA into a process where an image and a natural-language question are inputs, and the model generates a free-form, open-ended text answer. Answering questions in VQA requires various intelligent capabilities, including not only image recognition but also complex reasoning. For this task, we employVQA v2[49] dataset containing 204,721 images, more than 1 Billion (1B) questions and 10B ground-truth answers in total.There are three main types of answers: Yes/No, a Number, and Other.

Model.In our experiments, we employed our proposed Bella on top of LlaVA-1.5-7B[13, 50]. Further details about the dataset, model and metrics are deferred to AppendixF.

Results (Performance).The primary outcomes for Yes/No and Number answer queries are detailed in Table5, respectively. We chose these question types to ensure a fair comparison and avoid semantic mismatches in open-ended answers in Others.A distinctive feature of our Bella is its reduced uncertainty estimates to accurate predictions, coupled with increased uncertainty estimates to incorrect ones. Moreover, it surpasses the Single base model in terms of Accuracy and Exact Match metrics.

Notably, our method is efficient, particularly in contrast to the resource-intesive baseline Bayesian models tailored for this task (*e.g*. [51, 52, 6]). The computational requirements for using these baselines render the application of full SVGD or ensemble alternatives impractical, thereby highlighting the impact on practical applications by harnessing the effectiveness of Bayesian models with Bella.

VQA: Evaluation of Yes/No Questions | ||||

Models | Entropy of Corrects ($\downarrow$) | Entropy of Incorrect ($\uparrow$) | Accuracy ($\uparrow$) | Exact Match ($\uparrow$) |

Single base | 0.3336 | 0.5920 | 91.59 | 86.74 |

Ensemble Bella | 0.3438 | 0.5935 | 91.20 | 86.21 |

SVGD Bella | 0.3245 | 0.5950 | 92.46 | 87.83 |

VQA: Evaluation of Number Questions | ||||

Single base | 0.4148 | 0.9911 | 58.69 | 49.26 |

Ensemble Bella | 0.4248 | 0.9929 | 57.86 | 48.68 |

SVGD Bella | 0.4059 | 0.9816 | 60.19 | 50.99 |

Results (Model Uncertainty and Human Confidence).Further, we investigate the relationship between model uncertainty attained from Bella and human confidence, facilitated by multiple annotators in the VQA dataset. For this, we measure the correlation between the model disagreement by measuring the predictive entropy vs the human annotations of the answers.

In Fig. 6 we shows the correlation between the entropy of the model outputs and human confidence levels. By setting a specific threshold for the model’s entropy, we effectively bifurcate our predictions into two distinct categories: those with lower entropy fall into the ‘Low Entropy’ group, signalling reduced uncertainty within the model’s assessments, while predictions surpassing this entropy threshold are allocated to the ‘High Entropy’ segment, indicative of greater uncertainty. Intriguingly, our observations reveal a consistent negative correlation between the model’s entropy levels and human confidence across different types of queries, such as Yes/No and Number questions. This pattern suggests that the model’s entropy can serve as a reliable gauge of certainty, mirroring human judgment in its response to varying levels of uncertainty.

## 6 Conclusion

In this paper, we introduce an innovative approach for creating an efficient Bayesian Neural Network (BNN) approximation using only a base pre-trained model. Our approach, called Bella, demonstrates remarkable compatibility with full-rank BNN approximations like SVGD or Ensemble methods, outperforming single-network solutions across a range of tasks, from pure classification to out-of-distribution (OOD) generalization and uncertainty quantification. Our research paves the way for effective BNN implementation, facilitating the development of reliable and robust machine-learning models capable of scaling to large networks and datasets. Bella reinforces the effectiveness of employing such a simple, efficient and effective training method with diverse representations over the conventional single fine-tuning techniques prevalent in the field.

## References

- [1]RadfordM Neal.Bayesian learning for neural networks, volume 118.Springer Science & Business Media, 2012.
- [2]Xuanqing Liu, Yao Li, WuChongruo, and Hsieh Cho-Jui.ADV-BNN: Improved Adversarial Defense Through Robust Bayesian Neural Network.In International Conference on Learning Representations (ICLR), 2019.
- [3]BaoGia Doan, EhsanM Abbasnejad, JavenQinfeng Shi, and DamithC Ranasinghe.Bayesian learning with information gain provably bounds risk for a robust adversarial defense.In International Conference on Machine Learning (ICML), 2022.
- [4]Nanyang Ye and Zhanxing Zhu.Bayesian adversarial learning.In Advances in Neural Information Processing Systems (NeurIPS), 2018.
- [5]AndrewGordon Wilson, Pavel Izmailov, MatthewD. Hoffman, Yarin Gal, Yingzhen Li, MelanieF. Pradier, Sharad Vikram, Andrew Foong, Sanae Lotfi, and Sebastian Farquhar.Evaluating approximate inference in bayesian deep learning.In Proceedings of the NeurIPS 2021 Competitions and Demonstrations Track.
- [6]Ehsan Abbasnejad, Iman Abbasnejad, QiWu, Javen Shi, and Anton vanden Hengel.Gold seeker: Information gain from policy distributions for goal-oriented vision-and-langauge reasoning.In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
- [7]ChristopherM. Bishop.Pattern Recognition and Machine Learning.Springer, 2006.
- [8]Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li.SegCLIP: Patch aggregation with learnable centers for open-vocabulary semantic segmentation.In Proceedings of the 40th International Conference on Machine Learning (ICML), 2023.
- [9]Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa.Git re-basin: Merging models modulo permutation symmetries.In The Eleventh International Conference on Learning Representations, 2023.
- [10]Almog Gueta, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen.Knowledge is a region in weight space for fine-tuned language models.arXiv preprint arXiv:2302.04863, 2023.
- [11]Grand Challenge.Camelyon17 leaderboard.https://camelyon17.grand-challenge.org/evaluation/challenge/leaderboard/, 2024.Accessed: 08-Mar-2024.
- [12]PangWei Koh, Shiori Sagawa, Henrik Marklund, SangMichael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, RichardLanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, BertonA. Earnshaw, ImranS. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang.WILDS: A benchmark of in-the-wild distribution shifts.In International Conference on Machine Learning (ICML), 2021.
- [13]Haotian Liu, Chunyuan Li, Qingyang Wu, and YongJae Lee.Visual instruction tuning.In Advanced in Neural Information Processing Systems (NeurIPS), 2023.
- [14]Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin deLaroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.Parameter-Efficient Transfer Learning for NLP.arXiv:1902.00751, 2019.
- [15]Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi.Learning multiple visual domains with residual adapters.arXiv:1705.08045, 2017.
- [16]Zhaojiang Lin, Andrea Madotto, and Pascale Fung.Exploring versatile generative language model via parameter-efficient transfer learning.In Findings of the Association for Computational Linguistics (EMNLP), 2020.
- [17]EdwardJ Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, LuWang, and Weizhu Chen.LoRA: Low-rank adaptation of large language models.In International Conference on Learning Representations (ICLR), 2022.
- [18]Xinjie Fan, Shujian Zhang, BoChen, and Mingyuan Zhou.Bayesian attention modules.In Advances in Neural Information Processing Systems (NeurIPS), 2020.
- [19]Shujian Zhang, Xinjie Fan, BoChen, and Mingyuan Zhou.Bayesian attention belief networks.In International Conference on Machine Learning (ICML), 2021.
- [20]AdamX Yang, Maxime Robeyns, XiWang, and Laurence Aitchison.Bayesian low-rank adaptation for large language models.In International Conference on Learning Representations (ICLR), 2024.
- [21]Pavel Izmailov, Sharad Vikram, MatthewD Hoffman, and AndrewGordon Wilson.What are bayesian neural network posteriors really like?In International Conference on Machine Learning (ICML), 2021.
- [22]Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.Simple and scalable predictive uncertainty estimation using deep ensembles.In Advances in neural information processing systems (NeurIPS), 2017.
- [23]XiWang, Laurence Aitchison, and Maja Rudolph.LoRA ensembles for large language model fine-tuning, 2024.
- [24]VeitDavid Wild, Sahra Ghalebikesabi, Dino Sejdinovic, and Jeremias Knoblauch.A Rigorous Link between Deep Ensembles and (Variational) Bayesian Methods.In Advances in Neural Information Processing Systems (NeurIPS), 2023.
- [25]Qiang Liu and Dilin Wang.Stein variational gradient descent: A general purpose bayesian inference algorithm.Advances in Neural Information Processing Systems (NeurIPS), 2016.
- [26]Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Hai-Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, and Maosong Sun.Parameter-efficient fine-tuning of large-scale pre-trained language models.Nature Machine Intelligence, 2023.
- [27]Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, etal.Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models.arXiv preprint arXiv:2203.06904, 2022.
- [28]Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.Qlora: Efficient finetuning of quantized llms.arXiv preprint arXiv:2305.14314, 2023.
- [29]RadfordM Neal etal.Mcmc using hamiltonian dynamics.Handbook of markov chain monte carlo, 2(11), 2011.
- [30]David MacKay.Bayesian model comparison and backprop nets.In Advances in Neural Information Processing Systems (NeurIPS), 1991.
- [31]Max Welling and YeeW Teh.Bayesian learning via stochastic gradient langevin dynamics.In Proceedings of the 28th international conference on machine learning (ICML-11), pages 681–688. Citeseer, 2011.
- [32]DavidM Blei, Alp Kucukelbir, and JonD McAuliffe.Variational inference: A review for statisticians.Journal of the American statistical Association, 2017.
- [33]AndrewG Wilson and Pavel Izmailov.Bayesian deep learning and a probabilistic perspective of generalization.In Advances in neural information processing systems (NeurIPS), 2020.
- [34]AndrewGordon Wilson and Pavel Izmailov.Deep ensembles as approximate bayesian inference.
- [35]Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, etal.Imagenet large scale visual recognition challenge.International journal of computer vision, 2015.
- [36]Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra.Weight uncertainty in neural network.In International Conference on Machine Learning (ICML), 2015.
- [37]Alex Krizhevsky, Geoffrey Hinton, etal.Learning multiple layers of features from tiny images.2009.
- [38]Dan Hendrycks and Thomas Dietterich.Benchmarking neural network robustness to common corruptions and perturbations.arXiv preprint arXiv:1903.12261, 2019.
- [39]Adam Coates, Andrew Ng, and Honglak Lee.An Analysis of Single Layer Networks in Unsupervised Feature Learning.In International Conference on Artificial Intelligence and Statistics (AISTATS), 2011.
- [40]Peter Bandi, Oscar Geessink, Quirine Manson, Marcory VanDijk, Maschenka Balkenhol, Meyke Hermsen, BabakEhteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, etal.From detection of individual metastases to classification of lymph node status at the patient level: the camelyon17 challenge.IEEE Transactions on Medical Imaging, 2018.
- [41]Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and BoWang.Moment matching for multi-source domain adaptation.In Proceedings of the IEEE/CVF international conference on computer vision (CVPR), 2019.
- [42]Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh.Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering.In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
- [43]Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt.Openclip, 2021.
- [44]Alec Radford, JongWook Kim, Chris Hallacy, A.Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.Learning transferable visual models from natural language supervision.In International Conference on Machine Learning (ICML), 2021.
- [45]Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard.Task arithmetic in the tangent space: Improved editing of pre-trained models.arXiv preprint arXiv:2305.12827, 2023.
- [46]Raphael Gontijo-Lopes, Yann Dauphin, and EkinD Cubuk.No one representation to rule them all: Overlapping features of training methods.arXiv preprint arXiv:2110.12899, 2021.
- [47]Yarin Gal etal.Uncertainty in deep learning.2016.
- [48]Nicolas Papernot, Fartash fa*ghri, Nicholas Carlini, Ian Goodfellow, Reuben Feinman, Alexey Kurakin, Cihang Xie, Yash Sharma, Tom Brown, Aurko Roy, Alexander Matyasko, Vahid Behzadan, Karen Hambardzumyan, Zhishuai Zhang, Yi-Lin Juang, Zhi Li, Ryan Sheatsley, Abhibhav Garg, Jonathan Uesato, Willi Gierke, Yinpeng Dong, David Berthelot, Paul Hendricks, Jonas Rauber, and Rujun Long.Technical report on the cleverhans v2.1.0 adversarial examples library.arXiv preprint arXiv:1610.00768, 2018.
- [49]Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C.Lawrence Zitnick, and Devi Parikh.VQA: Visual Question Answering.In International Conference on Computer Vision (ICCV), 2015.
- [50]Haotian Liu, Chunyuan Li, Yuheng Li, and YongJae Lee.Improved baselines with visual instruction tuning.arXiv preprint arXiv:2310.03744, 2023.
- [51]M.Ehsan Abbasnejad, Qinfeng Shi, Anton vanden Hengel, and Lingqiao Liu.A generative adversarial density estimator.In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 10782–10791. Computer Vision Foundation / IEEE, 2019.
- [52]M.Ehsan Abbasnejad, Javen Shi, Anton vanden Hengel, and Lingqiao Liu.GADE: A generative adversarial approach to density estimation and its applications.Int. J. Comput. Vis., 128(10):2731–2743, 2020.
- [53]MichaelS Matena and ColinA Raffel.Merging models with fisher-weighted averaging.Advances in Neural Information Processing Systems (NeurIPS), 2022.
- [54]Mitchell Wortsman, Gabriel Ilharco, SamirYa Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, AriS Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, etal.Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.In International Conference on Machine Learning (ICML), 2022.
- [55]Mitchell Wortsman, Gabriel Ilharco, JongWook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, RaphaelGontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, etal.Robust fine-tuning of zero-shot models.In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7959–7971, 2022.
- [56]Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord.Diverse weight averaging for out-of-distribution generalization.Advances in Neural Information Processing Systems (NeurIPS), 2022.

Bayesian Low-Rank LeArning (Bella):

A Practical Approach to Bayesian Neural Networks

## Overview of Materials in the Appendices

We provide a brief overview of the extensive set of additional experimental results and findings in the Appendices that follows.

- 1.
Diversity measures between SVGD Bella and its SVGD base model (AppendixA)

- 2.
The generalization of CIFAR-10 pre-trained models on OOD dataset (STL-10) (AppendixB)

- 3.
Detailed uncertainty measured on corrupted CIFAR-10-C dataset (AppendixC)

- 4.
Additional study on weight averaging (AppendixD)

- 5.
Impact of the pushing parameter on the robustness of models (AppendixE)

- 6.
Detailed information about VQA task (AppendixF)

- 7.
Detailed information regarding RBF kernel computation in SVGD (AppendixG)

- 8.
Detailed information regarding datasets and hyper-parameters utilized in the paper (AppendixH)

## Appendix A Diversity Measures

This section assesses the diversity of the trained Bella, comparing it to the foundational SVGD model. The goal is to determine if Bella can maintain the diversity level of the SVGD base model while utilizing significantly fewer parameters.

Given the lack of a conventional metric for evaluating the diversity among parameter particles, we suggest employing the Kullback–Leibler (KL) Divergence. This measure compares the expected parameters of a Bayesian model to the softmax output for each parameter particle, serving as an indicator of model diversity. We calculate this divergence across 10,000 test samples from the CIFAR-10 dataset. In particular,

$\text{Diversity}=\frac{1}{N}\sum_{i=1}^{N}{KL\Big{[}\mathbb{E}_{\boldsymbol{%\theta}}[p(y\mid\mathbf{x}_{i},\boldsymbol{\theta})],p(y\mid\mathbf{x}_{i},%\boldsymbol{\theta})\Big{]}}$ |

where KL is the Kullback–Leibler divergence, N is the number of samples.

Results.Figure7 presents intriguing findings, where the SVGD Bella exhibits comparable and even greater diversity compared to the base SVGD model. We speculate that this increased diversity may stem from the Bella’s use of significantly fewer trainable parameters, reducing the likelihood of overfitting and facilitating the identification of parameter particles with enhanced diversity, all while preserving model performance. These results further underscore the efficiency and effectiveness of our method in implementing a Bayesian neural network with substantial diversity.

## Appendix B Generalization of CIFAR-10 Models on OOD dataset (STL-10)

In this section, we evaluate the generalization of networks trained on CIFAR-10 on a similar datasets, which is STL-10.

STL-10 dataset.The STL-10 dataset is a benchmark for image recognition algorithms, containing 5,000 labeled training images, 8,000 labeled test images across 10 classes, and 100,000 unlabeled images for unsupervised learning. Images are 96x96 pixels, larger than those in similar datasets like CIFAR-10, facilitating more detailed models and promoting unsupervised learning techniques.

Experimental Set-up.We choose eight labels in STL-10 test set, same labels observed in CIFAR-10 dataset to set a OOD dataset. This includes 6,400 images acting as the OOD dataset for CIFAR-10 pre-trained models.

Results.The data presented in Figure8 indicate that our approach slightly outperforms the SVGD and Ensemble baseline models, and significantly surpasses the capabilities of a singular network. We propose that the incorporation of a low-rank adaptation mechanism contributes to reducing overfitting, thereby enhancing the model’s generalization ability on out-of-distribution (OOD) datasets. This underscores the efficacy of our method in fostering robust generalization across OOD datasets.

## Appendix C Measure Uncertainty on Corrupted CIFAR-10-C Dataset

In this section, we apply Mutual Information (MI) as a metric for assessing uncertainty. This approach enables us to compare the performance of our Bella method with models based on Stein Variational Gradient Descent (SVGD) in handling Out-of-Distribution (OOD) tasks, focusing on the CIFAR-10-C dataset.We adhere to the corruption types listed in Table4 for this analysis.

Results.The outcomes, depicted in Figure9, affirm the efficacy of Bella. It demonstrates comparable, and in some instances, superior performance to SVGD-based models, especially notable in most cases such as involving contrast adjustments, Gaussian blur, and pixelation. This indicates a robust ability to detect increased uncertainty in OOD datasets, which is a valuable attribute for enhancing the reliability of machine learning models in unpredictable environments.

## Appendix D Additional Studies

In this section, we conduct further studies on the benefits of our approach utilizing recently developed weight averaging concept.

Recent research has increasingly explored the concept of weight averaging, a technique where parameters from multiple models are combined to create a unified model that may offer superior predictive performance[53, 54, 55]. This approach is grounded in the theory that models starting from the same pre-trained state tend to have a linear loss landscape, making averaging a viable strategy. For a detailed mathematical discussion, see[56]. Corroborating this,Figure10 features density plots andTable7 that illustrate the predictive entropy for samples that are correctly and incorrectly classified by the CIFAR-10 and CIFAR-100 datasets using the ‘soup’ models. These results show that the performance of Bella models, post-averaging, is comparable to that of more computationally intensive baseline models. This suggests that Bella models, despite having fewer trainable parameters, are capable of achieving the same level of performance as their more complex counterparts.

Layer name | TrainableParams | Accuracy | |

FC | Proj. | ||

0-11 | 0-11 | 1.85M | 94.89 |

0-11 | – | 925K | 95.11 |

– | 0-11 | 925K | 95.83 |

0-5 | 0-5 | 925K | 92.7 |

6-11 | 6-11 | 925K | 91.39 |

Ablations on fine-tuned layers.In our paper, we propose the concept of substituting the network’s complete weight matrices with low-rank adapters to approximate BNNs. This section is dedicated to assessing the effectiveness of these low-rank adapters across various layer configurations within CLIP, utilizing CAMELYON17. It should be noted that CLIP consists of 12 ResidualAttentionBlocks, each comprising Fully Connected (FC) and Projection (Proj.) linear layers, which we aim to modify for our Bella. The data, presented inTable6, reveals interesting performance of our Bella as it surpasses the SOTA in one of the experiments. This highlights the adaptability of our proposed approach, which does not necessitate extensive fine-tuning to select specific layers for optimal performance.

Model Name | CIFAR-10 | CIFAR-100 | ||||

Accuracy | PECorrect | PEIncorrect | Accuracy | PECorrect | PEIncorrect | |

Ensemble Bella soup | 95.74 | 0.0597 | 0.6063 | 80.82 | 0.3461 | 1.2261 |

Ensemble base soup | 97.42 | 0.0986 | 0.6917 | 86.42 | 0.3025 | 1.1148 |

SVGD Bella soup | 96.32 | 0.0771 | 0.6988 | 83.59 | 0.2935 | 1.1044 |

SVGD base soup | 97.62 | 0.0438 | 0.5528 | 86.73 | 0.2609 | 1.0426 |

Ensemble base | 97.54 | 0.1125 | 0.5947 | 86.89 | 0.3738 | 1.0677 |

SVGD base | 97.87 | 0.0572 | 0.4692 | 87.32 | 0.3256 | 1.0015 |

Ensemble Bella | 97.58 | 0.0685 | 0.5098 | 86.50 | 0.4100 | 1.1218 |

SVGD Bella | 97.95 | 0.0603 | 0.5193 | 88.09 | 0.4000 | 1.0915 |

Comparison of SVGD Bella and VBL.

In this section, we compare the performance of SVGD Bella and VBL in distinguishing between correctly and misclassified samples. Specifically, we assess their ability to assign high entropy to misclassified samples and low entropy to correctly classified ones. Figure11 shows that SVGD Bella outperforms VBL in this regard.

## Appendix E Impact of $\gamma$ values on Robustness.

This section assesses how the repulsive force parameter, denoted as $\gamma$, influences model robustness. Experiments were carried out on the CIFAR-10 dataset using SVGD Bella with a configuration of $r=16,n=5$, while varying $\gamma$ values. As illustrated inTable8, adjustments in the repulsive force significantly affect robustness. Increasing $\gamma$ enhances robustness by promoting greater diversity among parameter particles. However, prioritizing diversity to an excessive degree (i.e., $\gamma$ values above 0.05) does not further enhance robustness and may even counteract improvements, although performance remains superior to scenarios with minimal repulsive force ($\gamma=0.01$).

Attack Budgets | ||||

PushingParameter | 0.001 | 0.005 | 0.01 | 0.03 |

$\gamma=0.01$ | 53.7 | 50.4 | 48.1 | 40.9 |

$\gamma=0.025$ | 54.9 | 51.3 | 49.2 | 41.8 |

$\gamma=0.05$ | 56.1 | 53.7 | 51.2 | 42.5 |

$\gamma=0.1$ | 54.4 | 51.9 | 48.1 | 42 |

$\gamma=0.2$ | 55.0 | 50.6 | 47.3 | 40.7 |

$\gamma=0.3$ | 54.2 | 52.6 | 49.6 | 42.5 |

## Appendix F Detailed information regarding VQA task

In this section, we provide more details about the experiments we conducted on the VQA task in Section5.7.

### F.1 Task Definition and Experiment Setup

Task.Visual Question Answering (VQA) [49] is a free-form and open-ended task, taking as input an image and a natural-language question about the image and producing a natural-language answer as the output.Questions in VQA require various intelligence capabilities to answer, including image recognition and object detection, as well as reasoning like commonsense reasoning.Below is an overview of the VQA dataset utilized in our paper:

- •
VQA v2 [42]: This dataset contains 204,721 images, more than 1 Billion (1B) questions, and 10B ground truth answers in total.There are three main types of answers: Yes/No, Number, and Other. The evaluation set contains 80,541, 28,134 and 105,679 questions for Yes/No, Number, and Other respectively.

As open-ended questions may result in a diverse set of possible answers, VQA gather 10 human annotations for each question as the ground-truth answers.These answers can be different from each other, and even incorrect.

Models.In our experiments, we applied Bella on top of LLaVA [13, 50] to address the VQA task.Specifically, we utilized the LLaVA-1.5 7B model, an improved version of the original LLaVA with superficial modifications.We followed the public training method using a deepspeed codebase.

However, manipulating parameter gradients is non-trivial, and there is no public way of doing this using deepspeed.Therefore, we simulated the low-rank approximation from the same initialization by conducting several end-to-end fine-tuning settings with different random seeds, learning rates, and gradient accumulation steps.Together with the official public model, based on our Bella, there are four generated variants we call Bella-0, Bella-1, Bella-2, Bella-3 respectively.

Metrics.Below are a detailed metrics used during our evaluations.

- •
Accuracy: In VQA dataset, there are 10 human annotations for each question. The model prediction accuracy is calculated by

$Accuracy({\color[rgb]{1,0,0}ans})=\min\left\{\frac{\#humans\ that\ said\ {%\color[rgb]{1,0,0}ans}}{3},1\right\}.$ (5) In order to be consistent with human accuracy, this metric is averaged over all 10 chosen 9 sets of human annotations.

- •
Exact Match: We defined the metric as follows:

$EM=\begin{cases}1,&\text{if $Accuracy=1$}.\\0,&\text{otherwise}.\end{cases}$ (6) When $Accuracy$ equals 1, it means the predicted answer is 100%, same with the ground-truth annotation, i.e., Exact Match with one another.

To measure the model uncertainty along with the human confidence in Figure6, we define Entropy and Human Confidence as follows.

- •
Entropy: Given a single question, we can calculate the entropy of the model prediction after applying $softmax$ over the logits of LLaVA:

$Entropy=\frac{1}{N}\sum_{i}^{N}\sum_{j}^{V}{-p_{ij}\times\ln(p_{ij})}$ (7) Note that $N$ is the output sequence length, $V$ is the vocabulary size, and $p_{ij}$ is the output of softmax function.We expect the entropy to be lower for correct predictions, as it stands for lower uncertainty.We expect the entropy to be higher for incorrect predictions, i.e., more uncertain on the prediction.

- •
Human Confidence (HC): We calculate the confidence of 10 human annotators as follows:

$HC=\frac{1}{10}\max\{\#humans\ that\ said\ {ans}\}$ (8) More annotators agree with the same answer, higher HC will be.

### F.2 Experimental Results

Correlation.Given a question, our model generates a prediction accompanied by logits. We calculate the Entropy using the logits and assess Human Confidence by leveraging 10 ground-truth responses for this prediction. After compiling all predictions, we introduce a specific $threshold$, segregating them into two distinct categories: those not meeting the $threshold$ are classified under ‘Low Entropy’, while those exceeding it are allocated to the ‘High Entropy’ segment. In Figure6, the $threshold$ values are set within the range {0.1, 0.2, …, 0.8}. For each threshold level, we compute the average Human Confidence for both ‘Low Entropy’ and ‘High Entropy’ segments, and these averages are plotted as distinct curves.

Similarity.Initially, we computed the pairwise distances among the trained Bella models. We employed cosine similarity, which varies between 0 and 1, as our metric. As shown in Table9, the choice of training strategies influences the degree of similarity among model parameters. Notably, Bella-0 exhibits the greatest dissimilarity when compared to the rest, whereas Bella-2 and Bella-3 display the highest similarity.

Bella-0 | Bella-1 | Bella-2 | Bella-3 | |

Bella-0 | 1 | 0.0286 | 0.0327 | 0.0317 |

Bella-1 | 0.0286 | 1 | 0.6289 | 0.5508 |

Bella-2 | 0.0327 | 0.6289 | 1 | 0.8047 |

Bella-3 | 0.0317 | 0.5508 | 0.8047 | 1 |

Results.The performance of different single models and ensembles are presented in Table5, Table10.We ensembled two Bella models by simply averaging two softmax output scores.Given the model similarity matrix in Table9, some observations and conclusions can be made from these tables.

- •
Across all models, the averaged entropy of correct predictions is lower than the incorrect predictions. It shows models usually have higher certainty in correct predictions but lack confidence in wrong predictions.

- •
To some extent, entropy can be used as an indicator of the model’s accuracy. For example, models with high accuracy often have lower entropy for correct predictions and higher entropy for incorrect predictions.

- •
We find that an ensemble of two dissimilar models, e.g., SVGD Bella (Bella-0, Bella-1) (parameters are away from each other), usually ends with higher accuracy and exact match scores, as well as lower entropy over correct predictions. While two similar models ensemble ends with lower accuracy and exact match scores, e.g., Ensemble Bella (Bella-2, Bella-3).

- •
For other questions in VQA, the EM score of the SVGD Bella is slightly lower than the Single base. As models would generate longer textual answers for those questions, it is non-trivial to investigate a better ensemble strategy. In Figure13, we present some incorrect predictions yet reasonable generation by SVGD Bella, where human annotators show high uncertainty as well.

Models | Correct Entropy | Incorrect Entropy | Accuracy | Exact Match |

Single base | 0.5782 | 1.4826 | 70.00 | 57.17 |

Ensemble Bella | 0.6079 | 1.5457 | 67.05 | 55.26 |

SVGD Bella | 0.5359 | 1.4602 | 68.38 | 56.68 |

In addition, we show the evaluation of uncertainty using Mutual Information in Figure12 and provide case studies in Figure13 and Figure14 to show predicated answers from different models.The Figure12 show the efficacy of Bella.It enables large-scale networks like LLaVA to predict outcomes with uncertainty for both VQA ‘Yes/No’ and ‘Number’ questions, a capability that a single base model lacks.

## Appendix G RBF kernel computation in SVGD

We wish to evaluate RBF kernel $k(\mathbf{B}_{i}\mathbf{A}_{i},\mathbf{B}_{j}\mathbf{A}_{j})$ where$\mathbf{B}_{i}\in\mathbb{R}^{d_{1}\times r}$,$\mathbf{A}_{i}\in\mathbb{R}^{r\times d_{2}}$:

$\displaystyle k(\mathbf{B}_{i}\mathbf{A}_{i},\mathbf{B}_{j}\mathbf{A}_{j})$ | $\displaystyle=\exp\left(-\frac{1}{2\sigma^{2}}\|\mathbf{B}_{i}\mathbf{A}_{i}-%\mathbf{B}_{j}\mathbf{A}_{j}\|_{F}^{2}\right)$ | ||

$\displaystyle=\exp\left(-\frac{1}{2\sigma^{2}}\mathrm{Tr}[(\mathbf{B}_{i}%\mathbf{A}_{i}-\mathbf{B}_{j}\mathbf{A}_{j})^{\top}(\mathbf{B}_{i}\mathbf{A}_{%i}-\mathbf{B}_{j}\mathbf{A}_{j})]\right).$ |

The trace term can be written as

$\displaystyle\mathrm{Tr}[\mathbf{A}_{i}\mathbf{A}_{i}^{\top}\mathbf{B}_{i}^{%\top}\mathbf{B}_{i}]+\mathrm{Tr}[\mathbf{A}_{i}\mathbf{A}_{i}^{\top}\mathbf{B}%_{i}^{\top}\mathbf{B}_{i}]-\mathrm{Tr}[\mathbf{A}_{j}\mathbf{A}_{i}^{\top}%\mathbf{B}_{i}^{\top}\mathbf{B}_{j}]-\mathrm{Tr}[\mathbf{A}_{i}\mathbf{A}_{j}^%{\top}\mathbf{B}_{j}^{\top}\mathbf{B}_{i}].$ |

To compute $\mathbf{A}_{j}\mathbf{A}_{i}^{\top}$ and $\mathbf{B}_{j}^{\top}\mathbf{B}_{i}$ the cost is$O(r^{2}d_{2})$ and $O(r^{2}d_{1})$ respectively. And each trace term is ofthe form$\mathrm{Tr}(\mathbf{U}^{\top}\mathbf{V})=\langle\mathbf{U},\mathbf{V}\rangle_{%F}=\mathrm{vec}(\mathbf{U})^{\top}\mathrm{vec}(\mathbf{V})$which is an $O(r^{2})$ operation and requires no further matrixmultiplication.

## Appendix H Detailed Parameters and Datasets

In this section, we mention in detail the description for each of the dataset utilized in our experiment as below:

- •
CIFAR-10: This dataset comprises $60,000$ $32\times 32$ color images divided into 10 distinct classes, with each class containing 6,000 images. It is partitioned into 50,000 training images and $10,000$ test images. The classes cover a range of subjects from animals to vehicles, providing a fundamental challenge in image classification.

- •
CIFAR-10-C: This dataset is an extension of the CIFAR-10 dataset, designed to evaluate the robustness of machine learning models against common image corruptions. It contains the same 60,000 images as CIFAR-10, however, the images in CIFAR-10-C have been systematically altered using a range of corruption techniques, including noise, blur, weather, and digital effects, resulting in 19 different corruption types each at 5 severity levels. This dataset is used to test the performance of models in recognizing objects under various real-world conditions, making it a valuable tool for improving the reliability and robustness of image recognition systems.

- •
STL-10: This dataset is a benchmark for evaluating image recognition algorithms, featuring 13,000 color images. This dataset is divided into 5,000 training images and 8,000 test images, distributed across 10 different classes that include a variety of objects such as animals and vehicles. Each image in the dataset is $96\times 96$ pixels, offering higher resolution than many similar datasets such as CIFAR-10. The STL-10 dataset is tailored for supervised learning tasks in image recognition, providing a structured framework for developing and testing algorithms’ ability to classify images into predefined categories.

- •
CIFAR-100: Similar to CIFAR-10, the CIFAR-100 dataset is composed of $100$ classes, each with $600$ images, offering a more detailed classification challenge compared to CIFAR-10.

- •
CAMELYON17: The CAMELYON17 dataset is utilized in a domain generalization context, where the domains are represented by different hospitals. The primary objective is to develop models capable of generalizing to data from hospitals not included in the training set. Focusing on binary classification, the dataset comprises $96\times 96$ histopathological images as input, with the task to identify the presence of tumor tissue in the central $32\times 32$ region, indicated by a binary label.

- •
ImageNet (ILSVRC2012): which is a subset of the ImageNet dataset specifically used forthe ImageNet Large Scale Visual Recognition Challenge in $2012$, contains over $1.2$ million imagesdistributed across $1,000$ different classes.

- •
DomainNet: DomainNet is one of the largest and most diverse datasets available for domain adaptation studies. It contains approximately $600,000$ images across $345$ categories, spanning six distinct visual domains (Real, Clip-Art, Infograph, Paint, Sketch, Quick). This diversity in domains and categories enables the dataset to simulate real-world scenarios where models must adapt to different visual representations and styles.

Hyper-Parameters.

Detailed information about the hyper-parameters used can be found inTable11.

Name | Value | Notes |

$r$ | ImageNet:16, Others:4 | Rank values |

$\gamma$ | 0.01 | Weight to control the repulsive force |

$n$ | 5 | #Parameter particles |

$optimizer$ | AdamW | With adaptive scheduler |