Developing robust models for medical imaging that effectively generalize across diverse image characteristics remains a significant challenge, primarily due to the limited availability of annotated data. Vision Transformer models trained with the use of self-supervised learning, knowledge distillation, and contrastive learning have demonstrated a strong capability to capture diverse features in images, rendering them effective in providing meaningful feature representations. Nevertheless, these approaches necessitate large-scale training data to provide semantically rich feature representations otherwise, the training process is susceptible to supervision collapse. This challenge is prominent in the medical imaging domain, where the available annotated data are limited. It becomes even more crucial in few-shot learning scenarios when the model is tasked with adapting to an unseen domain by training on a few data of a different domain. To address the aforementioned issues, we implement a supervised knowledge distillation loss adaptation stage designed to leverage knowledge information between the patch tokens of other samples. Experiments in skin lesion classification datasets demonstrate that our proposed method outperformed other commonly used self-supervised training techniques.