Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspective

Tian Liu1, Anwesha Basu1, James Caverlee1, Shu Kong2
1Texas A&M University
  2University of Macau
Paper Code arXiv Poster
5-min Talk Slides

ECCV 2026

tl;dr: We diagnose the failures of finetuning VLMs for SSL,
and propose simple remedies via temperatures.

Motivation

Semi-supervised few-shot learning (SSFSL) resembles realworld “auto-annotation” application, which aims to learn a model over a few labeled and abundant unlabeled task-specific examples to automatically annotate the unlabeled ones. Existing SSL methods have mostly focused on finetuning ImageNet-pretrained backbones or prompts a frozen VLM, which underperforms recent few-shot learning methods that finetune the VLM and retrieve task-relevant open data. To bridge the gap, we explore these techniques for SSFSL.

Insights

1

Over five challenging fine-grained SSL datasets, our experiments show that:

  1. Influential Semi-Supervised Learning (SSL) methods, such as FixMatch, DebiasPL (finetuned on ImageNet-pretrained ResNet50) underperform the recent zero-shot learning (ZSL) methods like REAL.
  2. SOTA SSL method FineSSL that prompts a frozen VLM (e.g., CLIP) achieves notable gains, yet still lags behind recent few-shot learning methods like Few-Shot Finetune and SWAT that finetune the VLM's visual encoder and retrieves open data, motivating us to explore these techniques for SSFSL.
  3. Directly finetuning VLM with FixMatch and DebiasPL significantly underperforms Few-Shot Finetune, which do not even exploit unlabeled data. We reveal that the root cause of such failures lies in the flat softmax probabilities from VLMs, leading to weak training supervision and zero utilization of unlabeled data.
  4. Our SSFSL method SWIFT, incorporates our simple temperatures to remedy the above issues, exploiting retrieved open data to achieve SOTA performance, even rivalling fully supervised scenario.

Challenges

VLM produces ''flat'' softmax probabilities

1

We run OpenCLIP ViT-B/32 on the unlabeled images of the semi-Aves dataset by zero-shot prompting its 200 class names.

  • (A) VLMs produce a ''flat'' distribution of softmax probabilities (200-dim column vector per unlabeled example), resulting in low confidence scores and weak supervision signals that prevent effective finetuning. (B) Applying a temperature sharpens the softmax distribution, illustrated by the more prominent red diagonal line.
  • (C) Without temperature, the original low confidence scores result in zero utilization of unlabeled examples for FixMatch (threshold set at 0.8)! (D) Using a temperature increases confidence scores, improving the utilization of unlabeled data. Note the difference in confidence range between (C) and (D).

Solution

FixMatch with Temperatures

1

We apply temperatures for (1) sharpening the softmax probability distributions and (2) strengthening supervision signals. We illustrate our temperatures with FixMatch as an example. Specifically, we introduce two temperatures: (a) loss temperature T_loss to sharpen the softmax probabilities when computing the cross-entropy loss, and (b) confidence temperature T_conf to scale the softmax probabilities when determining whether the confidence exceeds the threshold for utilizing unlabeled data. By applying these two temperatures, we can effectively mitigate the issues caused by flat softmax probabilities from VLMs, leading to improved finetuning performance.

Stage-wise Finetuning with Temperatures (SWIFT)

1

To mitigate the domain gap in retrieved data, we propose a stage-wise finetuning approach. Specifically, we initialize the classification head by linear probing with few-shot examples, providing a better starting point for semi-supervised finetuning. We also apply few-shot finetuning at the third stage to calibrate the model to the target domain.

Results

SWIFT achieves SOTA SSFSL performance

1

  • Left: across five SSL benchmark datasets, our simple temperature solution effectively improves existing SSL methods like FixMatch and DebiasPL by a large margin (up to 20%) when finetuning VLMs. Our final method SWIFT significantly outperforms all existing SSL methods and FSL methods by 5% on average.
  • Right: each component in SWIFT brings significant gains. Notably, our SWIFT recipe generalizes well to various existing SSL methods.

Loss temperature strengthens training supervision

comparison with SOTA.

We illustrate the impact of loss temperature T_loss through few-shot finetuning the visual encoder of OpenCLIP ViT-B/32. Training loss (left) and test accuracy (right) over epochs show that finetuning without temperature (i.e., T_loss=1.0) yields slow convergence (slow reduction in training loss and increase in test accuracy), due to the weak supervision. In contrast, applying a loss temperature, either by fixing T_loss to a moderately small value (e.g., 0.1 or 0.07, solid lines) or by learning it dynamically (dashed lines), greatly accelerates convergence and improves test accuracy, demonstrating the strengthening of training supervisions.

Confidence temperature enables utilization of pseudo-labeled data

comparison with SOTA.

We illustrate the impact of confidence temperature T_conf through finetuning OpenCLIP ViT-B/32 with FixMatch on semi-Aves. The left figure shows that without temperature (i.e., T_conf=1.0), the utilization of unlabeled data is zero with a default confidence threshold of 0.8, resulting no accuracy gains over the few-shot finetuning. However, reducing the confidence temperature, e.g., fixing T_conf to a moderately small value (0.1 or 0.07) significantly increases the utilization of unlabeled data, yielding notable accuracy gains (right).

BibTeX

If you find our work useful, please consider citing our papers:


@article{liu2026solving,
title={Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspective},
author={Liu, Tian and Basu, Anwesha and Caverlee, James and Kong, Shu},
journal={European Conference on Computer Vision (ECCV)},
year={2026}
}

@inproceedings{liu2025few,
    title={Few-Shot Recognition via Stage-Wise Retrieval-Augmented Finetuning},
    author={Liu, Tian and Zhang, Huixin and Parashar, Shubham and Kong, Shu},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    year={2025}
}