This repository contains a detailed review, summary, and hands-on implementation of the Swin Transformer—a state-of-the-art vision transformer architecture that introduces a hierarchical structure and shifted windows for efficient vision tasks.

Authors: Bahaar Khalilian, Hesam Mohebi
Course: Deep Learning – Final Project

The Swin Transformer (ICCV 2021) addresses limitations in prior vision transformers (ViT, DeiT), especially for dense prediction tasks. It reduces computational complexity from quadratic to linear by applying window-based self-attention and introduces shifted windows for cross-window connection.

Key improvements over ViT:

• Hierarchical structure with progressive downsampling
• Shifted Window (SW-MSA) mechanism
• Strong scalability and flexibility for classification, detection, and segmentation

• Input image → split into 4x4 patches
• Linear embedding projects to C dimensions
• Hierarchical structure reduces spatial dimensions while increasing channel depth
• Final structure:

• Replaces global attention with:
  • Window-based MSA (W-MSA): localized attention
  • Shifted Window MSA (SW-MSA): connects non-overlapping windows
• Uses MLPs, LayerNorm, and residuals in Transformer-style architecture

• Best results from Swin-L (384²) with fine-tuning: 87.26% Top-1 accuracy
• Efficient training using AdamW, cosine decay LR, warm-up, large batch sizes
• Scales well with ImageNet-22K pretraining

• Tested with Cascade Mask R-CNN, ATSS, Sparse RCNN
• Swin-T improves AP when replacing ResNet50
• Swin-L + HTC++ achieves top results in full detection pipeline

• Swin + UperNet achieves 62.8 mIoU
• Outperforms both CNNs and older Transformer-based models

• Shifted Windows (SW-MSA) consistently improve:
  • +1.1% accuracy (ImageNet)
  • +2.0 AP (COCO)
  • +0.8 mIoU (ADE20K)

• Repository: Swin Transformer GitHub
• Challenges faced:
  • PyTorch DDP errors on Windows → bypassed with single-GPU mode
  • Folder naming required WordNet synsets
  • Reorganized ImageNet validation data using ground-truth mappings
• Final reproduced results:

• Repo: Swin Object Detection
• Tried on WSL with Anaconda
• MMDetection and MMCV version mismatches caused errors
• Scripts not compatible with newer MMEngine-based frameworks

• Repo used MMSegmentation with MMCV
• Version conflicts caused dependency deadlocks
• Difficult to upgrade without breaking CUDA or PyTorch compatibility

• Swin V1 suffers from exploding gradients with 1B+ parameters
• Swin V2 introduces:
  • Pre-normalization
  • RMSNorm for better gradient flow

• Swin V1 uses fixed-size learnable bias → not scale-invariant
• Swin V2 uses continuous log-spaced bias → generalizes to any resolution

1. Swin Transformer – ICCV 2021
2. ViT – ICLR 2021
3. DeiT – PMLR 2021
4. Swin V2 – CVPR 2022

.
├── swin_review.md                # This README
├── notebooks/
│   └── swin_eval.ipynb           # Image classification experiments
├── scripts/
│   └── eval_single_gpu.py        # Modified evaluation script
├── reports/
│   └── swin_review.pdf           # Full paper summary and evaluation