Presentation for the graduate course CSD 662: Advanced Deep Learning.
Abstract
This presentation provides an in-depth exploration of two significant advancements in self-supervised learning: SimCLR and I-JEPA. We delve into the core concepts of contrastive learning, which enables models to learn meaningful representations from unlabeled data by maximizing agreement between different views of the same data point.
SimCLR (A Simple Framework for Contrastive Learning of Visual Representations)
We analyze the key components of the SimCLR framework, including:
- The role of data augmentations in creating positive pairs.
- The importance of a projection head (MLP) for defining the contrastive loss.
- The effectiveness of the NT-Xent (Normalized Temperature-scaled Cross-Entropy) loss function.
- The impact of larger batch sizes and longer training on representation quality.
I-JEPA (Image-based Joint-Embedding Predictive Architecture)
Next, we shift focus to I-JEPA, a non-contrastive, generative approach that learns by predicting representations of masked-out image blocks in an abstract space. Key aspects covered include:
- The concept of predicting in representation space rather than pixel space to encourage semantic feature learning.
- The architecture, comprising a context encoder, a predictor, and a target encoder.
- The multi-block masking strategy and its benefits for learning scalable and efficient representations.
- How I-JEPA avoids the collapse problem common in self-supervised methods without needing negative pairs or momentum encoders.
"By comparing and contrasting these two powerful methods, the presentation illuminates the evolving landscape of self-supervised learning and its potential to reduce reliance on large labeled datasets."