Lecture 11: Multimodal Data Processing

(Last updated: Sep 27, 2026)

This lecture introduces the overall picture of machine learning tasks with multimodal data.

Check the GenAI usage policy if you are using the course materials with GenAI for self-study and fact-checking.

Preparation

Read the required course readings.

Lecture

Below are the slides:

• Slides for Lecture 11: Multimodal Data Processing

Required Course Readings

• Section 5.5.2 (Recurrent neural networks) and 5.5.3 (Transformers) in book Computer Vision: Algorithms and Applications (Szeliski, 2022).

Optional Course Readings

• Liang, P. P., Zadeh, A., & Morency, L. P. (2024). Foundations & trends in multimodal machine learning: Principles, challenges, and open questions. ACM computing surveys.
• Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., … & Sutskever, I. (2021, July). Learning transferable visual models from natural language supervision. In International conference on machine learning.

Exercises

See the instruction in the syllabus about how to use the exercises.

• Describe the role of positional encoding in the Transformer architecture. Why do we need it? How does it work exactly (e.g., which functions to use to compute the position, and how to do that)?
• Describe the intuition about how contrastive learning works when training a joint text-image representation using a large number of image-text pairs.
• Explain the differences between cross-attention (which is the one that you learned in the text data processing lecture) and self-attention. Specifically, explain how the query, key, and value vectors are computed differently (or in the same way).