Advanced Deep Learning

EECS E6691 - Topics and Data Driven Analysis and Computation

Advanced Deep Learning

Columbia University Course

Zoran Kostic, Ph.D., Dipl. Ing., Professor of Professional Practice, zk2172(at)columbia.edu

Electrical Engineering Department, Data Sciences Institute, Columbia University in the City of New York

Course in a nutshell:

Advanced theory and practice of Deep Learning. Applications and projects.

Description: Advanced (second) course on Deep Learning

Bulletin Description: Regularized autoencoders, sparse coding and predictive sparse decomposition, denoising autoencoders, representation learning, manifold perspective on representation learning, structured probabilistic models for deep learning, Monte Carlo methods, training and evaluating models with intractable partition functions, restricted Boltzmann machines, approximate inference, deep belief networks, deep learning in speech and object recognition.

Detailed Description for Spring 2023

Instructor: Dr. Mehmet Kerem Turkcan mkt2126 (at) columbia.edu

This is an advanced-level course in which the students study topics in deep learning. It is required that students had previously taken a first-course in deep learning. The course consists of: (i) lectures on state-of-the-art architectural and modeling concepts, (ii) assignments, (iii) exam, and a (iv) final project. The course will address topics beyond material covered in the first course on Deep Learning (such as ECBM E4040), with applications of interest to students. In 2023, the main subject of the lectures will be object detection.

Students entering the course have to have prior experience with deep learning and neural network architectures including Convolutional Neural Nets (CNNs), Recurrent Neural Networks (RNNs), Long Short Term Memories (LSTMs), and autoencoders. They need to have working knowledge of coding in Python, Python libraries, Jupyter notebook, TensorFlow both on local machines and on Google Cloud, and of GitHub or similar code hosting tools. The framework and associated tools which will be the focus of this course are PyTorch and Google Cloud. Students have to be self-sufficient learners and to take an active role during classroom activities.

There will be a few (3-4) assignments throughout the semester focusing on coding. In the second half of the course, there will be a midterm exam comprised of multiple-choice questions.

Final projects need to be documented in a conference-style report, with code deposited in a GitHub repository. The code needs to be documented and instrumented such that the instructor can run it after a download from the repository. A Google Slides presentation of the project suitable for a poster presentation is required.

Prerequisites

(i) Machine Learning (taken previously, or in parallel with this course).

(ii) ECBM E4040 Neural Networks and Deep Learning, or an equivalent neural network/DL university course taken for academic credit.

(iii) The course requires an excellent theoretical background in probability and statistics, and linear algebra.

Students are strongly advised to drop the class if they do not have an adequate theoretical background and/or previous experience with programming of deep learning models. It is strongly advised (the instructor’s requirement) that students take no more than 12 credits of any coursework (including this course and project courses) during the semester while this course is being taken.

Registration

The enrollment is limited to several dozen students. The instructor’s permission is required to register. Students interested in the course need to populate the SSOL waitlist, and MUST also populate the questionnaire. The instructor will move the students off of the SSOL waitlist after reviewing the questionnaire.

(Tentative) Grading for the course (2023 Spring)

Assignments: 30%

Midterm Exam (Delivered at Week 11): 30%

Project (Final report & Code Repository): 40%

(Potential) Class Contribution: x

2022 Spring Projects

• Pix2Pix Image-to-Image Translation with Conditional Adversarial Networks

• Jump rope counter

• Black Box Adversarial Attack with Style Information

• Learning Signed Distance Function for 3D Shape Representation (DeepSDF)

• Representation learning without any labeled data

• Comparison of Self-Supervised Models for Music Classification

• Subcellular localization of proteins using deep learning

• Image Descriptions Generator

• Predicting remaining surgery duration

• Vision Transformer

• Adversarial Audio Synthesis

• PlaNet - Latent Dynamics from Pixels

• RecoNET: Understanding What happens in Videos

2021 Spring Projects

• Multi-Graph Graph Attention Network for Music Recommendation

• 3D Facial Reenactment from 2D Video

• Temporal Fusion Transformers for Time Series Forecasting

• Deep Reinforcement Learning for Environmental Policy

• Stochastic & Split Convolutions for High Resolution Image Classification (Integrated with EfficientNet)

• Temporal Fusion Transformers for Time Series Forecasting

• Forecasting Corn Yields with Semiparametric CNNs

• Pose Estimation + Instance Segmentation

• Speaker Independent Speech Separation

• Generalized Autoencoder-based Transfer Learning for Structural Damage Assessment

• Deep Reinforcement Learning for Environmental Policy

• End-to-end object detection with Transformers

• Exploring latent space of InfoGAN

• Multi-Graph Graph Attention Network for Music Recommendation

• 3D Facial Reenactment from 2D Video

• Pose Estimation + Instance Segmentation

2018-2020 Projects

• See list of projects under E6040 link

Course sponsored by equipment and financial contributions of:

• NVidia GPU Education Center, Google Cloud, IBM Bluemix, AWS Educate, Atmel, Broadcom (Wiced platform); Intel (Edison IoT platform), Silicon Labs.