Neural Networks and Deep Learning

Neural Networks and Deep Learning

Columbia University course ECBM E4040

Zoran Kostic, Ph.D., Dipl. Ing., Professor of Professional Practice, zk2172(at)columbia.edu

Electrical Engineering Department, Data Sciences Institute, Columbia University in the City of New York

Course in a nutshell:

Theoretical underpinnings and practical aspects of Neural Networks and Deep Learning. Convolutional and Recurrent Neural Networks. Focus on applications and projects.

Bulletin Description:

  • Developing features & internal representations of the world, artificial neural networks, classifying handwritten digits with logistics regression, feedforward deep networks, back propagation in multilayer perceptrons, regularization of deep or distributed models, optimization for training deep models, convolutional neural networks, recurrent and recursive neural networks, deep learning in speech and object recognition.

Time:

  • Fall 2021: To register into the course: (i) students need to get onto the Columbia SSOL waitlist, and (ii) populate this form (using CU UNI email). Those students who satisfy the requirements will be moved from the waitlist into the registered list by early September. The course can be taken by undergraduate and graduate student. See additional course information here.

  • Spring 2021, Fall 2020, Fall 2019, Fall 2018, Fall 2017, Fall 2016.

Prerequisites:

  • Required prerequisites: knowledge of linear algebra, probability and statistics, programming. Strongly recommended: machine learning.

  • Suggested prerequisite courses: (BMEB W4020) or (BMEE E4030) or (ECBM E4090) or (EECS E4750) or (COMS W4771) or similar.

Organization

  • Lectures:

    • Presentation of material by instructors and guest lecturers

  • Homeworks:

    • Combination of analytical and programming assignments

  • Projects:

    • Team-based

    • Students with complementary backgrounds

    • Significant design

    • Reports and presentations to Columbia and NYC community

    • Best could qualify for publications and/or funding

  • Industry participation:

    • Project definition and sponsoring

    • Weekly presentations

    • Interaction with students through mentoring


Content

  • Introduction to neural networks.

  • Convolutional and recurrent neural networks.

  • Focuses on the intuitive understanding of deep learning.

  • Review of underpinning theory - linear algebra, statistics, machine learning.

  • Analytical study and software design.

  • Three-four assignments in Python and one DL framework (Tensorflow or PyTorch)

  • Significant project.

  • Enables further exploration of key concepts in deep learning.

Typical Syllabus

  • Introduction to Course E4040

  • Introduction to Deep Learning (DL)

  • Introduction to DL computing Resources

  • Machine Learning Algorithms

  • t-SNE Data Visualization

  • Universal approximation theorem - Visual proof

  • Algebra review

  • Deep Feed-forward Networks

  • Back Propagation

  • Optimization

  • Convolutional Neural Nets (CNNs)

  • CNN applications

  • CNN Examples

  • Regularization

  • Practical Methodology for Deep Learning

  • Recurrent Neural Nets (RNNs)

  • RNN applications

  • Deel Learning Applications

  • Autoencoders

  • Generative Networks GANs, Variational Encoders

  • Generative Models

  • Guest Lectures

  • Deep Learning Trends

Books, Tools and Resources

Project Areas

  • Medical

  • Autonomous cars

  • Environmental

  • Smart cities

  • Physical data analytics

2021 Fall Projects

  • Composing Music With Recurrent Neural Networks

  • A New Backbone for Hyperspectral Image Reconstruction

  • A Recurrent Latent Variable Model for Sequential Data

  • Custom DenseNet models for Tiny ImageNet Classification

  • Deep compressive autoencoder for action potential compression in large-scale neural recording

  • Deep Networks with Stochastic Depth

  • DeepPainter: Painter Classification Using Deep Convolutional Autoencoders

  • Fairness without Demographics through Adversarially Reweighted Learning

  • FishNet: A Versatile Backbone for Image, Region, and Pixel Level Prediction

  • MoEL: Mixture of Empathetic Listeners

  • Neural Distance Embeddings for Biological Sequences

  • Neural Distance Embeddings for Biological Sequences

  • Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations

  • QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension

  • Residual Attention Network for Image Classification

  • Social GAN for Human Driving Trajectories Prediction

  • Spectral Representations for Convolutional Neural Networks

  • Squeeze-and-Excitation Networks Code avaiable

  • Towards Accurate Binary Convolutional Neural Network Xiaofan Lin, Cong Zhao, Wei Pan, (2017NIPS)

  • Video-based human emotion recognition

2021 Spring Projects

  • A deep learning framework for financial time series using stacked autoencoders and long-short term memory

  • A Neural Algorithm of Artistic Style

  • A neural attention model for speech command recognition

  • BinaryConnect: Training Deep Neural Networks with Binary Weights during Propagations

  • Composing Music With Recurrent Neural Networks

  • Conditional Generative Adversarial Net

  • Deep Double Descent: Where Bigger Models and More Data Hurt

  • Deep Koalarization: Image Colorization using CNNs and Inception-ResNet-v2

  • Deep Learning for Price Prediction of Cryptocurrencies

  • DeepPaint -- Classification of Paintings using Convolutional Autoencoder

  • Densely Connected Convolutional Networks

  • Enhancing Detection of Steady-State Visual Evoked Potentials Using Deep Learning

  • From 2D to 3D: Kidney Tumor Segmentation Challenge

  • GRUV: Algorithmic Music Generation using Recurrent Neural Networks

  • Highway Networks

  • MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

  • Neural Networks for Automated Essay Grading

  • Recurrent Neural Networks to Create Comprehensive 3D Models of Granular Media in YADE

  • Residual Attention Network for Image Classification

  • Singing Voice Separation from Monaural Recordings Using Deep Recurrent Neural Networks

  • U-Net: Convolutional Networks for Biomedical Image Segmentation

  • Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

2020 Fall Projects

  • Custom DenseNet models for Tiny ImageNet Classification

  • Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

  • A neural attention model for speech command recognition

  • CNN-Generated Images Are Surprisingly Easy to Spot... for Now

  • Composing Music With Recurrent Neural Networks

  • Custom DenseNet models for Tiny ImageNet Classification

  • Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding

  • Deep Learning and the Cross-Section of Stock Returns: Neural Networks Combining Price and Fundamental Information

  • Image Super-Resolution Using Deep Convolutional Networks

  • Machine learning methods for crop yield prediction and climate change impact assessment in agriculture

  • MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

  • Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations

  • PoseCNN: A convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes

  • Random Erasing Data Augumentation

  • Residual Attention Network for Image Classification

  • Rethinking Model Scaling for Convolutional Neural Networks - EfficientNet

  • Semi-Supervised Learning with Graph Convolutional Neural Networks

  • Time series forecasting of petroleum production using deep LSTM recurrent networks

  • Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

2019 Projects

  • A deep learning framework for financial time series using stacked autoencoders and long-short term memory

  • Adversarial Autoencoder Assisted Artifact Reduction of Ballistocardiogram in Simultaneous EEG-fMRI Recordings

  • Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks

  • Automated Gleason Grading of Prostate Cancer Tissue Microarrays via Deep Learning

  • Deep learning-based feature engineering for stock price movement prediction

  • Deformable Convolutional Networks:(Expanding the receptive field through Deformable Convolution)

  • Development and validation a RNN model to teach machines to read and comprehend

  • Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

  • FishNet: A Versatile Backbone for Image, Region, and Pixel Level Prediction

  • MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

  • Multi-Digit Number Recognition from Street View Imagery Using Deep Convolutional Neural Networks

  • One-Shot Video Object Segmentation for Mobile Vision Applications

  • Pelee: A Real-Time Object Detection System on Mobile Devices

  • Residual Attention Network for Image Classification

  • Short Term Electricity Consumption Forecasting in Residential Building with a Selected Auto-regressive Features&ConvLSTM Neural Network Method

  • Show and Tell: A Neural Image Caption Generator

  • SSD: Single Shot MultiBox Detector

  • Stock market's price movement prediction with LSTM neural networks

  • Towards Accurate Binary Convolutional Neural Network

  • Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks

  • Why Should I Trust You? Explaining the Predictions of Any Classifier

  • World Models

2018 Projects

  • A deep learning framework for relationship extraction from articles using long-short term memory and named entity recognition

  • A Neural Algorithm of Artistic Style

  • A Neural Representation of Sketch Drawings

  • Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks

  • Backprop KF: Learning Discriminative Deterministic State Estimators

  • Deep contextualized word representations

  • Dynamic Routing Between Capsules

  • Gesture Recognition

  • Learned in Translation: Contextualized Word Vectors

  • Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

  • Maximum Classifier Discrepancy for Unsupervised Domain Adaptation

  • Multi-Digit Number Recognition from Street View Imagery Using Deep Convolutional Neural Networks

  • Neural Networks for Automated Essay Grading

  • Parallel Multi-Dimensional LSTM,With Application to Fast Biomedical Volumetric Image Segmentation

  • PixelGAN Autoencoders

  • Prevention of catastrophic forgetting in Neural Networks for lifelong learning

  • Semantic Image Inpainting with Deep Generative Models

  • Towards Accurate Binary Convolutional Neural Network

  • Universal Style Transfer via Feature Transforms

  • Unsupervised Image-to-Image Translation Networks

2016 Projects

  • Striving for Simplicity: The All Convolutional Net

  • A Combined Semi-supervised Learning mechanism for Video Data via Deep Learning

  • A Neural Algorithm of Artistic Style

  • Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network

  • Colorful Image Colorization

  • Deep Networks with Stochastic Depth

  • Highway Networks

  • Image Super-Resolution Using Deep Convolutional Networks

  • Learning to Protect Communications with Adversarial Neural Cryptography

  • Singing Voice Separation from Monaural Recordings Using Deep

  • Recurrent Neural Networks

  • Spatial Transformer Networks

  • Spoken Language Understanding Using Long-Short Term Memory Neural Networks

  • Striving for Simplicity: The All Convolutional Net

  • Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Course sponsored by equipment and financial contributions of:

  • NVidia GPU Education Center, Google Cloud, IBM Bluemix, AWS Educate, Atmel, Broadcom (Wiced platform); Intel (Edison IoT platform), Silicon Labs.