Deep Learning on the Edge

EECS E6792 Deep Learning on the Edge

(previous: EECS E6692 Topics in Data Driven Analysis and Computation  - Deep Learning on the Edge)

Columbia University Course 

Zoran Kostic, Ph.D., Dipl. Ing.,  Professor of Professional Practice, zk2172(at)columbia.edu

Electrical Engineering Department, Data Sciences Institute, Columbia University in the City of New York

Course in a Nutshell

Theory and Practice of Deep Learning on the Edge, with Labs Using Nvidia Jetson Nano Devices.

Description

This is an advanced-level course with labs in which students build and experiment with deep learning models, implementing them on a low-power GPU edge computing device. The topics covered by the course are: (*) architectures of low power GPU devices; (*) algorithms and DL models suitable for edge implementation; (*) CUDA language; (*) pre-processing to condition the data; (*) labeling for ground-truth annotation; (*) profiling techniques; (*) connectivity between edge devices and cloud computing servers; (*) real-life data for experimentation; (*) comparison of performance for a variety of methods; (*) comparison of performance of edge-computing and cloud computing approaches.

The labs will use NVIDIA Jetson Nano as the edge device.

From year to year, the course may focus on a contemporary application, such as a smart city intersection or patient health monitoring.

The course is interactive and requires students’ active participation in every session by presenting, interpreting, and implementing deep learning papers and models. The study of concepts is accompanied by lab sessions in which students will bring the models under consideration to life and experiment with them. There will be half a dozen lab sessions. There will be a final project, preceded by a proposal. Final projects need to be documented in a conference-style report, with code deposited in a GitHub repository. The code needs to be documented and instrumented so the instructor can run it after downloading from the repository. A Google Slides presentation of the project (suitable for poster version) is required. A web-hosted record/documentation of both labs and projects may be expected.

Students must be self-sufficient learners and ready for hands-on coding. They have to take an active role during the lab activities.

Prerequisites

(i) Machine Learning (taken previously for academic credit).

(ii) Students entering the course have to have prior experience with deep learning and neural network architectures, including Convolutional Neural Nets (CNNs), Recurrent Neural Networks (RNNs), Long Short Term Memories (LSTMs), and autoencoders. They need to have a working knowledge of coding in Python, Python libraries, Jupyter notebook, Tensorflow, both on local machines and on the Google Cloud, and of Bitbucket, Github, or similar.

(iii) ECBM E4040 Neural Networks and Deep Learning, or an equivalent neural network/DL university course taken for academic credit. Whereas the quality of online ML and DL courses (Coursera, Udacity, edX) is outstanding, many takers of online courses do the hands-on coding assignments superficially and therefore do not gain practical coding skills, which are essential to participation in this advanced course.

(iv) The course requires an excellent theoretical background in probability and statistics and linear algebra. 

Students are strongly advised to drop the class if they do not have adequate theoretical background and/or previous experience with programming deep learning models. It is strongly advised (the instructor’s requirement) that students take no more than 12 credits of any coursework (including projects) during the semester while this course is being taken.

Registration

The enrollment is limited to several dozen students. Instructor’s permission is required to register. Students interested in the course need to register on the SSOL waitlist and may be asked to complete a questionnaire (to be announced). The instructor will move the students off the SSOL waitlist after reviewing the questionnaire.

2025 Spring Projects

• Anomaly Detection of Retinal Fundus Images on Jetson Nano

• Can we Compress the Smaller LLaMas onto the Jetson Nano?

• Real-time financial sentiment analysis based on distilled finBert

• MUSE – User-Sensitive musical Expression

• Real-time single/multi-person pose estimation on Edge AI devices

• Energy Efficient Edge AI Inference on Jetson Nano

• Real-Time Birdsong Recognition on Edge Devices

• Real-Time Artist Recognition and Insight Generation (Claude MonAI)

• Implement Traffic Sign Detection with Latest YOLO Models

• Crosswalk detection on Jetson Nano

• Real-Time Hierarchical Visual Localization and SLAM on Jetson Nano

• BitNet : Analysis of 1-bit LLMs for Edge device

• Autonomous navigation for blind on Jetson Nano

• Smart Recycling: Household Waste Classification Using Deep Learning

• Emotions at the Edge: Real-Time Detection with Jetson Nano

2024 Spring Projects

• Real-time single/multi-person pose estimation on Edge AI devices

• AutoCaption: Generating Engaging Instagram Captions from Photos

• Pupil Center Based Eye Tracking

• Tool learning for LLM assistants responding to voice commands

• Live-Inference Skin Cancer Detection 

• Landmark Lens: Real-Time NYC Landmark Recognition with YOLO and DETR Models

• Listen, Chat, and Edit on Edge: Text-Guided Soundscape Modification for Enhanced Auditory Experience

• Sports video Analysis on Jetson Nano

• Mirror and Glass Detection/ Segmentation

• LR-Bot: An Autonomous Litter Rescuer

• NanoFaceCheck

• Plant Classification

2022 Spring Projects

• Real-Time Chess Bot

• Sign Language Translation

• Fall Detection

• Conditional GAN Compression

• Deployment of a DL model on Edge Devices

• Sleep Apnea Detection

• Object tracking Using RL

• Small-Footprint Keyword Spotting

• Conditional GAN Compression

Course sponsored by equipment and financial contributions of:

• Amazon AWS, NVidia GPU Education Center, Google Cloud, IBM Bluemix, AWS Educate, Atmel, Broadcom (Wiced platform); Intel (Edison IoT platform), Silicon Labs.