Course Information

  • Instructor: Genya Ishigaki
  • Office Hours:
    • Monday 1:30 - 3 PM (In-person)
    • Thursday 3:30 - 5 PM (Zoom)
    • By appointment
  • Class Days/Time: Monday & Wednesday 10:45 AM - 12 PM
  • Classroom: MacQuarrie Hall 225
  • Prerequisites: CS 157A or an equivalent course. Allowed Declared Major: Computer Science, Bioinformatics, Data Science.

Course Description

Introduction to reinforcement learning, deep reinforcement learning, federated learning, and their applications in networking research.

Course Learning Outcomes (CLO)

Upon successful completion of this course, students will be able to:

  • Understand different types of reinforcement learning algorithms and when to use them.
  • Understand theoretical aspects of reinforcement learning.
  • Understand the basis of deep reinforcement learning.
  • Understand architectural benefits of federated learning.
  • Build a machine learning project to solve a social or technical issue.
  • Develop reinforcement learning applications.

Textbook

  • Richard S. Sutton and Andrew G. Barto, Reinforcement learning: An introduction (Second edition), MIT press, 2018.
    • This book is available online for free on the authors’ page.
    • We do not cover all the topics in the book as it is a comprehensive textbook. Appropriate sections will be indicated in syllabus and classes.
  • Open AI, Spinning Up in Deep RL
    • While the page says “Deep” RL, many of their resources explain the basics of RL itself.

  • For the Federated Learning and application parts, a set of papers will be listed in the course schedule.

  • (Optional) Yoav Shoham and Kevin Leyton-Brown, Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations, Cambridge University Press, 2009.

Other equipment

  • Computer
  • Python development environment
  • LaTeX (*Recommended for Project Paper)

Grading

Exams, Assignments, and Projects

  • This course is designed as a research-oriented course so that students can experience a simplified process of machine learning projects: problem formulation, modeling, method selection, and development.
    • The project requires students to apply (deep) reinforcement learning to some practical problems.
    • It is recommended to form a group of TWO students. I may approve exceptions (individual or group of three) with a valid reason.
    • Some example topics will be presented and discussed during a class, but students can choose any topic that they found interesting.
    • Major programming contribution from each group member is required for a passing grade. Details will be explained in class.
  • Assignments may include both theoretical and programming questions.
Item % in Final Grade
Exam 1 16 %
Exam 2 16 %
Assignment 1 13 %
Assignment 2 13 %
Assignment 3 13 %
Project Idea/Proposal Presentations 5 %
Project Final Presentation 8 %
Project Paper 16 %

Grading Table

Total Grade Letter Grade
92 and above A
90-91 A-
87-89 B+
82-86 B
80-81 B-
77-79 C+
72-76 C
70-71 C-
67-69 D+
62-66 D
60-61 D-
59 and below F

Late Submission

Late submissions within 24 hours will be deducted 10% of its final grade. Submissions over 24 hours late will have 20% grade deducted. Late submissions over 2 days will not be accepted.

Attendance

I do not take attendance except for the first two classes. Students not attending either of the first two classes will be dropped to make room for students on the waiting list. Attempting to get marked as present (by have someone else attend in your place or using technological deceptions) will be considered academic dishonesty and at a minimum will result in you getting dropped from the course.

Grading Policy

The University Policy S16-9, Course Syllabi (http://www.sjsu.edu/senate/docs/S16-9.pdf) requires the following language to be included in the syllabus:

“Success in this course is based on the expectation that students will spend, for each unit of credit, a minimum of 45 hours over the length of the course (normally three hours per unit per week) for instruction, preparation/studying, or course related activities, including but not limited to internships, labs, and clinical practica. Other course structures will have equivalent workload expectations as described in the syllabus.”

University Policies

Per University Policy S16-9, university-wide policy information relevant to all courses, such as academic integrity, accommodations, etc. will be available on Office of Graduate and Undergraduate Programs’ Syllabus Information web page at http://www.sjsu.edu/gup/syllabusinfo/. Make sure to review these policies and resources.

Tentative Schedule and Topics

  • Please note the following is a tentative schedule.
Week Date Topic Reference Note
1 8/23 Overview    
1 8/25 What is Learning? Shoham & Leyton-Brown Chap 7
Paper		  
2 8/30 MDP Sutton & Barto Chap 3  
2 9/1 Policies and Value Functions Sutton & Barto Chap 3  
3 9/6 No Class (Labor Day)    
3 9/8 Dynamic Programming Sutton & Barto Chap 4  
4 9/13 Dynamic Programming   Assignment 1 due
4 9/15 Monte Carlo Method Sutton & Barto Chap 5  
5 9/20 Monte Carlo Method    
5 9/22 TD Learning Sutton & Barto Chap 6  
6 9/27 TD Learning: Q-learning    
6 9/29 Taxonomy and Review Spinning Up in Deep RL: Taxonomy Assignment 2 due
7 10/4 Exam 1    
7 10/6 Deep RL Spinning Up in Deep RL  
8 10/11 Deep RL and Baseline Implementation   Project Pair due
8 10/13 MAB Sutton & Barto Chap 2  
9 10/18 MAB and Regret Shoham & Leyton-Brown Chap 7  
9 10/20 Application of RL Paper 1
Paper 2		  
10 10/25 Application of RL Paper 3
Paper 4		 Assignment 3 due
10 10/27 Integrating Learning and Planning Sutton & Barto Chap 8  
11 11/1 Project Discussion   Project Idea Slides due
11 11/3 Policy Gradient Methods Sutton & Barto Chap 13  
12 11/8 Policy Gradient Methods    
12 11/10 Proposal Presentation   Project Proposal Slides due
13 11/15 Actor-Critic and Review Sutton & Barto Chap 13  
13 11/17 Exam 2    
14 11/22 Federated Learning Article
Paper
Survey		  
14 11/24 No Class (No Instruction Day)    
15 11/29 Federated Learning    
15 12/1 Final presentation   Final Presentation Slides due
16 12/6 Final presentation    
  12/10     Project Paper due
  • When you are struggling to find a space to study
  • If you do not have right equipment (laptop, etc.)
  • If you want to talk to someone
    • “Whether you are struggling with stress, depression, anxiety or relationship problems, Counseling and Psychological Services is here to provide the support you need to succeed at SJSU. In our current state of remote online instruction, CAPS is providing all of its services through confidential telehealth sessions.”
    • https://www.sjsu.edu/counseling/
  • If you need additional accommodation for your learning
    • “The Accessible Education Center (AEC) proudly presents its vision of redefining ability at San Jose State University by providing comprehensive services in support of the educational development and success of student with disabilities.”
    • https://www.sjsu.edu/aec/
  • If you find a financial challenge
    • “SJSU Cares is here to provide assistance when you need it most. We provide resources and services for SJSU students facing an unforseen financial crisis. If you’re having trouble paying for food, housing or other bills, face homelessness, food insecurity, etc.”
    • https://www.sjsu.edu/sjsucares/