Natural Language Processing

Course Description

The course mainly introduces the theories and applications of Natural Language Processing (NLP). This course starts from basic concepts of NLP to the-state-of-the-art NLP algorithms such as transformer. Also, this course will introduce how to apply the concepts that learnt in this course to task-oriented dialog system.

Lecture 1: Introduction to Natural Language Processing

• Lecture Slides: PPT
• Reading Materials
  1. Can We Automate Scientific Reviewing?
  2. The Design and Implementation of XiaoIce, an Empathetic Social Chatbot
  3. Recipes for building an open-domain chatbot
  4. Semantically-Aligned Equation Generation for Solving and Reasoning MathWord Problems
  5. Stanza : A Python Natural Language Processing Toolkit for Many Human Languages

Lecture 2: Distributed Representaion of Words

• Lecture Slides: PPT
• Reading Materials
  1. Efficient Estimation of Word Representations in Vector Space
  2. Distributed Representations of Words and Phrases and their Compositionality
  3. Stanford University CS224N Lecture1 (2019 Edition)

Lecture 3: Advanced Word Vector and Introduction to Neural Network

• Lecture Slides: PPT
• Reading Materials
  1. GloVe: Global Vectors for Word Representation
  2. Speech and Language Processing: Information Extraction (Chapter number may vary according to the edition)
  3. Stanford University CS224N Lecture2 and Lecture3 (2019 Edition)

Lecture 4: Backpropagation and Computation Graphs

• Lecture Slides: PPT
• Reading Materials
  1. Maxout Networks
  2. Understanding the difficulty of training deep feedforward neural networks
  3. Dropout: A Simple Way to Prevent Neural Networks from Overfitting
  4. Adam: A Method for Stochastic Optimization
  5. An overview of gradient descent optimization algorithms
  6. Stanford University CS224N Lecture4 (2019 Edition)

Lecture 5: Dependency Parsing

• Lecture Slides: PPT
• Reading Materials
  1. Incrementality in Deterministic Dependency Parsing
  2. Globally Normalized Transition-Based Neural Networks
  3. Fast and Robust Neural Network Joint Models for Statistical Machine Translation
  4. A Fast and Accurate Dependency Parser using Neural Networks
  5. Stanford University CS224N Lecture5 (2019 Edition)
• Universal Dependency Parsing

Lecture 6: Recurrent Neural Networks and Language Models

• Lecture Slides: PPT
• Reading Materials
  1. Speech and Language Processing: N-gram Language Models (Chapter number may vary according to the edition)
  2. Deep Visual-Semantic Alignments for Generating Image Descriptions
  3. Speechless? Here’s how AI learns to finish your sentences
  4. Stanford University CS224N Lecture6 (2019 Edition)
  5. Deep Learning - Sequence Modeling: Recurrent and Recursive Nets

Lecture 7: Vanishing Gradients and Fancy RNNs

• Lecture Slides: PPT
• Reading Materials
  1. On the difficulty of training Recurrent Neural Networks
  2. Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
  3. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
  4. Deep Residual Learning for Image Recognition
  5. Densely Connected Convolutional Networks
  6. Highway Networks
  7. Stanford University CS224N Lecture7 (2019 Edition)

Lecture 8: Machine Translation, Seq2Seq, Attention, and Transformer

• Lecture Slides: PPT
• Reading Materials
  1. Attention Is All You Need
  2. Attention Is All You Need (Code Explanation)
  3. Layer Normalization
  4. Universal Language Model Fine-tuning for Text Classification
  5. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
  6. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
  7. Stanford University CS224N Lecture8 (2019 Edition)
  8. Stanford University CS224N Lecture13 (2019 Edition)

Lecture 9: Task Oriented Dialogue Systems & Multi-Modal Dialog System

• Lecture Slides: PPT
• Reading Materials
  1. Speech and Language Processing: Chatbots and Dialogue Systems (Chapter number may vary according to the edition)
  2. Continuously Learning Neural Dialogue Management
  3. Sample-efficient Actor-Critic Reinforcement Learning with Supervised Data for Dialogue Management
  4. Augment Information with Multimodal Information

Textbooks

• Natural Language Processing
  • Jurafsky and Martin, Speech and Language Processing (3rd ed.)
• Deep Learning
  • Goodfellow, Bengio, and Courville, Deep Learning
  • Zhang et al., Dive into Deep Learning

Online Courses

• Natural Language Processing and Deep Learning
  • CS224n: Natural Language Processing with Deep Learning (2019 Edition)