# DeepLearning **Repository Path**: derrickwu/overall_knowledges_of_DeepLearning ## Basic Information - **Project Name**: DeepLearning - **Description**: 这个项目是从https://github.com/wangshusen/DeepLearning.git引进的，其包含了几乎全部的深度学习及其相关的内容。 - **Primary Language**: Python - **License**: Not specified - **Default Branch**: master - **Homepage**: None - **GVP Project**: No ## Statistics - **Stars**: 0 - **Forks**: 1 - **Created**: 2023-09-07 - **Last Updated**: 2023-09-07 ## Categories & Tags **Categories**: Uncategorized **Tags**: None ## README # CS583: Deep Learning 1. **Machine learning basics.** This part briefly introduces the fundamental ML problems-- regression, classification, dimensionality reduction, and clustering-- and the traditional ML models and numerical algorithms for solving the problems. * ML basics [[slides-1](https://github.com/wangshusen/DeepLearning/blob/master/Slides/1_ML_Basics.pdf)] [[slides-2](https://github.com/wangshusen/DeepLearning/blob/master/Slides/1_Models.pdf)]. * Regression [[slides-1](https://github.com/wangshusen/DeepLearning/blob/master/Slides/2_Regression_1.pdf)] [[slides-2](https://github.com/wangshusen/DeepLearning/blob/master/Slides/2_Regression_2.pdf)]. * Classification. - Logistic regression [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/3_Classification_1.pdf)] [[lecture note](https://github.com/wangshusen/DeepLearning/blob/master/LectureNotes/Logistic/paper/logistic.pdf)]. - SVM [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/3_Classification_2.pdf)]. - Softmax classifier [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/3_Classification_3.pdf)]. - KNN classifier [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/3_Classification_4.pdf)]. * Regularizations [[slides-1](https://github.com/wangshusen/DeepLearning/blob/master/Slides/3_Optimization.pdf)] [[slides-2](https://github.com/wangshusen/DeepLearning/blob/master/Slides/3_Regularizations.pdf)]. * Clustering [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/3_Clustering.pdf)]. * Dimensionality reduction [[slides-1](https://github.com/wangshusen/DeepLearning/blob/master/Slides/5_DR_1.pdf)] [[slides-2](https://github.com/wangshusen/DeepLearning/blob/master/Slides/5_DR_2.pdf)] [[lecture note](https://github.com/wangshusen/DeepLearning/blob/master/LectureNotes/SVD/svd.pdf)]. * Scientific computing libraries. [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/5_DR_3.pdf)]. 2. **Neural network basics.** This part covers the multilayer perceptron, backpropagation, and deep learning libraries, with focus on Keras. * Multilayer perceptron and backpropagation [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/6_NeuralNet_1.pdf)] [[lecture note](https://github.com/wangshusen/DeepLearning/blob/master/LectureNotes/BP/bp.pdf)]. * Keras [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/6_NeuralNet_2.pdf)]. * Further reading: - [[activation functions](https://adl1995.github.io/an-overview-of-activation-functions-used-in-neural-networks.html)] - [[parameter initialization](https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79)] - [[optimization algorithms](http://ruder.io/optimizing-gradient-descent/)] 3. **Convolutional neural networks (CNNs).** This part is focused on CNNs and its application to computer vision problems. * CNN basics [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/7_CNN_1.pdf)]. * Tricks for improving test accuracy [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/7_CNN_2.pdf)]. * Feature scaling and batch normalization [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/7_CNN_3.pdf)]. * Advanced topics on CNNs [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/7_CNN_4.pdf)]. * Popular CNN architectures [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/7_CNN_5.pdf)]. * Further reading: - [style transfer (Section 8.1, Chollet's book)] - [visualize CNN (Section 5.4, Chollet's book)] 4. **Recurrent neural networks (RNNs).** This part introduces RNNs and its applications in natural language processing (NLP). * Categorical feature processing [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/9_RNN_0.pdf)] [[video (Chinese)](https://youtu.be/NWcShtqr8kc)]. * Text processing and word embedding [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/9_RNN_1.pdf)] [[video (Chinese)](https://youtu.be/6_2_2CPB97s)]. * RNN basics [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/9_RNN_2.pdf)] [[video (Chinese)](https://youtu.be/Cc4ENs6BHQw)]. * LSTM [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/9_RNN_3.pdf)] [[reference](http://colah.github.io/posts/2015-08-Understanding-LSTMs/)] [[video (Chinese)](https://youtu.be/vTouAvxlphc)]. * Making RNNs more effective [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/9_RNN_4.pdf)] [[video (Chinese)](https://youtu.be/pzWHk_M23a0)]. * Text generation [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/9_RNN_5.pdf)] [[video (Chinese)](https://youtu.be/10cjvcrU_ZU)]. * Machine translation [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/9_RNN_6.pdf)] [[video (Chinese)](https://youtu.be/gxXJ58LR684)]. * Attention [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/9_RNN_8.pdf)] [[video (English)](https://youtu.be/B3uws4cLcFw)] [[video (Chinese)](https://youtu.be/XhWdv7ghmQQ)] [[reference](https://distill.pub/2016/augmented-rnns/)]. * Self-attention [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/9_RNN_9.pdf)] [[video (English)](https://youtu.be/06r6kp7ujCA)] [[video (Chinese)](https://youtu.be/Vr4UNt7X6Gw)]. * Image caption generation [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/9_RNN_7.pdf)] [[reference](https://machinelearningmastery.com/develop-a-deep-learning-caption-generation-model-in-python/)]. 5. **Language Models beyond RNNs.** * Transformer (1/2): attention without RNN [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/10_Transformer_1.pdf)] [[video (English)](https://youtu.be/FC8PziPmxnQ)] [[video (Chinese)](https://youtu.be/aButdUV0dxI)]. * Transformer (2/2): from shallow to deep [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/10_Transformer_2.pdf)] [[video (English)](https://youtu.be/J4H6A4-dvhE)] [[video (Chinese)](https://youtu.be/aJRsr39F4dI)] [[reference](https://arxiv.org/pdf/1706.03762.pdf)]. * BERT: pre-training Transformer [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/10_BERT.pdf)] [[video (English)](https://youtu.be/EOmd5sUUA_A)] [[video (Chinese)](https://youtu.be/UlC6AjQWao8)] [[reference](https://arxiv.org/pdf/1810.04805.pdf)]. 6. **Autoencoders.** This part introduces autoencoders for dimensionality reduction and image generation. * Autoencoder for dimensionality reduction [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/8_AE_1.pdf)]. * Variational Autoencoders (VAEs) for image generation [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/8_AE_2.pdf)]. 7. **Generative Adversarial Networks (GANs).** * DC-GAN [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/12_GAN.pdf)]. 8. **Deep Reinforcement Learning.** * Reinforcement learning basics [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/13_RL_1.pdf)] [[lecture note](https://github.com/wangshusen/DeepLearning/blob/master/LectureNotes/DRL/DRL.pdf)] [[video (Chinese)](https://youtu.be/vmkRMvhCW5c)]. * Value-based learning [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/13_RL_2.pdf)] [[video (Chinese)](https://youtu.be/jflq6vNcZyA)]. * Policy-based learning [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/13_RL_3.pdf)] [[video (Chinese)](https://youtu.be/qI0vyfR2_Rc)]. * Actor-critic methods [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/13_RL_4.pdf)] [[video (Chinese)](https://youtu.be/xjd7Jq9wPQY)]. * AlphaGo and Monte Carlo tree search [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/13_RL_5.pdf)] [[video (Chinese)](https://youtu.be/zHojAp5vkRE)]. 9. **Parallel Computing.** * Basics and MapReduce [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/14_Parallel_1.pdf)] [[lecture note](https://github.com/wangshusen/DeepLearning/blob/master/LectureNotes/Parallel/Parallel.pdf)] [[video (Chinese)](https://youtu.be/gVcnOe6_c6Q)]. * Parameter server and decentralized network [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/14_Parallel_2.pdf)] [[video (Chinese)](https://youtu.be/Aga2Lxp3G7M)]. * TensorFlow's mirrored strategy and ring all-reduce [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/14_Parallel_3.pdf)] [[video (Chinese)](https://youtu.be/rj-hjS5L8Bw)]. * Federated learning [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/14_Parallel_4.pdf)] [[video (Chinese)](https://youtu.be/STxtRucv_zo)]. 10. **Adversarial Robustness.** This part introduces how to attack neural networks using adversarial examples and how to defend from the attack. * Data evasion attack and defense [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/11_Evasion.pdf)] [[lecture note](https://github.com/wangshusen/DeepLearning/blob/master/LectureNotes/Adversarial/DataAttacks.pdf)]. * Data poisoning attack [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/11_Poisoning.pdf)] [[video (Chinese)](https://youtu.be/_K0nZcqdu5w)]. * Further reading: [[Adversarial Robustness - Theory and Practice](https://adversarial-ml-tutorial.org/)]. 11. **Meta Learning.** * Few-shot learning: basic concepts [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/16_Meta_1.pdf)] [[video (English)](https://youtu.be/hE7eGew4eeg)] [[video (Chinese)](https://youtu.be/UkQ2FVpDxHg)]. * Siamese network [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/16_Meta_2.pdf)] [[video (English)](https://youtu.be/4S-XDefSjTM)] [[video (Chinese)](https://youtu.be/Er8xH_k0Vj4)]. * Pretraining + fine tuning [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/16_Meta_3.pdf)] [[video (English)](https://youtu.be/U6uFOIURcD0)] [[video (Chinese)](https://youtu.be/3zSYMuDm6RU)]. 12. **Recommender System.** This part is focused on the collaborative filtering approach to recommendation based on the user-item rating data. This part covers matrix completion methods and neural network approaches. * Collaborative filtering [[slides](https://github.com/wangshusen/DeepLearning/blob/master/Slides/15_Recommender.pdf)].