Downloaded 387 times
Uploaded byinside-BigData.com
Tutorial on Deep Learning
The document provides a comprehensive overview of deep learning, including its historical development, key concepts such as neural networks, and notable figures in the field. It discusses various techniques like back-propagation, convolutional networks, and support vector machines, along with applications in image recognition and natural language processing. The conclusion emphasizes the potential of deep learning to transform industries and improve learning processes, while also noting challenges that remain in achieving advanced AI.
More Related Content
Similar to Tutorial on Deep Learning
![Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]](https://cdn.slidesharecdn.com/ss_thumbnails/codedagentsdeck-251215155422-5497c599-thumbnail.jpg?width=640&height=640&fit=bounds)
Tutorial on Deep Learning
- 1.
- 2. Introduction of Deep Learning! ZaikunXu USI, Master of Informatics HPC Advisory Council Switzerland Conference 2016 2!
- 3. Overview! Introduction to deeplearning! Big data + Big computational power (GPUs)! Latest updates of deep learning! Examples! Feature extraction! 3!
- 4. Machine Learning! source :http://www.cs.utexas.edu/~eladlieb/RLRG.html! Reinforcement learning Introduction to deep learning! 4!
- 5. Inspiration from Human brain! Introductionto deep learning! • Over 100 billion neurons ! • Around 1000 connections ! • Electrical signal transmitted through axon, can be inhibitory or excitatory! • Activation of a neuron depends on inputs! 5!
- 6. Artificial Neuron! • Circles—————————- Neuron! source : http://cs231n.stanford.edu/! • Arrow —————————— Synapse! • Weight —————————- Electrical signal! Introduction to deep learning! 6!
- 7. Nonlinearity! Introduction to deeplearning! • Neural network learns complicated structured data with non-linear activation functions! 7!
- 8. Perceptron! • Perceptron withlinear activation! Introduction to deep learning! • In1969, cannot learn XOR function and takes very long to train! Winter of AI! 8!
- 9. Back-Propogation! • 1970, gotBS on psychology and keep studying NN at University of Edinburg ! Introduction to deep learning! • 1986, Geoffrey Hinton and David Rumelhart, Nature paper "Learning Representations by Back-propagating errors”! • Hidden layers added! • Computation linearly depends on # of neurons ! • Of course, computers at that time is much faster! 9!
- 10. Convolutionary NN! • YannLecun, born at Paris, post-doc of Hinton at U Toronto.! Introduction to deep learning! • LeNet on hand-digital recognition at Bell Labs, 5% error rate! • Attack skill : convolution, pooling! 10!
- 11. Local connection! 11! Introduction todeep learning!
- 12. Weight sharing! 12! Introduction todeep learning!
- 13. Convolution! • Feature maptoo big! 13! Introduction to deep learning!
- 14. Pooling! 14! Introduction to deeplearning! http://vaaaaaanquish.hatenablog.com/entry/2015/01/26/060622!
- 15. Support Vector Machine! Introduction todeep learning! • Vladmir Vapnik, born at Soviet Union, colleague of Yann Lecun , invented SVM at 1963! • Attack skill : Max-margin + kernel mapping! 15!
- 16. Another winter looming! Introductionto deep learning! • SVM : good at “capacity control”, easy to use, repeat! • NN : good at modelling complicated structure! • SVM achieved 0.8% error rate as 1998 and 0.56% in 2002 on the same hand-digit recognition task. ! • NN stops at local optima, hard and takes long to train, overfitting! 16!
- 17. Recurrent Neural Nets! Introductionto deep learning! http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-2-implementing-a-language-model-rnn-with-python-numpy-and-theano/! 17! Model sequential data!
- 18. Vanish gradient! Introduction todeep learning! 18!
- 19.
- 20. 10 years funding! • Hinton was happy and rebranded its neural network to deep learning. The story goes on …………..! Introduction to deep learning! 20!
- 21. Other Pioneers! Introduction todeep learning! 21!
- 22. Data preparation! • Input: Can be image, video, text, sounds….! • Output : Can be a translation of another language, location of an object, a caption describing a image ….! Introduction to deep learning! Training data! Validation data ! Test data ! Model! 22!
- 23. Training ! • Forwardpass, propagating information from input to ouput! Introduction to deep learning! 23!
- 24. Training! • Backward pass,propagating errors back and update weights accordingly! Introduction to deep learning! 24!
- 25. Parameter update! • Gradientdescent (sgd, mini-batch)! Introduction to deep learning! animation by Alec Radford! • x = x - lr * dx! source : http://cs231n.stanford.edu/! 25!
- 26. Parameter update! • Gradientdescent! Introduction to deep learning! • x = x - lr * dx! animation by Alec Radford)! • Second order method ? Quasi-Newton’s method! 26!
- 27. Problems! • While neuralnetworks can approximate any function of input X to output y, it is very hard to train with multiple layers! • Initialization! Introduction to deep learning! • Pre-training! • More data/Dropout to avoid overfitting! • Accelerate with GPU! • Support training in parallel! 27!
- 28. ImageNet! • source : https://devblogs.nvidia.com/parallelforall/nvidia-ibm-cloud-support-imagenet-large-scale-visual-recognition- challenge/! Bigdata + Big computational power! • >1M well labeled images of 1000 classes! 28!
- 29. Big data +Big computational power! 29!
- 30. GPUs! • source : http://techgage.com/article/gtc-2015-in-depth-recap-deep-learning-quadro-m6000-autonomous-driving-more/! Bigdata + Big computational power! 30!
- 31. Parallelization! • Data Parallelization:same weight but different data, gradient needs to be synchronized! Big data + Big computational power! • Not scale well with size of cluster! • Works well on smaller clusters and easy to implement ! source : Nvidia! 31!
- 32. • Model Parallelization:same data, but different parts of weight! Parallelization! Big data + Big computational power! • Parameters in a big Neural net can not fit into memory of a single GPU! Figure from Yi Wang! 32!
- 33. Model+Data Parallelization! Big data+ Big computational power! Krizhevsky et al. 2014! computation! heavy! weight heavy! 33!
- 34. ! • Trained on30M moves + Computation + Clear Reward scheme! Big data + Big computational power! 34!
- 35. • Task :A classifier to recognize hand written digits! Feature extraction! source :https://indico.io/blog/visualizing-with-t-sne/! 35!
- 36. Hierarchal feature extraction! Feature extraction! • source : http://www.rsipvision.com/exploring-deep-learning/! 36!
- 37. Transfer learning! • Usefeatures learned on ImageNet to tune your only classifier on other tasks.! Feature extraction! • 9 categories + 200, 000 images! • extract features from last 2 layers + linear SVM! 37!
- 38. Word embedding! • Embedwords into vector space such that similar words are more close to each other in vector space! Feature extraction! source : http://anthonygarvan.github.io/wordgalaxy/! • Calculate cosine similarity ! Pairs - France+ Italy = Rome! 38!
- 39.
- 40. Go! • 3361 differentlegal positions, roughly 10170! Examples! Brute Force does not work !!!! • Monto Carlo Tree Search ! Heuristic Searching method! Still slow since the search tree is so big! 40!
- 41. AlphaGo! Examples! • Offline learning: from human go players, learn a Deep learning based policy network and a rollout policy • Use self-playing result to train a value network, which evaluate the current situation! 41!
- 42. Online Playing! Examples! • Whenplaying with Lee Sedol, policy network give possible moves (about 10 to 20), the value network calculates the weight of position.! • MCTS update the weight of each possible moves and select one move.! • The engineering effort on systems with CPU and GPU with data , model parallelism matters 42!
- 43. Image recognition! Examples! Train ina supervised fashion !43!
- 44.
- 45. Neural MT! • Themain idea behind RNNs is to compress a sequence of input symbols into a fixed-dimensional vector by using recursion.! Examples! source : https://devblogs.nvidia.com/parallelforall/introduction-neural-machine-translation-with-gpus/! 45!
- 46.
- 47. Go deeper ! Updatesof deep learning! • Deep Residual Learning for Image Recognition, MSRA, 2015!47!
- 48. Go multi-modal! Updates ofdeep learning! 48!
- 49. Go End-End! Updates ofdeep learning! ps://devblogs.nvidia.com/parallelforall/deep-speech-accurate-speech-recognition-gpu-accelerated-deep-learning/#.VO56E8ZtEkM.google_plus 49!
- 50. Go with attention! Updatesof deep learning! 50!
- 51. Go with SpecalizedChips! Updates of deep learning! • Prototypes! 51!
- 52. Go where! • Videounderstanding! Updates of deep learning! • Deep reinforcement learning! • Natural language understanding! • Unsupervised learning !!!! 52!
- 53.
- 54.
- 55.
- 56. Summary! • “The takeawayis that deep learning excels in tasks where the basic unit, a single pixel, a single frequency, or a single word/character has little meaning in and of itself, but a combination of such units has a useful meaning.” — Dallin Akagi! • Structured data + well defined cost function/ rewards ! • learning in a End-End fashion is nice trend! 56!
- 57. Thoughts! • Deep leaningwill change our lives in a positive way! • We learn fast by either competing with friend like AlphaGo, or they directly teach us in a effective way • What might happen in the future! • Still long way to go towards real AI! 57!
- 58. list of materials! • https://github.com/ChristosChristofidis/awesome- deep-learning! 58!
- 59. Acknowledgement! • Tim Dettmers! • Lyu xiyu! 59!