Christian Perone, profile picture
Uploaded byChristian Perone
73,500 views

Deep Learning - Convolutional Neural Networks

This document provides an agenda for a presentation on deep learning, neural networks, convolutional neural networks, and interesting applications. The presentation will include introductions to deep learning and how it differs from traditional machine learning by learning feature representations from data. It will cover the history of neural networks and breakthroughs that enabled training of deeper models. Convolutional neural network architectures will be overviewed, including convolutional, pooling, and dense layers. Applications like recommendation systems, natural language processing, and computer vision will also be discussed. There will be a question and answer section.

Embed presentation

DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A Deep Learning Convolutional Neural Networks Christian S. Perone christian.perone@gmail.com
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHO AM I Christian S. Perone Software Designer Blog http://blog.christianperone.com Open-source projects https://github.com/perone Twitter @tarantulae
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A AGENDA DEEP LEARNING Introduction Traditional vs Deep learning ImageNet Challenge Deep learning in art NEURAL NETWORKS Neural network basics Making it possible CONVOLUTIONAL NEURAL NETWORKS Architecture overview Convolutional layer Pooling layer Dense layers and classification Deep CNNs Important ideas Transfer learning INTERESTING CASES Recommendation Natural language processing Image/video processing Q&A
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A Section I DEEP LEARNING
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT IS DEEP LEARNING ? Multiple definitions, however, these definitions have in common: Multiple layers of processing units; Supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A COMPOSITIONAL DATA NATURAL DATA IS COMPOSITIONAL.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A COMPOSITIONAL DATA Image Source: Convolutional Deep Belief Networks. Honglak Lee, et. al.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A COMPOSITIONAL DATA Sound Source: Large Scale Deep Learning. Jeff Dean, joint work with Google.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ONE SLIDE INTRO TO MACHINE LEARNING Source: Scikit-Learn (scikit-learn.org)
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRADITIONAL VS DEEP LEARNING For many years, we developed feature extractors. Source: Deep Learning Methods for Vision (Honglak Lee)
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRADITIONAL VS DEEP LEARNING Feature extractors, required: Expert knowledge Time-consuming hand-tuning In industrial applications, this is 90% of the time Sometimes are problem specific
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRADITIONAL VS DEEP LEARNING Feature extractors, required: Expert knowledge Time-consuming hand-tuning In industrial applications, this is 90% of the time Sometimes are problem specific But, what if we could learn feature extractors ?
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRADITIONAL VS DEEP LEARNING TRADITIONAL APPROACH The traditional approach uses fixed feature extractors.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRADITIONAL VS DEEP LEARNING TRADITIONAL APPROACH The traditional approach uses fixed feature extractors. DEEP LEARNING APPROACH Deep Learning approach uses trainable feature extractors.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRADITIONAL VS DEEP LEARNING Source: Lee et.al., ICML2009
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A IMAGENET Source: t-SNE visualization of CNN codes. Andrej Karpathy ≈ 20.000 object classes ≈ 14 million images
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A IMAGENET Source: ImageNet
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A IMAGE CLASSIFICATION Source: We’ve Been Dressing Animals Up as People Way Before the Internet. Jes Greene. Image classification, can get really hard.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A IMAGENET CHALLENGE Source: Musings on Deep Learning. Li Jiang.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A DEEP DREAMS Source: Google Inceptionism
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ART STYLE Source: A Neural Algorithm of Artistic Style. Leon A. Gatys et. al.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ART STYLE Source: A Neural Algorithm of Artistic Style. Leon A. Gatys et. al.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ART STYLE Source: A Neural Algorithm of Artistic Style. Leon A. Gatys et. al.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ART STYLE Source: A Neural Algorithm of Artistic Style. Leon A. Gatys et. al.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A Section II NEURAL NETWORKS
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NEURAL NETWORK ARCHITECTURE Source: Neural Networks and Deep Learning. Michael Nielsen. Source: Practical Deep N. Networks. Yuhuang Hu et. al.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A MNIST DIGITS CLASSIFICATION Segmented digits MNIST digit format (28 x 28 = 784 pixels) Source: Neural Networks and Deep Learning. Michael Nielsen.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NEURAL NETWORK ARCHITECTURE Source: Neural Networks and Deep Learning. Michael Nielsen. 2.225 of 10.000 test images (22.25 % accuracy)
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NEURAL NETWORK ARCHITECTURE Source: Neural Networks and Deep Learning. Michael Nielsen. 2.225 of 10.000 test images (22.25 % accuracy) An SVM classifier can get 9.435 of 10.000 ( % 94.35)
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NEURAL NETWORK ARCHITECTURE Source: Neural Networks and Deep Learning. Michael Nielsen. 2.225 of 10.000 test images (22.25 % accuracy) An SVM classifier can get 9.435 of 10.000 ( % 94.35) SVM with hyperparameter optimization can get 98.5%
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NEURAL NETWORK ARCHITECTURE Can we do better ?
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NEURAL NETWORK ARCHITECTURE Can we do better ? In fact, yes. The current record is from 2013 and it classifies 9.979 of 10.000 images correctly. The performance is human-equivalent (or better).
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NEURAL NETWORK ARCHITECTURE Can we do better ? In fact, yes. The current record is from 2013 and it classifies 9.979 of 10.000 images correctly. The performance is human-equivalent (or better). Source: Neural Networks and Deep Learning. Michael Nielsen. Neural networks can accurately classify all but 21 of the 10,000 test images.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? For approximately 20 years, attempts were made to train deeper neural networks (with more than one hidden layer), however rarely with benefits (vanishing gradient).
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? In 2006, a major breakthrough was made in deep architectures, following three key principles: Unsupervised learning of representations is used to pre-train each layer
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? In 2006, a major breakthrough was made in deep architectures, following three key principles: Unsupervised learning of representations is used to pre-train each layer Unsupervised training of one layer at a time, on top of the previously trained ones. The representation learned at each level is the input for the next layer.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? In 2006, a major breakthrough was made in deep architectures, following three key principles: Unsupervised learning of representations is used to pre-train each layer Unsupervised training of one layer at a time, on top of the previously trained ones. The representation learned at each level is the input for the next layer. Use supervised training to fine-tune all the layers (in addition to one or more additional layers that are dedicated to producing predictions).
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? After the 2006 breakthrough, a lot of ideas were also developed. Nowadays, pre-training is almost obsolete.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? After the 2006 breakthrough, a lot of ideas were also developed. Nowadays, pre-training is almost obsolete. New activation functions
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? After the 2006 breakthrough, a lot of ideas were also developed. Nowadays, pre-training is almost obsolete. New activation functions Regularization methods
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? After the 2006 breakthrough, a lot of ideas were also developed. Nowadays, pre-training is almost obsolete. New activation functions Regularization methods Initialization methods
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? After the 2006 breakthrough, a lot of ideas were also developed. Nowadays, pre-training is almost obsolete. New activation functions Regularization methods Initialization methods Data augmentation
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? After the 2006 breakthrough, a lot of ideas were also developed. Nowadays, pre-training is almost obsolete. New activation functions Regularization methods Initialization methods Data augmentation Optimization techniques
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? Another reason on why Deep Learning is possible, is the availability of lots of data (i.e. ImageNet).
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? Another reason on why Deep Learning is possible, is the availability of lots of data (i.e. ImageNet). GPGPU also plays an important role on this. For instance, an NVIDIA GPU (1 Tesla K40 GPU) training a 7 layer Convolutional Neural Network is nearly 9x faster than CPU. Convolutions — 80-90% of execution time Pooling Activations
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? Companies are working on solutions for Deep Learning acceleration:
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? Companies are working on solutions for Deep Learning acceleration: NVIDIA NVIDIA created a entire plaftorm stack dedicated to work with Deep Learning, called DIGITS. Their GPUs are widely used in Deep Learning.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? Companies are working on solutions for Deep Learning acceleration: NVIDIA NVIDIA created a entire plaftorm stack dedicated to work with Deep Learning, called DIGITS. Their GPUs are widely used in Deep Learning. AMAZON Amazon AWS also create EC2 instances with NVIDIA GPUs (with 4GB of memory and 1536 CUDA cores). Lots of AMIs with Deep Learning software ecosystem already installed.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? Companies are working on solutions for Deep Learning acceleration: NVIDIA NVIDIA created a entire plaftorm stack dedicated to work with Deep Learning, called DIGITS. Their GPUs are widely used in Deep Learning. AMAZON Amazon AWS also create EC2 instances with NVIDIA GPUs (with 4GB of memory and 1536 CUDA cores). Lots of AMIs with Deep Learning software ecosystem already installed. MICROSOFT Microsoft announced that it will offer NVIDIA GPUs on its Azure cloud platform.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A Section III CONVOLUTIONAL NEURAL NETWORKS
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A CONVOLUTIONAL NEURAL NETWORKS Convolutional Neural Networks (or convnets) are based on the following principles: Local receptive fields Shared weights Pooling (or down-sampling) This special neural network architecture takes advantage of the spatial structure of data.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A CONVOLUTIONAL NEURAL NETWORKS Convolutional Neural Networks (or convnets) are based on the following principles: Local receptive fields Shared weights Pooling (or down-sampling) This special neural network architecture takes advantage of the spatial structure of data. Source: Deeply-Supervised Nets. Zhuowen Tu.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A LOCAL CONNECTIVITY Let’s take the MNIST digits images as input of our convnet. These images are 28x28 pixels: 28x28 image Local connectivity (5x5) Source: Neural Networks and Deep Learning. Michael Nielsen.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A LOCAL CONNECTIVITY
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A LOCAL CONNECTIVITY By “sliding” it, we create a feature map of or 24x24 neurons in the hidden layer. We can also have a different stride and padding.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A SHARED WEIGHTS In this local receptive field, Convolutional Neural Networks use the same shared weights for each of the 24x24 hidden neurons. This means that we have a great advantage of parameter reduction, for instance, for a 5x5 receptive field, we’ll need only 25 shared weights1. 1 Excluding the bias
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A SHARED WEIGHTS In this local receptive field, Convolutional Neural Networks use the same shared weights for each of the 24x24 hidden neurons. This means that we have a great advantage of parameter reduction, for instance, for a 5x5 receptive field, we’ll need only 25 shared weights1. 20 feature maps using 5x5 — 20*26 = 520 weights A fully connected first layer, with 784=28*28 input neurons, and a relatively modest 30 hidden neurons, would produce 784*30 = 23.520 weights, more than 40 times as many parameters as the convolutional layer. 1 Excluding the bias
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A CONVOLUTIONAL LAYER The shared weights and bias are called kernel or filter. Convolutional layers provides translation invariance. Since these filters works on every part of the image, they are “searching” for the same feature everywhere in the image.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ARCHITECTURE OVERVIEW Input layer – 28x28 pixels Convolutional layer — 3 feature maps (5x5 kernel)
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A POOLING LAYER Pooling layers are usually present after a convolutional layer. They provide a down-sampling of the convolution output. In the example above, a 2x2 region is being used as input of the pooling.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A POOLING LAYER There are different types of pooling, the most used is the max-pooling and average pooling: Pooling layers downsamples the volume spatially, reducing small translations of the features. They also provide a parameter reduction. Source: CS231n Convolutional Neural Networks for Visual Recognition.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A POOLING LAYER Max-pooling is how the network asks whether a feature is found anywhere in some region of the image. After that, it will lose the exact position.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ARCHITECTURE OVERVIEW Input layer – 28x28 pixels Convolutional layer — 3 feature maps (5x5 kernel) Pooling Layer — 2x2, resulting in 3x12x12
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A CLASSIFICATION As you can see, we then add a dense fully-connected layer (usually using softmax) at the end of the neural network in order to get predictions for the problem we’re working on (10 classes, 10 digits).
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A GOING DEEPER We have defined all the components required to create a Convolutional Neural Network, but you’ll rarely see a shallow convnet like that.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A GOING DEEPER Actually, experiments demonstrated that the replication of convolutional + pooling layers produces better results the deeper you go. Winners of ImageNet challenge, have more than 15 layers (VGGNet has 19 layers).
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A GOING DEEPER Source: Rotation-invariant convolutional neural networks for galaxy morphology prediction. Sander Dieleman et. al. Galaxy Zoo best performing network (winner of the challenge).
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A DROPOUT TECHNIQUE Source: “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”. Nitish Srivastava et. al. The dropout technique helps with the overfitting, specially on dense layers. Drop occur only at training time, not on test time.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ACTIVATION FUNCTIONS Source: Big Data Analytics. Fei Wang. ReLu helps with the vanishing gradient problem ReLu generates sparsity
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A DATA AUGMENTATION Data augmentation can help with overfitting and will certainly improve improve results. Source: Rotation-invariant convolutional neural networks for galaxy morphology prediction. Sander Dieleman et. al.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A DATA AUGMENTATION Data augmentation can help with overfitting and will certainly improve improve results. Source: Rotation-invariant convolutional neural networks for galaxy morphology prediction. Sander Dieleman et. al. Small rotations Small translation Scaling Flipping Brightness Noise
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRANSFER LEARNING Features learned by Convolutional Neural Networks on large dataset problem (i.e. ImageNet), can be helpful on different problems. It’s very common to pre-train a convnet on ImageNet and then use it as a fixed feature extractor or as initialization. CONVNETS AS FEATURE EXTRACTORS We can remove the last layer and then use these features to extract features, these features are very useful features for classification. Some people use these features with LSH (locality-sensitive hashing) to scale large databases for image search. You can also use these features as input for a SVM classifier for instance.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRANSFER LEARNING Features learned by Convolutional Neural Networks on large dataset problem (i.e. ImageNet), can be helpful on different problems. It’s very common to pre-train a convnet on ImageNet and then use it as a fixed feature extractor or as initialization. FINE-TUNING THE CONVNETS You can use a pre-trained convnet to continue its training on your data and thus fine-tune the weights for your problem. First layers of a convnet contains generic features (i.e. edge detectors, etc.) that should be helpful in many tasks. Deeper layers becomes progressively specific to the details of the classes of the original problem.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A Section IV INTERESTING CASES
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A MUSIC RECOMMENDATION Source: Recommending music on Spotify with deep learning. Sander Dieleman. This is an example architecture from Spotify, using Convolutional Neural Network for music recommendation.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A MUSIC RECOMMENDATION Learned filters at first convolutional layer. The time axis is horizontal, the frequency axis is vertical (frequency increases from top to bottom)
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NATURAL LANGUAGE PROCESSING Source: Text understanding from scratch. Xiang Zhang, Yann LeCun. Deciding if a review posted on Amazon is positive or negative with 96% accuracy, and predict the actual number of stars with 73% accuracy.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WORD2VEC Source: Distributed Representations of Sentences and Documents. Quoc Le, Tomas Mikolov. Word vectors (trained with up to hundreds of billions of words).
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WORD2VEC Source: Distributed Representations of Sentences and Documents. Quoc Le, Tomas Mikolov. Word vectors (trained with up to hundreds of billions of words). With nice properties: v(’Paris’) - v(’France’) + v(’Italy’ ) ≈ v(’Rome’) v(’king’) - v(’man’) + v(’woman’) ≈ v(’queen’) No deep learning and no convnet, but a great distributed representation example.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WORD2VEC Machine Translation Source: DL4J.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A DOC2VEC Sentiment analysis Source: Distributed Representations of Sentences and Documents. Quoc Le, Tomas Mikolov.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A INTERESTING FRAMES Google recently put in production a Deep Neural Network to improve YouTube video thumbnails. Source: Google Research.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A SPATIAL TRANSFORMER NETWOKRS Spatial Transformer Networks can learn transformations. Source: Spatial Transformer Networks. Max Jaderberg, et. al.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A DEEPFACE BY FACEBOOK Source: DeepFace: Closing the Gap to Human-Level Performance in Face Verification. Yaniv Taigman, et. al.
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A Section V Q&A
DEEP LEARNING NEURAL NETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A Q&A

More Related Content

PPTX
Convolutional neural network
byFerdous ahmed
 
PPTX
Convolutional Neural Network (CNN) - image recognition
byYUNG-KUEI CHEN
 
PPTX
CNN and its applications by ketaki
byKetaki Patwari
 
PPTX
Machine Learning - Convolutional Neural Network
byRichard Kuo
 
PDF
Convolutional neural network
byYan Xu
 
PPTX
Deep Learning - CNN and RNN
byAshray Bhandare
 
PPT
Cnn method
byAmirSajedi1
 
PPTX
CONVOLUTIONAL NEURAL NETWORK
byMd Rajib Bhuiyan
 
Convolutional neural network
byFerdous ahmed
 
Convolutional Neural Network (CNN) - image recognition
byYUNG-KUEI CHEN
 
CNN and its applications by ketaki
byKetaki Patwari
 
Machine Learning - Convolutional Neural Network
byRichard Kuo
 
Convolutional neural network
byYan Xu
 
Deep Learning - CNN and RNN
byAshray Bhandare
 
Cnn method
byAmirSajedi1
 
CONVOLUTIONAL NEURAL NETWORK
byMd Rajib Bhuiyan
 

What's hot

PPTX
Deep Learning Tutorial
byAmr Rashed
 
PPTX
What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...
bySimplilearn
 
PDF
Convolutional Neural Network Models - Deep Learning
byMohamed Loey
 
PDF
Neural networks and deep learning
byJörgen Sandig
 
PPTX
Introduction to Deep learning
byleopauly
 
PPT
Deep learning ppt
byBalneSridevi
 
PPTX
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...
bySimplilearn
 
PPTX
Intro to deep learning
byDavid Voyles
 
PPTX
Deep neural networks
bySi Haem
 
PDF
Training Neural Networks
byDatabricks
 
PPTX
Introduction to CNN
byShuai Zhang
 
PDF
Deep learning - A Visual Introduction
byLukas Masuch
 
PPTX
CNN Machine learning DeepLearning
byAbhishek Sharma
 
PPTX
Introduction to Deep Learning
byOswald Campesato
 
PPTX
Deep Learning With Neural Networks
byAniket Maurya
 
PPTX
Convolutional neural network
byMojammilHusain
 
PPTX
CNN Tutorial
bySungjoon Choi
 
PDF
Image segmentation with deep learning
byAntonio Rueda-Toicen
 
PPTX
Convolutional Neural Network
byVignesh Suresh
 
PPTX
Image classification with Deep Neural Networks
byYogendra Tamang
 
Deep Learning Tutorial
byAmr Rashed
 
What Is Deep Learning? | Introduction to Deep Learning | Deep Learning Tutori...
bySimplilearn
 
Convolutional Neural Network Models - Deep Learning
byMohamed Loey
 
Neural networks and deep learning
byJörgen Sandig
 
Introduction to Deep learning
byleopauly
 
Deep learning ppt
byBalneSridevi
 
Convolutional Neural Network - CNN | How CNN Works | Deep Learning Course | S...
bySimplilearn
 
Intro to deep learning
byDavid Voyles
 
Deep neural networks
bySi Haem
 
Training Neural Networks
byDatabricks
 
Introduction to CNN
byShuai Zhang
 
Deep learning - A Visual Introduction
byLukas Masuch
 
CNN Machine learning DeepLearning
byAbhishek Sharma
 
Introduction to Deep Learning
byOswald Campesato
 
Deep Learning With Neural Networks
byAniket Maurya
 
Convolutional neural network
byMojammilHusain
 
CNN Tutorial
bySungjoon Choi
 
Image segmentation with deep learning
byAntonio Rueda-Toicen
 
Convolutional Neural Network
byVignesh Suresh
 
Image classification with Deep Neural Networks
byYogendra Tamang
 

Viewers also liked

PPTX
Deep Learning - Convolutional Neural Networks - Architectural Zoo
byChristian Perone
 
PDF
Deep Learning And Business Models (VNITC 2015-09-13)
byHa Phuong
 
PDF
Intro To Convolutional Neural Networks
byMark Scully
 
PDF
Word Embeddings - Introduction
byChristian Perone
 
PPTX
Moving object detection
byRaviraj singh shekhawat
 
PDF
An Introduction to Deep Learning
byPoo Kuan Hoong
 
PPTX
Deep Learning Models for Question Answering
bySujit Pal
 
PDF
Intro to Deep Learning for Question Answering
byTraian Rebedea
 
PPTX
Deep Learning for Data Scientists - Data Science ATL Meetup Presentation, 201...
byAndrew Gardner
 
PDF
Deep learning - Conceptual understanding and applications
byBuhwan Jeong
 
PPTX
Artificial Intelligence, Machine Learning and Deep Learning
bySujit Pal
 
PDF
Deep Learning through Examples
bySri Ambati
 
PDF
Transform your Business with AI, Deep Learning and Machine Learning
bySri Ambati
 
PPTX
AI and the Future of Growth
byAccenture Technology
 
Deep Learning - Convolutional Neural Networks - Architectural Zoo
byChristian Perone
 
Deep Learning And Business Models (VNITC 2015-09-13)
byHa Phuong
 
Intro To Convolutional Neural Networks
byMark Scully
 
Word Embeddings - Introduction
byChristian Perone
 
Moving object detection
byRaviraj singh shekhawat
 
An Introduction to Deep Learning
byPoo Kuan Hoong
 
Deep Learning Models for Question Answering
bySujit Pal
 
Intro to Deep Learning for Question Answering
byTraian Rebedea
 
Deep Learning for Data Scientists - Data Science ATL Meetup Presentation, 201...
byAndrew Gardner
 
Deep learning - Conceptual understanding and applications
byBuhwan Jeong
 
Artificial Intelligence, Machine Learning and Deep Learning
bySujit Pal
 
Deep Learning through Examples
bySri Ambati
 
Transform your Business with AI, Deep Learning and Machine Learning
bySri Ambati
 
AI and the Future of Growth
byAccenture Technology
 

Similar to Deep Learning - Convolutional Neural Networks

PDF
Deep Learning: a birds eye view
byRoelof Pieters
 
PPTX
Introduction to deep learning
bydoppenhe
 
PDF
MLIP - Chapter 3 - Introduction to deep learning
byCharles Deledalle
 
PDF
Yann le cun
byYandex
 
PPTX
Introduction to Deep learning
byMassimiliano Patacchiola
 
PPTX
Strata London - Deep Learning 05-2015
byTuri, Inc.
 
PDF
Understanding Convolutional Neural Networks
byJeremy Nixon
 
PDF
AINL 2016: Filchenkov
byLidia Pivovarova
 
PDF
DLD meetup 2017, Efficient Deep Learning
byBrodmann17
 
PPT
lecun-01.ppt
byVenkyChinna8
 
PDF
imageclassification-160206090009.pdf
byKammetaJoshna
 
PDF
Introduction to Deep Learning: Concepts, Architectures, and Applications
byAmr Rashed
 
PPTX
Deep learning tutorial 9/2019
byAmr Rashed
 
PDF
PyData2015
byMatthew Opala
 
PPTX
Deep Learning: Towards General Artificial Intelligence
byRukshan Batuwita
 
PDF
Deep Learning AtoC with Image Perspective
byDong Heon Cho
 
PPTX
Learn to Build an App to Find Similar Images using Deep Learning- Piotr Teterwak
byPyData
 
PPTX
Deep Learning for Artificial Intelligence (AI)
byEr. Shiva K. Shrestha
 
PDF
Introducing Deep Learning - Mélanie Ducoffe (UNS-CNRS-I3S)
byTelecomValley
 
PDF
Deep Learning - Overview of my work II
byMohamed Loey
 
Deep Learning: a birds eye view
byRoelof Pieters
 
Introduction to deep learning
bydoppenhe
 
MLIP - Chapter 3 - Introduction to deep learning
byCharles Deledalle
 
Yann le cun
byYandex
 
Introduction to Deep learning
byMassimiliano Patacchiola
 
Strata London - Deep Learning 05-2015
byTuri, Inc.
 
Understanding Convolutional Neural Networks
byJeremy Nixon
 
AINL 2016: Filchenkov
byLidia Pivovarova
 
DLD meetup 2017, Efficient Deep Learning
byBrodmann17
 
lecun-01.ppt
byVenkyChinna8
 
imageclassification-160206090009.pdf
byKammetaJoshna
 
Introduction to Deep Learning: Concepts, Architectures, and Applications
byAmr Rashed
 
Deep learning tutorial 9/2019
byAmr Rashed
 
PyData2015
byMatthew Opala
 
Deep Learning: Towards General Artificial Intelligence
byRukshan Batuwita
 
Deep Learning AtoC with Image Perspective
byDong Heon Cho
 
Learn to Build an App to Find Similar Images using Deep Learning- Piotr Teterwak
byPyData
 
Deep Learning for Artificial Intelligence (AI)
byEr. Shiva K. Shrestha
 
Introducing Deep Learning - Mélanie Ducoffe (UNS-CNRS-I3S)
byTelecomValley
 
Deep Learning - Overview of my work II
byMohamed Loey
 

More from Christian Perone

PDF
PyTorch 2 Internals
byChristian Perone
 
PDF
Gradient-based optimization for Deep Learning: a short introduction
byChristian Perone
 
PDF
Bayesian modelling for COVID-19 seroprevalence studies
byChristian Perone
 
PDF
Uncertainty Estimation in Deep Learning
byChristian Perone
 
PDF
PyTorch under the hood
byChristian Perone
 
PDF
Apache Spark - Intro to Large-scale recommendations with Apache Spark and Python
byChristian Perone
 
PDF
Machine Learning com Python e Scikit-learn
byChristian Perone
 
PDF
Python - Introdução Básica
byChristian Perone
 
PDF
C++0x :: Introduction to some amazing features
byChristian Perone
 
PyTorch 2 Internals
byChristian Perone
 
Gradient-based optimization for Deep Learning: a short introduction
byChristian Perone
 
Bayesian modelling for COVID-19 seroprevalence studies
byChristian Perone
 
Uncertainty Estimation in Deep Learning
byChristian Perone
 
PyTorch under the hood
byChristian Perone
 
Apache Spark - Intro to Large-scale recommendations with Apache Spark and Python
byChristian Perone
 
Machine Learning com Python e Scikit-learn
byChristian Perone
 
Python - Introdução Básica
byChristian Perone
 
C++0x :: Introduction to some amazing features
byChristian Perone
 

Recently uploaded

PDF
AI Meets Integration: Delivering AI-Ready Data in Real Time.pdf
byPrecisely
 
PPTX
Cybersecurity Best Practices - Step by Step guidelines
byYasir Naveed Riaz
 
PPTX
cybercrime in Information security .pptx
byismaeelsahib032
 
PDF
Cross-Cultural Agile Development -Challenges and Strategies for Overcoming Them-
byTakashi Makino
 
PDF
Unser Jahresrückblick – MarvelClient in 2025
bypanagenda
 
PDF
Decoding the DNA: The Digital Networks Act, the Open Internet, and IP interco...
byCSUC - Consorci de Serveis Universitaris de Catalunya
 
PDF
Security Forum Sessions from Houston 2025 Event
byMark Simos
 
PDF
Dev Dives: AI that builds with you - UiPath Autopilot for effortless RPA & AP...
byUiPathCommunity
 
PDF
Unlocking the Power of Salesforce Architecture: Frameworks for Effective Solu...
byvarsha30tiwari
 
PPTX
Unit-4-ARTIFICIAL NEURAL NETWORKS.pptx ANN ppt Artificial neural network
byssuserd4481a
 
PDF
Cybersecurity: Safeguarding Digital Assets
byYasir Naveed Riaz
 
PPTX
Building Cyber Resilience for 2026: Best Practices for a Secure, AI-Driven Bu...
byYasir Naveed Riaz
 
PDF
Eredità digitale sugli smartphone: cosa resta di noi nei dispositivi mobili
bySpeck&Tech
 
PDF
Six Shifts For 2026 (And The Next Six Years)
byDavid Armano
 
PPTX
Software Analysis &Design ethiopia chap-2.pptx
byAbbas484771
 
PPTX
UiPath Autonomous Agents | Building and Orchestrating Agents End-to-End
byUiPathCommunity
 
PPTX
Cybercrime in the Digital Age: Risks, Impact & Protection
byYasir Naveed Riaz
 
PPTX
From Backup to Resilience: How MSPs Are Preparing for 2026
byMSP360
 
PPTX
wob-report.pptxwob-report.pptxwob-report.pptx
byssuser0d171c
 
PPTX
Why Most GenAI Projects Fail to Scale and How to Become One of the Success St...
byEarley Information Science
 
AI Meets Integration: Delivering AI-Ready Data in Real Time.pdf
byPrecisely
 
Cybersecurity Best Practices - Step by Step guidelines
byYasir Naveed Riaz
 
cybercrime in Information security .pptx
byismaeelsahib032
 
Cross-Cultural Agile Development -Challenges and Strategies for Overcoming Them-
byTakashi Makino
 
Unser Jahresrückblick – MarvelClient in 2025
bypanagenda
 
Decoding the DNA: The Digital Networks Act, the Open Internet, and IP interco...
byCSUC - Consorci de Serveis Universitaris de Catalunya
 
Security Forum Sessions from Houston 2025 Event
byMark Simos
 
Dev Dives: AI that builds with you - UiPath Autopilot for effortless RPA & AP...
byUiPathCommunity
 
Unlocking the Power of Salesforce Architecture: Frameworks for Effective Solu...
byvarsha30tiwari
 
Unit-4-ARTIFICIAL NEURAL NETWORKS.pptx ANN ppt Artificial neural network
byssuserd4481a
 
Cybersecurity: Safeguarding Digital Assets
byYasir Naveed Riaz
 
Building Cyber Resilience for 2026: Best Practices for a Secure, AI-Driven Bu...
byYasir Naveed Riaz
 
Eredità digitale sugli smartphone: cosa resta di noi nei dispositivi mobili
bySpeck&Tech
 
Six Shifts For 2026 (And The Next Six Years)
byDavid Armano
 
Software Analysis &Design ethiopia chap-2.pptx
byAbbas484771
 
UiPath Autonomous Agents | Building and Orchestrating Agents End-to-End
byUiPathCommunity
 
Cybercrime in the Digital Age: Risks, Impact & Protection
byYasir Naveed Riaz
 
From Backup to Resilience: How MSPs Are Preparing for 2026
byMSP360
 
wob-report.pptxwob-report.pptxwob-report.pptx
byssuser0d171c
 
Why Most GenAI Projects Fail to Scale and How to Become One of the Success St...
byEarley Information Science
 

Deep Learning - Convolutional Neural Networks

  • 1.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A Deep Learning Convolutional Neural Networks Christian S. Perone christian.perone@gmail.com
  • 2.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHO AM I Christian S. Perone Software Designer Blog http://blog.christianperone.com Open-source projects https://github.com/perone Twitter @tarantulae
  • 3.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A AGENDA DEEP LEARNING Introduction Traditional vs Deep learning ImageNet Challenge Deep learning in art NEURAL NETWORKS Neural network basics Making it possible CONVOLUTIONAL NEURAL NETWORKS Architecture overview Convolutional layer Pooling layer Dense layers and classification Deep CNNs Important ideas Transfer learning INTERESTING CASES Recommendation Natural language processing Image/video processing Q&A
  • 4.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A Section I DEEP LEARNING
  • 5.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT IS DEEP LEARNING ? Multiple definitions, however, these definitions have in common: Multiple layers of processing units; Supervised or unsupervised learning of feature representations in each layer, with the layers forming a hierarchy from low-level to high-level features.
  • 6.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A COMPOSITIONAL DATA NATURAL DATA IS COMPOSITIONAL.
  • 7.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A COMPOSITIONAL DATA Image Source: Convolutional Deep Belief Networks. Honglak Lee, et. al.
  • 8.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A COMPOSITIONAL DATA Sound Source: Large Scale Deep Learning. Jeff Dean, joint work with Google.
  • 9.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ONE SLIDE INTRO TO MACHINE LEARNING Source: Scikit-Learn (scikit-learn.org)
  • 10.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRADITIONAL VS DEEP LEARNING For many years, we developed feature extractors. Source: Deep Learning Methods for Vision (Honglak Lee)
  • 11.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRADITIONAL VS DEEP LEARNING Feature extractors, required: Expert knowledge Time-consuming hand-tuning In industrial applications, this is 90% of the time Sometimes are problem specific
  • 12.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRADITIONAL VS DEEP LEARNING Feature extractors, required: Expert knowledge Time-consuming hand-tuning In industrial applications, this is 90% of the time Sometimes are problem specific But, what if we could learn feature extractors ?
  • 13.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRADITIONAL VS DEEP LEARNING TRADITIONAL APPROACH The traditional approach uses fixed feature extractors.
  • 14.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRADITIONAL VS DEEP LEARNING TRADITIONAL APPROACH The traditional approach uses fixed feature extractors. DEEP LEARNING APPROACH Deep Learning approach uses trainable feature extractors.
  • 15.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRADITIONAL VS DEEP LEARNING Source: Lee et.al., ICML2009
  • 16.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A IMAGENET Source: t-SNE visualization of CNN codes. Andrej Karpathy ≈ 20.000 object classes ≈ 14 million images
  • 17.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A IMAGENET Source: ImageNet
  • 18.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A IMAGE CLASSIFICATION Source: We’ve Been Dressing Animals Up as People Way Before the Internet. Jes Greene. Image classification, can get really hard.
  • 19.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A IMAGENET CHALLENGE Source: Musings on Deep Learning. Li Jiang.
  • 20.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A DEEP DREAMS Source: Google Inceptionism
  • 21.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ART STYLE Source: A Neural Algorithm of Artistic Style. Leon A. Gatys et. al.
  • 22.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ART STYLE Source: A Neural Algorithm of Artistic Style. Leon A. Gatys et. al.
  • 23.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ART STYLE Source: A Neural Algorithm of Artistic Style. Leon A. Gatys et. al.
  • 24.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ART STYLE Source: A Neural Algorithm of Artistic Style. Leon A. Gatys et. al.
  • 25.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A Section II NEURAL NETWORKS
  • 26.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NEURAL NETWORK ARCHITECTURE Source: Neural Networks and Deep Learning. Michael Nielsen. Source: Practical Deep N. Networks. Yuhuang Hu et. al.
  • 27.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A MNIST DIGITS CLASSIFICATION Segmented digits MNIST digit format (28 x 28 = 784 pixels) Source: Neural Networks and Deep Learning. Michael Nielsen.
  • 28.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NEURAL NETWORK ARCHITECTURE Source: Neural Networks and Deep Learning. Michael Nielsen. 2.225 of 10.000 test images (22.25 % accuracy)
  • 29.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NEURAL NETWORK ARCHITECTURE Source: Neural Networks and Deep Learning. Michael Nielsen. 2.225 of 10.000 test images (22.25 % accuracy) An SVM classifier can get 9.435 of 10.000 ( % 94.35)
  • 30.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NEURAL NETWORK ARCHITECTURE Source: Neural Networks and Deep Learning. Michael Nielsen. 2.225 of 10.000 test images (22.25 % accuracy) An SVM classifier can get 9.435 of 10.000 ( % 94.35) SVM with hyperparameter optimization can get 98.5%
  • 31.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NEURAL NETWORK ARCHITECTURE Can we do better ?
  • 32.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NEURAL NETWORK ARCHITECTURE Can we do better ? In fact, yes. The current record is from 2013 and it classifies 9.979 of 10.000 images correctly. The performance is human-equivalent (or better).
  • 33.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NEURAL NETWORK ARCHITECTURE Can we do better ? In fact, yes. The current record is from 2013 and it classifies 9.979 of 10.000 images correctly. The performance is human-equivalent (or better). Source: Neural Networks and Deep Learning. Michael Nielsen. Neural networks can accurately classify all but 21 of the 10,000 test images.
  • 34.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? For approximately 20 years, attempts were made to train deeper neural networks (with more than one hidden layer), however rarely with benefits (vanishing gradient).
  • 35.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? In 2006, a major breakthrough was made in deep architectures, following three key principles: Unsupervised learning of representations is used to pre-train each layer
  • 36.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? In 2006, a major breakthrough was made in deep architectures, following three key principles: Unsupervised learning of representations is used to pre-train each layer Unsupervised training of one layer at a time, on top of the previously trained ones. The representation learned at each level is the input for the next layer.
  • 37.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? In 2006, a major breakthrough was made in deep architectures, following three key principles: Unsupervised learning of representations is used to pre-train each layer Unsupervised training of one layer at a time, on top of the previously trained ones. The representation learned at each level is the input for the next layer. Use supervised training to fine-tune all the layers (in addition to one or more additional layers that are dedicated to producing predictions).
  • 38.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? After the 2006 breakthrough, a lot of ideas were also developed. Nowadays, pre-training is almost obsolete.
  • 39.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? After the 2006 breakthrough, a lot of ideas were also developed. Nowadays, pre-training is almost obsolete. New activation functions
  • 40.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? After the 2006 breakthrough, a lot of ideas were also developed. Nowadays, pre-training is almost obsolete. New activation functions Regularization methods
  • 41.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? After the 2006 breakthrough, a lot of ideas were also developed. Nowadays, pre-training is almost obsolete. New activation functions Regularization methods Initialization methods
  • 42.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? After the 2006 breakthrough, a lot of ideas were also developed. Nowadays, pre-training is almost obsolete. New activation functions Regularization methods Initialization methods Data augmentation
  • 43.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? After the 2006 breakthrough, a lot of ideas were also developed. Nowadays, pre-training is almost obsolete. New activation functions Regularization methods Initialization methods Data augmentation Optimization techniques
  • 44.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? Another reason on why Deep Learning is possible, is the availability of lots of data (i.e. ImageNet).
  • 45.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? Another reason on why Deep Learning is possible, is the availability of lots of data (i.e. ImageNet). GPGPU also plays an important role on this. For instance, an NVIDIA GPU (1 Tesla K40 GPU) training a 7 layer Convolutional Neural Network is nearly 9x faster than CPU. Convolutions — 80-90% of execution time Pooling Activations
  • 46.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? Companies are working on solutions for Deep Learning acceleration:
  • 47.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? Companies are working on solutions for Deep Learning acceleration: NVIDIA NVIDIA created a entire plaftorm stack dedicated to work with Deep Learning, called DIGITS. Their GPUs are widely used in Deep Learning.
  • 48.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? Companies are working on solutions for Deep Learning acceleration: NVIDIA NVIDIA created a entire plaftorm stack dedicated to work with Deep Learning, called DIGITS. Their GPUs are widely used in Deep Learning. AMAZON Amazon AWS also create EC2 instances with NVIDIA GPUs (with 4GB of memory and 1536 CUDA cores). Lots of AMIs with Deep Learning software ecosystem already installed.
  • 49.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WHAT CHANGED ? Companies are working on solutions for Deep Learning acceleration: NVIDIA NVIDIA created a entire plaftorm stack dedicated to work with Deep Learning, called DIGITS. Their GPUs are widely used in Deep Learning. AMAZON Amazon AWS also create EC2 instances with NVIDIA GPUs (with 4GB of memory and 1536 CUDA cores). Lots of AMIs with Deep Learning software ecosystem already installed. MICROSOFT Microsoft announced that it will offer NVIDIA GPUs on its Azure cloud platform.
  • 50.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A Section III CONVOLUTIONAL NEURAL NETWORKS
  • 51.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A CONVOLUTIONAL NEURAL NETWORKS Convolutional Neural Networks (or convnets) are based on the following principles: Local receptive fields Shared weights Pooling (or down-sampling) This special neural network architecture takes advantage of the spatial structure of data.
  • 52.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A CONVOLUTIONAL NEURAL NETWORKS Convolutional Neural Networks (or convnets) are based on the following principles: Local receptive fields Shared weights Pooling (or down-sampling) This special neural network architecture takes advantage of the spatial structure of data. Source: Deeply-Supervised Nets. Zhuowen Tu.
  • 53.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A LOCAL CONNECTIVITY Let’s take the MNIST digits images as input of our convnet. These images are 28x28 pixels: 28x28 image Local connectivity (5x5) Source: Neural Networks and Deep Learning. Michael Nielsen.
  • 54.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A LOCAL CONNECTIVITY
  • 55.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A LOCAL CONNECTIVITY By “sliding” it, we create a feature map of or 24x24 neurons in the hidden layer. We can also have a different stride and padding.
  • 56.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A SHARED WEIGHTS In this local receptive field, Convolutional Neural Networks use the same shared weights for each of the 24x24 hidden neurons. This means that we have a great advantage of parameter reduction, for instance, for a 5x5 receptive field, we’ll need only 25 shared weights1. 1 Excluding the bias
  • 57.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A SHARED WEIGHTS In this local receptive field, Convolutional Neural Networks use the same shared weights for each of the 24x24 hidden neurons. This means that we have a great advantage of parameter reduction, for instance, for a 5x5 receptive field, we’ll need only 25 shared weights1. 20 feature maps using 5x5 — 20*26 = 520 weights A fully connected first layer, with 784=28*28 input neurons, and a relatively modest 30 hidden neurons, would produce 784*30 = 23.520 weights, more than 40 times as many parameters as the convolutional layer. 1 Excluding the bias
  • 58.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A CONVOLUTIONAL LAYER The shared weights and bias are called kernel or filter. Convolutional layers provides translation invariance. Since these filters works on every part of the image, they are “searching” for the same feature everywhere in the image.
  • 59.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ARCHITECTURE OVERVIEW Input layer – 28x28 pixels Convolutional layer — 3 feature maps (5x5 kernel)
  • 60.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A POOLING LAYER Pooling layers are usually present after a convolutional layer. They provide a down-sampling of the convolution output. In the example above, a 2x2 region is being used as input of the pooling.
  • 61.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A POOLING LAYER There are different types of pooling, the most used is the max-pooling and average pooling: Pooling layers downsamples the volume spatially, reducing small translations of the features. They also provide a parameter reduction. Source: CS231n Convolutional Neural Networks for Visual Recognition.
  • 62.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A POOLING LAYER Max-pooling is how the network asks whether a feature is found anywhere in some region of the image. After that, it will lose the exact position.
  • 63.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ARCHITECTURE OVERVIEW Input layer – 28x28 pixels Convolutional layer — 3 feature maps (5x5 kernel) Pooling Layer — 2x2, resulting in 3x12x12
  • 64.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A CLASSIFICATION As you can see, we then add a dense fully-connected layer (usually using softmax) at the end of the neural network in order to get predictions for the problem we’re working on (10 classes, 10 digits).
  • 65.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A GOING DEEPER We have defined all the components required to create a Convolutional Neural Network, but you’ll rarely see a shallow convnet like that.
  • 66.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A GOING DEEPER Actually, experiments demonstrated that the replication of convolutional + pooling layers produces better results the deeper you go. Winners of ImageNet challenge, have more than 15 layers (VGGNet has 19 layers).
  • 67.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A GOING DEEPER Source: Rotation-invariant convolutional neural networks for galaxy morphology prediction. Sander Dieleman et. al. Galaxy Zoo best performing network (winner of the challenge).
  • 68.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A DROPOUT TECHNIQUE Source: “Dropout: A Simple Way to Prevent Neural Networks from Overfitting”. Nitish Srivastava et. al. The dropout technique helps with the overfitting, specially on dense layers. Drop occur only at training time, not on test time.
  • 69.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A ACTIVATION FUNCTIONS Source: Big Data Analytics. Fei Wang. ReLu helps with the vanishing gradient problem ReLu generates sparsity
  • 70.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A DATA AUGMENTATION Data augmentation can help with overfitting and will certainly improve improve results. Source: Rotation-invariant convolutional neural networks for galaxy morphology prediction. Sander Dieleman et. al.
  • 71.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A DATA AUGMENTATION Data augmentation can help with overfitting and will certainly improve improve results. Source: Rotation-invariant convolutional neural networks for galaxy morphology prediction. Sander Dieleman et. al. Small rotations Small translation Scaling Flipping Brightness Noise
  • 72.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRANSFER LEARNING Features learned by Convolutional Neural Networks on large dataset problem (i.e. ImageNet), can be helpful on different problems. It’s very common to pre-train a convnet on ImageNet and then use it as a fixed feature extractor or as initialization. CONVNETS AS FEATURE EXTRACTORS We can remove the last layer and then use these features to extract features, these features are very useful features for classification. Some people use these features with LSH (locality-sensitive hashing) to scale large databases for image search. You can also use these features as input for a SVM classifier for instance.
  • 73.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A TRANSFER LEARNING Features learned by Convolutional Neural Networks on large dataset problem (i.e. ImageNet), can be helpful on different problems. It’s very common to pre-train a convnet on ImageNet and then use it as a fixed feature extractor or as initialization. FINE-TUNING THE CONVNETS You can use a pre-trained convnet to continue its training on your data and thus fine-tune the weights for your problem. First layers of a convnet contains generic features (i.e. edge detectors, etc.) that should be helpful in many tasks. Deeper layers becomes progressively specific to the details of the classes of the original problem.
  • 74.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A Section IV INTERESTING CASES
  • 75.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A MUSIC RECOMMENDATION Source: Recommending music on Spotify with deep learning. Sander Dieleman. This is an example architecture from Spotify, using Convolutional Neural Network for music recommendation.
  • 76.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A MUSIC RECOMMENDATION Learned filters at first convolutional layer. The time axis is horizontal, the frequency axis is vertical (frequency increases from top to bottom)
  • 77.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A NATURAL LANGUAGE PROCESSING Source: Text understanding from scratch. Xiang Zhang, Yann LeCun. Deciding if a review posted on Amazon is positive or negative with 96% accuracy, and predict the actual number of stars with 73% accuracy.
  • 78.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WORD2VEC Source: Distributed Representations of Sentences and Documents. Quoc Le, Tomas Mikolov. Word vectors (trained with up to hundreds of billions of words).
  • 79.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WORD2VEC Source: Distributed Representations of Sentences and Documents. Quoc Le, Tomas Mikolov. Word vectors (trained with up to hundreds of billions of words). With nice properties: v(’Paris’) - v(’France’) + v(’Italy’ ) ≈ v(’Rome’) v(’king’) - v(’man’) + v(’woman’) ≈ v(’queen’) No deep learning and no convnet, but a great distributed representation example.
  • 80.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A WORD2VEC Machine Translation Source: DL4J.
  • 81.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A DOC2VEC Sentiment analysis Source: Distributed Representations of Sentences and Documents. Quoc Le, Tomas Mikolov.
  • 82.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A INTERESTING FRAMES Google recently put in production a Deep Neural Network to improve YouTube video thumbnails. Source: Google Research.
  • 83.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A SPATIAL TRANSFORMER NETWOKRS Spatial Transformer Networks can learn transformations. Source: Spatial Transformer Networks. Max Jaderberg, et. al.
  • 84.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A DEEPFACE BY FACEBOOK Source: DeepFace: Closing the Gap to Human-Level Performance in Face Verification. Yaniv Taigman, et. al.
  • 85.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A Section V Q&A
  • 86.
    DEEP LEARNING NEURALNETWORKS CONVOLUTIONAL NEURAL NETWORKS INTERESTING CASES Q&A Q&A