Outline of deep learning

This article contains dynamic lists that may never be able to satisfy particular standards for completeness. You can help by editing the page to add missing items, with references to reliable sources.

The following outline is provided as an overview of, and topical guide to, deep learning:

Deep learning is a subfield of machine learning and artificial intelligence based on artificial neural networks with multiple processing layers. It emphasizes representation learning and is widely used in areas such as computer vision, natural language processing, speech recognition, recommender systems, robotics, and generative artificial intelligence.

Ways to categorize deep learning

• A field of study
• A branch of artificial intelligence
• A subfield of machine learning
• A subfield of computer science
• A form of representation learning
• A class of methods based on artificial neural networks
• An approach used in computational statistics

History

Precursors

• Cybernetics
• Perceptron
• Connectionism
• Neocognitron
• Backpropagation

Milestones

• LeNet
• Long short-term memory
• Deep belief network
• AlexNet
• Sequence to sequence learning
• Generative adversarial network
• Residual neural network
• Transformer
• BERT
• Generative pre-trained transformer
• Diffusion model

Related histories

• History of artificial intelligence
• History of machine learning
• Timeline of machine learning

Core concepts

• Artificial neural network
• Representation learning
• Feature learning
• Gradient descent
• Backpropagation
• Loss function
• Optimization
• Training, validation, and test data sets
• Generalization
• Overfitting
• Underfitting
• Hyperparameter
• Hyperparameter optimization
• Foundation model
• Large language model

Learning settings

• Supervised learning
• Unsupervised learning
• Self-supervised learning
• Semi-supervised learning
• Reinforcement learning
• Transfer learning
• Multitask learning
• Multimodal learning
• Online machine learning
• Continual learning

Common tasks

• Image classification
• Object detection
• Image segmentation
• Automatic speech recognition
• Neural machine translation
• Question answering
• Automatic summarization
• Text-to-image model
• Protein structure prediction

Architectures

Feedforward and convolutional architectures

• Feedforward neural network
• Multilayer perceptron
• Convolutional neural network
• Radial basis function network
• Residual neural network
• U-Net

Recurrent and sequence architectures

• Recurrent neural network
• Long short-term memory
• Gated recurrent unit
• Sequence to sequence learning
• Recursive neural network

Representation-learning architectures

• Autoencoder
• Denoising autoencoder
• Sparse autoencoder
• Variational autoencoder
• Restricted Boltzmann machine
• Deep belief network

Attention and transformer architectures

• Attention (machine learning)
• Transformer
• BERT
• Generative pre-trained transformer
• Vision transformer

Generative and probabilistic architectures

• Autoregressive model
• Diffusion model
• Energy-based model
• Generative adversarial network
• Mixture of experts

Graph and memory architectures

• Graph neural network
• Graph convolutional network
• Siamese network
• Neural Turing machine
• Memory network
• Echo state network
• Capsule neural network

Neural network components and techniques

See also: Neural network and Neural network (machine learning)

• Artificial neuron
• Activation function
  • Rectified linear unit
  • Sigmoid function
  • Softmax function
• Embedding
• Convolution
• Pooling layer
• Attention
• Batch normalization
• Layer normalization
• Residual connections

Training and optimization

• Backpropagation
• Gradient descent
• Stochastic gradient descent
• Adam optimization
• Learning rate
• Loss function
  • Cross-entropy
  • Mean squared error
• Regularization
  • Dropout
  • Early stopping
• Batch normalization
• Data augmentation
• Transfer learning
• Knowledge distillation
• Ensemble learning
• Curriculum learning

Datasets and benchmarks

• CIFAR-10
• ImageNet
• MNIST database
• Common Objects in Context (COCO)
• General Language Understanding Evaluation (GLUE) benchmark
• LibriSpeech
• SQuAD

Applications

See also: Applications of artificial intelligence

Computer vision

• Computer vision
• Facial recognition system
• Image classification
• Image segmentation
• Medical imaging
• Object detection
• Optical character recognition

Natural language processing

• Automatic summarization
• Chatbot
• Information retrieval
• Large language model
• Natural language processing
• Neural machine translation
• Question answering
• Sentiment analysis

Speech and audio

See also: Music and artificial intelligence

• Automatic speech recognition
• Music information retrieval
• Speaker recognition
• Speech synthesis

Science and medicine

• Bioinformatics
• Computational biology
• Drug discovery
• Medical diagnosis
• Protein structure prediction

Robotics and control

• Autonomous car
• Computer game bot
• Control theory
• Robotics

Recommendation, search, and forecasting

• Anomaly detection
• Forecasting
• Fraud detection
• Recommender system
• Search engine

Generative artificial intelligence

• Deepfake
• Generative artificial intelligence
• Large language model
• Speech synthesis
• Text-to-image model

Computer graphics and video games

• Deep Learning Anti-Aliasing (DLAA)
• Deep Learning Super Sampling (DLSS)

Hardware

• AMD Instinct
• AMD XDNA
• Application-specific integrated circuit
• Deep learning processor, Neural processing unit (NPU), or Neural Engine
• Field-programmable gate array
• General-purpose computing on graphics processing units (GPGPU)
• Graphics processing unit
• NVIDIA Deep Learning Accelerator (NVDLA)
• Tensor processing unit
• Vision processing unit
• Wafer-scale integration

Supporting software platforms

• CUDA
• Metal
• ROCm

Software

See also: Comparison of deep learning software, Comparison of machine learning software, and List of Python software for deep learning and machine learning

Open-source frameworks and libraries

• Apache MXNet
• Apache Singa
• BigDL
• Caffe
• Chainer
• Deeplearning4j
• DeepSpeed
• Dlib
• Fastai
• Flux
• JAX
• Horovod
• Keras
• Microsoft Cognitive Toolkit
• MindSpore
• mlpack
• PyTorch
• PyTorch Lightning
• TensorFlow
• Theano
• Torch

Neural network software

See also: Neural network software

• EDLUT
• Emergent
• Encog
• JOONE
• Neuroph
• NeuroSolutions
• OpenNN
• Peltarion Synapse
• SNNS

Platforms, tools, and deployment

• Amazon SageMaker
• Google Colab
• Hugging Face
• Kaggle
• Kubeflow
• MLflow
• ONNX
• OpenVINO
• TensorFlow Hub

Algorithms for deep learning and neural networks

See also: List of artificial intelligence algorithms

• Backpropagation
• Conjugate gradient method
• Generalized Hebbian algorithm
• Gradient descent
• Levenberg–Marquardt algorithm
• Perceptron
• Quasi-Newton method
• Wake-sleep algorithm

Methods and related topics

Representation and metric learning

• Contrastive learning
• Embedding
• Feature learning
• Manifold learning
• Metric learning

Generative modeling

• Autoregressive model
• Diffusion model
• Generative adversarial network
• Generative model
• Variational inference

Efficient and scalable deep learning

• Knowledge distillation
• Low-rank approximation
• Mixture of experts
• Quantization
• Sparsity

Reliability, safety, and interpretability

• Adversarial machine learning
• AI alignment
• Algorithmic bias
• Catastrophic forgetting
• Differential privacy
• Explainable artificial intelligence
• Federated learning
• Hallucination (artificial intelligence)

Conferences and workshops

• Annual Meeting of the Association for Computational Linguistics
• Conference on Computer Vision and Pattern Recognition
• Conference on Neural Information Processing Systems
• International Conference on Computer Vision
• International Conference on Learning Representations
• International Conference on Machine Learning

Organizations

See also: List of artificial intelligence companies

Research laboratories and institutions

• Allen Institute for AI
• Alberta Machine Intelligence Institute
• European Laboratory for Learning and Intelligent Systems
• Google DeepMind
• Meta AI
• Mila
• Microsoft Research
• Vector Institute

Companies

• Anthropic
• Cerebras
• Cohere
• DeepSeek
• Mistral AI
• OpenAI
• Stability AI
• xAI

Publications

Books

• Deep Learning – Ian Goodfellow and Yoshua Bengio
• Neural Networks and Deep Learning – Michael Nielsen
• Perceptrons – Marvin Minsky and Seymour Papert

Journals

See also: List of artificial intelligence journals

• IEEE Transactions on Neural Networks and Learning Systems
• Neural Networks
• Neural Computation

Influential persons

• Alex Graves
• Alex Krizhevsky
• Andrew Ng
• Andrej Karpathy
• Ashish Vaswani
• Christopher Bishop
• Demis Hassabis
• Fei-Fei Li
• Geoffrey Hinton
• Ian Goodfellow
• Ilya Sutskever
• John Hopfield
• Jürgen Schmidhuber
• Noam Shazeer
• Oriol Vinyals
• Paul Werbos
• Quoc V. Le
• Ruslan Salakhutdinov
• Sepp Hochreiter
• Seppo Linnainmaa
• Terry Sejnowski
• Yann LeCun
• Yoshua Bengio

See also

• Artificial intelligence
• Artificial neural network
• Generative artificial intelligence
• Glossary of artificial intelligence
• Lists of open-source artificial intelligence software
• Machine learning
• Neural network software
• Outline of artificial intelligence
• Outline of computer vision
• Outline of machine learning
• Outline of robotics