Merge pull request #88 from samagra14/4e_pseudocode

norvig · web-flow · commit 4c9a330b502f · 2019-02-06T12:57:16.000-08:00
4e pseudocode
diff --git a/md/Adam-Optimizer.md b/md/Adam-Optimizer.md
@@ -0,0 +1,21 @@
+# ADAM-OPTIMIZER
+
+## AIMA4e
+
+__function__ ADAM-OPTIMIZER(_f_,_L_,***&theta;***,_&rho;_,_&alpha;_,_&delta;_) __returns__ updated ***&theta;***  
+&emsp;&emsp;&emsp; /\* _Defualts:_ _&rho;<sub>1</sub>_ = 0.9; _&rho;<sub>2</sub>_ = 0.999;_&alpha;_ = 0.001;_&delta;_ = 10<sup>-8</sup> \*/  
+&emsp;***s*** &larr; __0__  
+&emsp;***r*** &larr; __0__  
+&emsp; t &larr; 0  
+&emsp;&emsp;&emsp;__while__ ***&theta;*** has not converged  
+&emsp;&emsp;&emsp;&emsp;&emsp; ***x***, ***y*** &larr; a minibatch of _m_ examples from training set  
+&emsp;&emsp;&emsp;&emsp;&emsp; ***g*** &larr; <a href="https://www.codecogs.com/eqnedit.php?latex=\inline&space;\frac{1}{m}\nabla_{\theta}\sum_{i}L(f(\textbf{x}^{(i)};\boldsymbol{\theta}),\textbf{y}^{(i)})" target="_blank"><img src="https://latex.codecogs.com/gif.latex?\inline&space;\frac{1}{m}\nabla_{\theta}\sum_{i}L(f(\textbf{x}^{(i)};\boldsymbol{\theta}),\textbf{y}^{(i)})" title="\frac{1}{m}\nabla_{\theta}\sum_{i}L(f(\textbf{x}^{(i)};\boldsymbol{\theta}),\textbf{y}^{(i)})" /></a> /\* compute gradient \*/  
+&emsp;&emsp;&emsp;&emsp;&emsp;_t_ &larr; _t_ &plus; 1   
+&emsp;&emsp;&emsp;&emsp;&emsp;***s*** &larr; _&rho;<sub>1</sub>_***s*** &plus; (1 &minus; _&rho;<sub>1</sub>_)***g*** /\* _Update biased first moment estimate_ \*/  
+&emsp;&emsp;&emsp;&emsp;&emsp;***r*** &larr; _&rho;<sub>2</sub>_***r*** &plus; (1 &minus; _&rho;<sub>2</sub>_)***g***<a href="https://www.codecogs.com/eqnedit.php?latex=\inline&space;\odot" target="_blank"><img src="https://latex.codecogs.com/gif.latex?\inline&space;\odot" title="\odot" /></a>***g*** /\*_Update biased second moment estimate_ \*/  
+&emsp;&emsp;&emsp;&emsp;&emsp;<a href="https://www.codecogs.com/eqnedit.php?latex=\inline&space;\hat{\textbf{s}}\gets\frac{\textbf{s}}{1-\rho_1^t}" target="_blank"><img src="https://latex.codecogs.com/gif.latex?\inline&space;\hat{\textbf{s}}\gets\frac{\textbf{s}}{1-\rho_1^t}" title="\hat{\textbf{s}}\gets\frac{\textbf{s}}{1-\rho_1^t}" /></a>  /\* _Correct bias in first moment_ \*/   
+&emsp;&emsp;&emsp;&emsp;&emsp;<a href="https://www.codecogs.com/eqnedit.php?latex=\inline&space;\hat{\textbf{r}}\gets\frac{\textbf{r}}{1-\rho_2^t}" target="_blank"><img src="https://latex.codecogs.com/gif.latex?\inline&space;\hat{\textbf{r}}\gets\frac{\textbf{r}}{1-\rho_2^t}" title="\hat{\textbf{r}}\gets\frac{\textbf{r}}{1-\rho_2^t}" /></a> /\* _Correct bias in second moment_ \*/  
+&emsp;&emsp;&emsp;&emsp;&emsp;<a href="https://www.codecogs.com/eqnedit.php?latex=\inline&space;\boldsymbol{\Delta}\boldsymbol{\theta}&space;=&space;-\epsilon\frac{\hat{\textbf{s}}}{\sqrt{\hat{\textbf{r}}}&plus;\delta}" target="_blank"><img src="https://latex.codecogs.com/gif.latex?\inline&space;\boldsymbol{\Delta}\boldsymbol{\theta}&space;=&space;-\epsilon\frac{\hat{\textbf{s}}}{\sqrt{\hat{\textbf{r}}}&plus;\delta}" title="\boldsymbol{\Delta}\boldsymbol{\theta} = -\epsilon\frac{\hat{\textbf{s}}}{\sqrt{\hat{\textbf{r}}}+\delta}" /></a>  /\* _Compute update (operations applied element-wise)_ \*/  
+&emsp;&emsp;&emsp;&emsp;&emsp;<a href="https://www.codecogs.com/eqnedit.php?latex=\inline&space;\boldsymbol{\theta}&space;\gets&space;\boldsymbol{\theta}&space;&plus;&space;\boldsymbol{\Delta}\boldsymbol{\theta}" target="_blank"><img src="https://latex.codecogs.com/gif.latex?\inline&space;\boldsymbol{\theta}&space;\gets&space;\boldsymbol{\theta}&space;&plus;&space;\boldsymbol{\Delta}\boldsymbol{\theta}" title="\boldsymbol{\theta} \gets \boldsymbol{\theta} + \boldsymbol{\Delta}\boldsymbol{\theta}" /></a> /\* _Apply update_ \*/   
+
+
diff --git a/md/Cross-Validation-Wrapper.md b/md/Cross-Validation-Wrapper.md
@@ -20,5 +20,10 @@ __function__ CROSS\-VALIDATION(_Learner_, _size_, _k_, _examples_) __returns__ t
 &emsp;&emsp;&emsp;_fold\_errV_ &larr; _fold\_errV_ &plus; ERROR\-RATE(_h_, _validation\_set_)  
 &emsp;__return__ _fold\_errT_ &frasl; _k_, _fold\_errV_ &frasl; _k_  
 
+---  
+
+Figure?? An algorithm to select the model that has the lowest error rate on validation data by building models of increasing complexity, and choosing the one with best empirical error rate on validation data. Here _errT_ means error rate on the training data, and _errV_ means error rate on the validation data. _Learner_(_size_, _exmaples_) returns a hypothesis whose complexity is set by the parameter _size_, and which is trained on the _examples_. PARTITION(_examples_, _fold_, _k_) splits _examples_ into two subsets: a validation set of size _N_ &frasl; _k_ and a training set with all the other examples. The split is different for each value of _fold_.  
+
 ---
-__Figure ??__ An algorithm to select the model that has the lowest error rate on validation data by building models of increasing complexity, and choosing the one with best empirical error rate on validation data. Here _errT_ means error rate on the training data, and _errV_ means error rate on the validation data. _Learner_(_size_, _exmaples_) returns a hypothesis whose complexity is set by the parameter _size_, and which is trained on the _examples_. PARTITION(_examples_, _fold_, _k_) splits _examples_ into two subsets: a validation set of size _N_ &frasl; _k_ and a training set with all the other examples. The split is different for each value of _fold_.
+
+In the fourth edition, cross vaidation wrapper has been renamed to Model-Selection.
diff --git a/md/Gibbs-Ask.md b/md/Gibbs-Ask.md
@@ -14,4 +14,22 @@ __function__ GIBBS-ASK(_X_, __e__, _bn_, _N_) __returns__ an estimate of __P__(_
 &emsp;__return__ NORMALIZE(__N__)  
 
 ---
-__Figure__ ?? The Gibbs sampling algorithm for approximate inference in Bayesian networks; this version cycles through the variables, but choosing variables at random also works.
+__Figure__ ?? The Gibbs sampling algorithm for approximate inference in Bayesian networks; this version cycles through the variables, but choosing variables at random also works.  
+
+---
+  
+## AIMA4e  
+__function__ GIBBS-ASK(_X_, __e__, _bn_, _N_) __returns__ an estimate of __P__(_X_ &vert; __e__)  
+&emsp;__local variables__: __N__, a vector of counts for each value of _X_, initially zero  
+&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;__Z__, the nonevidence variables in _bn_  
+&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;__x__, the current state of the network, initially copied from __e__  
+  
+&emsp;initialize __x__ with random values for the variables in __Z__  
+&emsp;__for__ _j_ = 1 to _N_ __do__  
+&emsp;&emsp;&emsp;__choose__ any variable _Z<sub>i</sub>_ from __Z__ acoording to any distribution _&rho;(i)_  
+&emsp;&emsp;&emsp;&emsp;&emsp;set the value of _Z<sub>i</sub>_ in __x__ by sampling from __P__(_Z<sub>i</sub>_ &vert; _mb_(_Z<sub>i</sub>_))  
+&emsp;&emsp;&emsp;&emsp;&emsp;__N__\[_x_\] &larr; __N__\[_x_\] &plus; 1 where _x_ is the value of _X_ in __x__  
+&emsp;__return__ NORMALIZE(__N__)  
+
+---
+__Figure__ ?? The Gibbs sampling algorithm for approximate inference in Bayesian networks; this version cycles through the variables, but choosing variables at random also works.  
diff --git a/md/Model-Selection.md b/md/Model-Selection.md
@@ -0,0 +1,28 @@
+# MODEL-SELECTION
+
+## AIMA4e  
+__function__ MODEL-SELECTION(_Learner_, _examples_, _k_) __returns__ a hypothesis  
+&emsp;__local variables__: _err_, an array, indexed by _size_, storing validation\-set error rates  
+&emsp;__for__ _size_ = 1 __to__ &infin; __do__   
+&emsp;&emsp;&emsp;_err_\[_size_\] &larr; CROSS\-VALIDATION(_Learner_, _size_, _examples_, _k_)  
+&emsp;&emsp;&emsp;__if__ _err_ is starting to increase significantly __then do__  
+&emsp;&emsp;&emsp;&emsp;&emsp;_best\_size_ &larr; the value of _size_ with minimum _err_\[_size_\]  
+&emsp;&emsp;&emsp;&emsp;&emsp;__return__ _Learner_(_best\_size_, _examples_)  
+
+---
+__function__ CROSS\-VALIDATION(_Learner_, _size_, _examples_, _k_) __returns__ average training set error rate:  
+&emsp;_errs_ &larr; 0  
+&emsp;__for__ _fold_ = 1 __to__ _k_ __do__  
+&emsp;&emsp;&emsp;_training\_set_, _validation\_set_ &larr; PARTITION(_examples_, _fold_, _k_)  
+&emsp;&emsp;&emsp;_h_ &larr; _Learner_(_size_, _training\_set_)  
+&emsp;&emsp;&emsp;_errs_ &larr; _errs_ &plus; ERROR\-RATE(_h_, _validation\_set_)    
+&emsp;__return__ _errs_ &frasl; _k_   // average error rate on validation sets, across k-fold cross-validation 
+
+---  
+
+Figure?? An algorithm to select the model that has the lowest error rate on validation data by building models of increasing complexity, and choosing the one with best empirical error rate, _err_, on validation data. _Learner_(_size_, _exmaples_) returns a hypothesis whose complexity is set by the parameter _size_, and which is trained on the _examples_. PARTITION(_examples_, _fold_, _k_) splits _examples_ into two subsets: a validation set of size _N_ &frasl; _k_ and a training set with all the other examples. The split is different for each value of _fold_.  
+
+---
+
+In the fourth edition, cross vaidation wrapper has been renamed to Model-Selection.
+
diff --git a/md/Monte-Carlo-Tree-Search.md b/md/Monte-Carlo-Tree-Search.md
@@ -0,0 +1,36 @@
+
+
+
+# MONTE-CARLO-TREE-SEARCH
+
+## AIMA4e
+__function__ MONTE-CARLO-TREE-SEARCH(_state_) __returns__ an action  
+&emsp;tree &larr; NODE(_state_)
+&emsp;__while__ TIME\-REMAINING() __do__  
+&emsp;&emsp;&emsp;__tree__ &larr; PLAYOUT(_tree_)  
+&emsp;__return__ the _move_ in ACTIONS(_state_) with highest Q(_state_,_move_)  
+
+---
+
+__function__ PLAYOUT(_tree_) __returns__ _updated tree_  
+&emsp;_node_ &larr; _tree_  
+&emsp;__while__ _node_ is not terminal and was already in _tree_ __do__  
+&emsp;&emsp;&emsp;_move_ &larr; SELECT(_node_)  
+&emsp;&emsp;&emsp;_node_ &larr; FOLLOW\-LINK(_node_,_move_)  
+&emsp;_outcome_ &larr; SIMULATION(_node_.STATE)  
+&emsp;UPDATE(_node_,_outcome_)  
+&emsp;__return__ _tree_  
+
+---
+
+__function__ SELECT(_node_) __returns__ _an action_  
+&emsp;__return__ argmax<sub>m &isin; FEASIBLE\-ACTIONS(_node_)</sub> UCB(RESULT(_node_,_m_))  
+
+---
+
+__function__ UCB(_child_) __returns__ _a number_  
+&emsp;__return__ _child_.VALUE + C &times; <a href="https://www.codecogs.com/eqnedit.php?latex=\inline&space;\sqrt{\frac{\log{child.PARENT.N}}{child.N}}" target="_blank"><img src="https://latex.codecogs.com/png.latex?\inline&space;\sqrt{\frac{\log{child.PARENT.N}}{child.N}}" title="\sqrt{\frac{\log{child.PARENT.N}}{child.N}}" /></a> 
+
+
+---
+__FIGURE ??__ The Monte Carlo tree search algorithm. A game tree, _tree_, is initialized, and then grows by one node with each call to PLAYOUT. The function SELECT chooses a move that best balances exploitation and exploration according to the UCB formula. FOLLOW-LINK traverses from the current node by making a move; this could be to a previously-seen node, or to a new node that is added to the tree. Once we have added a new node, we exit the __while__ loop and SIMULATION chooses moves (with a randomized policy that is designed to favor good moves but to compute quickly) until the game is over. Then, UPDATE updates all the nodes in the tree from node to the root, recording the fact that the path led to the final __outcome__.
diff --git a/md/Policy-Iteration.md b/md/Policy-Iteration.md
@@ -18,3 +18,24 @@ __function__ POLICY-ITERATION(_mdp_) __returns__ a policy
 
 ---
 __Figure ??__ The policy iteration algorithm for calculating an optimal policy.
+
+---
+  
+## AIMA4e  
+__function__ POLICY-ITERATION(_mdp_) __returns__ a policy  
+&emsp;__inputs__: _mdp_, an MDP with states _S_, actions _A_(_s_), transition model _P_(_s&prime;_ &vert; _s_, _a_)  
+&emsp;__local variables__: _U_, a vector of utilities for states in _S_, initially zero  
+&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;_&pi;_, a policy vector indexed by state, initially random  
+  
+&emsp;__repeat__  
+&emsp;&emsp;&emsp;_U_ &larr; POLICY\-EVALUATION(_&pi;_, _U_, _mdp_)  
+&emsp;&emsp;&emsp;_unchanged?_ &larr; true  
+&emsp;&emsp;&emsp;__for each__ state _s_ __in__ _S_ __do__  
+&emsp;&emsp;&emsp;&emsp;&emsp;_a <sup> &#x2a; </sup>_ &larr; argmax<sub>_a_ &isin; _A_(_s_)</sub> Q-VALUE(_mdp_,_s_,_a_,_U_)  
+&emsp;&emsp;&emsp;&emsp;&emsp;__if__ Q-VALUE(_mdp_,_s_,_a<sup>&#x2a;</sup>_,_U_) &gt; Q-VALUE(_mdp_,_s_,_&pi;_\[_s_\],_U_) __then do__    
+&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;_&pi;_\[_s_\] &larr; _a<sup>&#x2a;</sup>_ ; _unchanged?_ &larr; false  
+&emsp;__until__ _unchanged?_  
+&emsp;__return__ _&pi;_  
+
+---
+__Figure ??__ The policy iteration algorithm for calculating an optimal policy.  
diff --git a/md/Value-Iteration.md b/md/Value-Iteration.md
@@ -18,3 +18,25 @@ __function__ VALUE-ITERATION(_mdp_, _&epsi;_) __returns__ a utility function
 
 ---
 __Figure ??__ The value iteration algorithm for calculating utilities of states. The termination condition is from Equation (__??__).
+
+---
+
+## AIMA4e  
+__function__ VALUE-ITERATION(_mdp_, _&epsi;_) __returns__ a utility function  
+&emsp;__inputs__: _mdp_, an MDP with states _S_, actions _A_(_s_), transition model _P_(_s&prime;_ &vert; _s_, _a_),  
+&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;rewards _R_(_s_,_a_,_s&prime;_), discount _&gamma;_  
+&emsp;&emsp;&emsp;_&epsi;_, the maximum error allowed in the utility of any state  
+&emsp;__local variables__: _U_, _U&prime;_, vectors of utilities for states in _S_, initially zero  
+&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;_&delta;_, the maximum change in the utility of any state in an iteration  
+
+&emsp;__repeat__  
+&emsp;&emsp;&emsp;_U_ &larr; _U&prime;_; _&delta;_ &larr; 0  
+&emsp;&emsp;&emsp;__for each__ state _s_ in _S_ __do__  
+&emsp;&emsp;&emsp;&emsp;&emsp;_U&prime;_\[_s_\] &larr; max<sub>_a_ &isin; _A_(_s_)</sub> Q-VALUE(_mdp_,_s_,_a_,_U_)  
+&emsp;&emsp;&emsp;&emsp;&emsp;__if__ &vert; _U&prime;_\[_s_\] &minus; _U_\[_s_\]  &vert; &gt; _&delta;_ __then__ _&delta;_ &larr; &vert; _U&prime;_\[_s_\] &minus; _U_\[_s_\]  &vert;  
+&emsp;__until__ _&delta;_ &lt; _&epsi;_(1 &minus; _&gamma;_)&sol;_&gamma;_  
+&emsp;__return__ _U_  
+
+---
+__Figure ??__ The value iteration algorithm for calculating utilities of states. The termination condition is from Equation (__??__).
+~                  
