Repository navigation
R23-T054: Add ELU edge-value and SELU/Leaky ReLU agreement doctests - #64
Open
code-paul-creator wants to merge 1 commit into
Open
code-paul-creator wants to merge 1 commit into
code-paul-creator wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Compares
exponential_linear_unit.pywithscaled_exponential_linear_unit.pyandleaky_rectified_linear_unit.py(API, numeric agreement, edge values, speed) and adds doctests to ELU that pin down the results: edge values, agreement with SELU at scale 1 and at SELU's documented constants, and agreement with Leaky ReLU on positive inputs.Related Issue
Fixes: #54
Type of Change
Changes Made
exponential_linear_unitfor 0, large negative (-1000 saturates at -alpha), alpha=0, ELU == SELU(lambda_=1), SELU defaults == 1.0507 * ELU(x, 1.6732), and ELU == Leaky ReLU for positive inputs (existing 2 doctests unchanged; 13 total).work/R23-T054/compare_test.py(19 checks over 14 shared inputs and 5 alphas),bench_compare.py, andCOMPARE.md(API differences, results, edge values, speed, beginner recommendation).Screenshots / Recordings
Not applicable (no UI changes).
Testing
python -m pytest --doctest-modules neural_network/activation_functions/exponential_linear_unit.py-> passed (13 doctests)python -m pytest work/R23-T054/compare_test.py-> 19 passedruff check neural_network/activation_functions/exponential_linear_unit.py-> all checks passedChecklist
Additional Notes
Findings: ELU == SELU exactly when lambda_=1; ELU == Leaky ReLU on all positive inputs; all three give 0 at 0; they differ on negatives (ELU saturates at -alpha, Leaky ReLU is unbounded). A large positive input (e.g. 1000) returns the right value but emits
RuntimeWarning: overflow encountered in expin ELU/SELU becausenp.whereevaluates both branches; not changed here. Details inwork/R23-T054/COMPARE.md.