Move step pre/post hook tests to OptimizerInfo #119288

janeyx99 · 2024-02-06T17:11:18Z

Note that this increases coverage from 1 config (vanilla SGD) to all the configs (13 optimizers at around 6-7 each). The test time seems fine though!

With the torch cuda synchronization:

(pytorch-3.10) [janeyx@devgpu023.odn1 ~/local/pytorch (b6093c03)]$ python test/test_optim.py -k test_step_pre_hook -k test_step_post_hook
/home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
/home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.17.3 and <1.25.0 is required for this version of SciPy (detected version 1.26.0
  warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
....................................................
----------------------------------------------------------------------
Ran 52 tests in 13.680s

OK

Excluding the torch cuda synchronization:

(pytorch-3.10) [janeyx@devgpu023.odn1 ~/local/pytorch (916f6fe3)]$ python test/test_optim.py -k test_step_pre_hook -k test_step_post_hook
/home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
/home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.17.3 and <1.25.0 is required for this version of SciPy (detected version 1.26.0
  warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
....................................................
----------------------------------------------------------------------
Ran 52 tests in 1.038s

OK

The old tests:

(pytorch-3.10) [janeyx@devgpu023.odn1 ~/local/pytorch (916f6fe3)]$ python test/test_optim.py -k test_pre_hook -k test_post_hook
/home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
  _torch_pytree._register_pytree_node(
/home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.17.3 and <1.25.0 is required for this version of SciPy (detected version 1.26.0
  warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
..
----------------------------------------------------------------------
Ran 2 tests in 0.518s

OK

Stack from ghstack (oldest at bottom):

[ghstack-poisoned]

pytorch-bot · 2024-02-06T17:11:21Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/119288

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 8dd1cd7 with merge base 499040a ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[ghstack-poisoned]

ghstack-source-id: 66c2ee7 Pull Request resolved: #119288

mikaylagawarecki

Looks good, one question: how does the test logic differentiate between whether the hook is happening pre/post step?

mikaylagawarecki · 2024-02-06T18:57:05Z

test/test_optim.py

+
+            optim.step(closure)
+            optim.step(closure)
+            # check if pre hooks were registered


Suggested change

# check if pre hooks were registered

# check if post hooks were registered

It definitely does not differentiate -> This test can certainly be improved, but I'm going to let it stay as is for now to quickly migrate tests away. I'll open an issue though!

See #119300

Note that this increases coverage from 1 config (vanilla SGD) to all the configs (13 optimizers at around 6-7 each). The test time seems fine though! With the torch cuda synchronization: ``` (pytorch-3.10) [janeyxdevgpu023.odn1 ~/local/pytorch (b6093c0)]$ python test/test_optim.py -k test_step_pre_hook -k test_step_post_hook /home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. _torch_pytree._register_pytree_node( /home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.17.3 and <1.25.0 is required for this version of SciPy (detected version 1.26.0 warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}" .................................................... ---------------------------------------------------------------------- Ran 52 tests in 13.680s OK ``` Excluding the torch cuda synchronization: ``` (pytorch-3.10) [janeyxdevgpu023.odn1 ~/local/pytorch (916f6fe)]$ python test/test_optim.py -k test_step_pre_hook -k test_step_post_hook /home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. _torch_pytree._register_pytree_node( /home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.17.3 and <1.25.0 is required for this version of SciPy (detected version 1.26.0 warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}" .................................................... ---------------------------------------------------------------------- Ran 52 tests in 1.038s OK ``` The old tests: ``` (pytorch-3.10) [janeyxdevgpu023.odn1 ~/local/pytorch (916f6fe)]$ python test/test_optim.py -k test_pre_hook -k test_post_hook /home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. _torch_pytree._register_pytree_node( /home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.17.3 and <1.25.0 is required for this version of SciPy (detected version 1.26.0 warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}" .. ---------------------------------------------------------------------- Ran 2 tests in 0.518s OK ``` [ghstack-poisoned]

Pull Request resolved: #119299 Approved by: https://github.com/mikaylagawarecki ghstack dependencies: #119283, #119288

Pull Request resolved: #119308 Approved by: https://github.com/mikaylagawarecki ghstack dependencies: #119283, #119288, #119299

Pull Request resolved: #119310 Approved by: https://github.com/albanD ghstack dependencies: #119283, #119288, #119299, #119308

Note that this increases coverage from 1 config (vanilla SGD) to all the configs (13 optimizers at around 6-7 each). The test time seems fine though! With the torch cuda synchronization: ``` (pytorch-3.10) [janeyx@devgpu023.odn1 ~/local/pytorch (b6093c0)]$ python test/test_optim.py -k test_step_pre_hook -k test_step_post_hook /home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. _torch_pytree._register_pytree_node( /home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.17.3 and <1.25.0 is required for this version of SciPy (detected version 1.26.0 warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}" .................................................... ---------------------------------------------------------------------- Ran 52 tests in 13.680s OK ``` Excluding the torch cuda synchronization: ``` (pytorch-3.10) [janeyx@devgpu023.odn1 ~/local/pytorch (916f6fe)]$ python test/test_optim.py -k test_step_pre_hook -k test_step_post_hook /home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. _torch_pytree._register_pytree_node( /home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.17.3 and <1.25.0 is required for this version of SciPy (detected version 1.26.0 warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}" .................................................... ---------------------------------------------------------------------- Ran 52 tests in 1.038s OK ``` The old tests: ``` (pytorch-3.10) [janeyx@devgpu023.odn1 ~/local/pytorch (916f6fe)]$ python test/test_optim.py -k test_pre_hook -k test_post_hook /home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. _torch_pytree._register_pytree_node( /home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.17.3 and <1.25.0 is required for this version of SciPy (detected version 1.26.0 warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}" .. ---------------------------------------------------------------------- Ran 2 tests in 0.518s OK ``` Pull Request resolved: #119288 Approved by: https://github.com/mikaylagawarecki ghstack dependencies: #119283

Pull Request resolved: #119299 Approved by: https://github.com/mikaylagawarecki ghstack dependencies: #119283, #119288

Pull Request resolved: #119308 Approved by: https://github.com/mikaylagawarecki ghstack dependencies: #119283, #119288, #119299

Pull Request resolved: #119310 Approved by: https://github.com/albanD ghstack dependencies: #119283, #119288, #119299, #119308

Note that this increases coverage from 1 config (vanilla SGD) to all the configs (13 optimizers at around 6-7 each). The test time seems fine though! With the torch cuda synchronization: ``` (pytorch-3.10) [janeyx@devgpu023.odn1 ~/local/pytorch (b6093c0)]$ python test/test_optim.py -k test_step_pre_hook -k test_step_post_hook /home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. _torch_pytree._register_pytree_node( /home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.17.3 and <1.25.0 is required for this version of SciPy (detected version 1.26.0 warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}" .................................................... ---------------------------------------------------------------------- Ran 52 tests in 13.680s OK ``` Excluding the torch cuda synchronization: ``` (pytorch-3.10) [janeyx@devgpu023.odn1 ~/local/pytorch (916f6fe)]$ python test/test_optim.py -k test_step_pre_hook -k test_step_post_hook /home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. _torch_pytree._register_pytree_node( /home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.17.3 and <1.25.0 is required for this version of SciPy (detected version 1.26.0 warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}" .................................................... ---------------------------------------------------------------------- Ran 52 tests in 1.038s OK ``` The old tests: ``` (pytorch-3.10) [janeyx@devgpu023.odn1 ~/local/pytorch (916f6fe)]$ python test/test_optim.py -k test_pre_hook -k test_post_hook /home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/transformers/utils/generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead. _torch_pytree._register_pytree_node( /home/janeyx/.conda/envs/pytorch-3.10/lib/python3.10/site-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.17.3 and <1.25.0 is required for this version of SciPy (detected version 1.26.0 warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}" .. ---------------------------------------------------------------------- Ran 2 tests in 0.518s OK ``` Pull Request resolved: #119288 Approved by: https://github.com/mikaylagawarecki ghstack dependencies: #119283

Pull Request resolved: #119299 Approved by: https://github.com/mikaylagawarecki ghstack dependencies: #119283, #119288

Pull Request resolved: #119308 Approved by: https://github.com/mikaylagawarecki ghstack dependencies: #119283, #119288, #119299

Pull Request resolved: #119310 Approved by: https://github.com/albanD ghstack dependencies: #119283, #119288, #119299, #119308

Move step pre/post hook tests to OptimizerInfo

591904e

[ghstack-poisoned]

janeyx99 mentioned this pull request Feb 6, 2024

Migrate last lbfgs test over to OptimizerInfo #119283

Closed

Update on "Move step pre/post hook tests to OptimizerInfo"

44080c3

[ghstack-poisoned]

janeyx99 added a commit that referenced this pull request Feb 6, 2024

Move step pre/post hook tests to OptimizerInfo

916f6fe

ghstack-source-id: 66c2ee7 Pull Request resolved: #119288

janeyx99 requested a review from mikaylagawarecki February 6, 2024 18:35

mikaylagawarecki approved these changes Feb 6, 2024

View reviewed changes

This was referenced Feb 6, 2024

Move step global hooks test to OptimizerInfo #119299

Closed

Optimizer step pre/post hook tests should try harder to ascertain the hook happens before/after the step call #119300

Open

This was referenced Feb 6, 2024

Migrate test_state_dict hooks to OptimizerInfo #119308

Closed

Migrate load_state_dict hook tests to OptimizerInfo #119310

Closed

pytorchmergebot closed this in 7b3762e Feb 7, 2024

pytorchmergebot pushed a commit that referenced this pull request Feb 7, 2024

Move step global hooks test to OptimizerInfo (#119299)

3625ccf

Pull Request resolved: #119299 Approved by: https://github.com/mikaylagawarecki ghstack dependencies: #119283, #119288

pytorchmergebot added the Merged label Feb 7, 2024

pytorchmergebot pushed a commit that referenced this pull request Feb 7, 2024

Migrate test_state_dict hooks to OptimizerInfo (#119308)

0320e62

Pull Request resolved: #119308 Approved by: https://github.com/mikaylagawarecki ghstack dependencies: #119283, #119288, #119299

pytorchmergebot pushed a commit that referenced this pull request Feb 7, 2024

Migrate load_state_dict hook tests to OptimizerInfo (#119310)

059994d

Pull Request resolved: #119310 Approved by: https://github.com/albanD ghstack dependencies: #119283, #119288, #119299, #119308

pytorch-bot bot pushed a commit that referenced this pull request Feb 8, 2024

Move step global hooks test to OptimizerInfo (#119299)

dfc5028

Pull Request resolved: #119299 Approved by: https://github.com/mikaylagawarecki ghstack dependencies: #119283, #119288

pytorch-bot bot pushed a commit that referenced this pull request Feb 8, 2024

Migrate test_state_dict hooks to OptimizerInfo (#119308)

ecec010

Pull Request resolved: #119308 Approved by: https://github.com/mikaylagawarecki ghstack dependencies: #119283, #119288, #119299

pytorch-bot bot pushed a commit that referenced this pull request Feb 8, 2024

Migrate load_state_dict hook tests to OptimizerInfo (#119310)

cae5407

Pull Request resolved: #119310 Approved by: https://github.com/albanD ghstack dependencies: #119283, #119288, #119299, #119308

clee2000 pushed a commit that referenced this pull request Feb 14, 2024

Move step global hooks test to OptimizerInfo (#119299)

05781ea

Pull Request resolved: #119299 Approved by: https://github.com/mikaylagawarecki ghstack dependencies: #119283, #119288

clee2000 pushed a commit that referenced this pull request Feb 14, 2024

Migrate test_state_dict hooks to OptimizerInfo (#119308)

8139e43

Pull Request resolved: #119308 Approved by: https://github.com/mikaylagawarecki ghstack dependencies: #119283, #119288, #119299

clee2000 pushed a commit that referenced this pull request Feb 14, 2024

Migrate load_state_dict hook tests to OptimizerInfo (#119310)

70a2bea

Pull Request resolved: #119310 Approved by: https://github.com/albanD ghstack dependencies: #119283, #119288, #119299, #119308

github-actions bot deleted the gh/janeyx99/136/head branch March 9, 2024 01:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Move step pre/post hook tests to OptimizerInfo #119288

Move step pre/post hook tests to OptimizerInfo #119288

Uh oh!

janeyx99 commented Feb 6, 2024 •

edited

Loading

Uh oh!

pytorch-bot bot commented Feb 6, 2024 •

edited

Loading

Uh oh!

mikaylagawarecki left a comment

Uh oh!

mikaylagawarecki Feb 6, 2024

Uh oh!

janeyx99 Feb 6, 2024 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	# check if pre hooks were registered
	# check if post hooks were registered

Move step pre/post hook tests to OptimizerInfo #119288

Move step pre/post hook tests to OptimizerInfo #119288

Uh oh!

Conversation

janeyx99 commented Feb 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Feb 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/119288

✅ No Failures

Uh oh!

mikaylagawarecki left a comment

Choose a reason for hiding this comment

Uh oh!

mikaylagawarecki Feb 6, 2024

Choose a reason for hiding this comment

Uh oh!

janeyx99 Feb 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

janeyx99 commented Feb 6, 2024 •

edited

Loading

pytorch-bot bot commented Feb 6, 2024 •

edited

Loading

janeyx99 Feb 6, 2024 •

edited

Loading