Skip to content

Test#5659

Draft
pbarejko wants to merge 4 commits into
isaac-sim:developfrom
pbarejko:pbarejko/crash
Draft

Test#5659
pbarejko wants to merge 4 commits into
isaac-sim:developfrom
pbarejko:pbarejko/crash

Conversation

@pbarejko
Copy link
Copy Markdown
Collaborator

Description

Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context.
List any dependencies that are required for this change.

Fixes # (issue)

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (existing functionality will not work without user modification)
  • Documentation update

Screenshots

Please attach before and after screenshots of the change if applicable.

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have updated the changelog and the corresponding version in the extension's config/extension.toml file
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

@github-actions github-actions Bot added the isaac-lab Related to Isaac Lab team label May 17, 2026
@greptile-apps
Copy link
Copy Markdown
Contributor

greptile-apps Bot commented May 17, 2026

Greptile Summary

This PR adds faulthandler instrumentation at the top of test_newton_schemas.py, apparently to help diagnose a hanging or crashing test run.

  • faulthandler.enable is a reasonable safety net for catching segfaults, but faulthandler.dump_traceback_later with repeat=True and a 1-second interval will print full thread stack traces to stderr every second for the entire test run, and faulthandler.cancel_dump_traceback_later() is never called to stop the timer.
  • No other changes were made; the test logic itself is unchanged. This appears to be temporary debugging scaffolding that was not meant to be merged.

Confidence Score: 3/5

Not safe to merge — contains leftover debug instrumentation that will continuously dump thread stack traces to stderr throughout every test run.

The only change is module-level faulthandler code that fires a repeating 1-second timer dumping all thread tracebacks to stderr. The timer is never cancelled, so it runs for the entire test session, making CI logs unreadable. This is clearly diagnostic scaffolding from a debugging session that was not cleaned up before opening the PR.

source/isaaclab_newton/test/sim/test_newton_schemas.py — the faulthandler block at lines 8–17 needs to be removed or, at minimum, the dump_traceback_later call needs to be dropped.

Important Files Changed

Filename Overview
source/isaaclab_newton/test/sim/test_newton_schemas.py Adds module-level faulthandler debugging code: faulthandler.enable (reasonable) plus dump_traceback_later with a 1-second repeating timer that is never cancelled, flooding stderr throughout the test run.

Reviews (1): Last reviewed commit: "Test" | Re-trigger Greptile

Comment on lines +12 to +17
faulthandler.dump_traceback_later(
1.0,
repeat=True,
file=sys.__stderr__,
exit=False,
)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Unintentional debug code — floods stderr every second

faulthandler.dump_traceback_later with repeat=True and a 1-second timeout will dump full thread stack traces to stderr every second for the entire test run. Because faulthandler.cancel_dump_traceback_later() is never called — not even in the setup_sim teardown — the timer runs until the process exits, producing a continuous wall of noise in CI logs. This looks like diagnostic scaffolding added while investigating a hang, and should be removed before merging.

@pbarejko pbarejko marked this pull request as draft May 17, 2026 03:28
Copy link
Copy Markdown

@isaaclab-review-bot isaaclab-review-bot Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Automated Code Review

Summary

This PR appears to be a debugging/testing PR that makes significant changes to the CI pipeline for troubleshooting test_newton_schemas.py. These changes should not be merged to develop in their current state.


🚨 Critical Issues

1. CI Pipeline Disabled

Severity: CRITICAL | Files: .github/workflows/build.yaml

22 CI jobs are disabled by setting if: false:

  • All test jobs (isaaclab-core, isaaclab-assets, isaaclab-devices, etc.)
  • Docker build jobs (build-curobo-docker-image)
  • cuRobo test jobs
  • Training and benchmark jobs

Impact: Merging this would disable the entire CI test suite on the develop branch, allowing untested code to be merged.

Suggestion: These changes should remain on a feature branch for debugging only, or be reverted before merging.


2. Debug Code in Test File

Severity: HIGH | Files: source/isaaclab_newton/test/sim/test_newton_schemas.py

The following debugging instrumentation was added:

import faulthandler
import sys

faulthandler.enable(file=sys.__stderr__, all_threads=True)
faulthandler.dump_traceback_later(
    1.0,
    repeat=True,
    file=sys.__stderr__,
    exit=False,
)

This enables periodic traceback dumps every 1 second, which is useful for debugging hangs but:

  • Adds noise to test output
  • May impact test performance
  • Should be removed once debugging is complete

⚠️ Moderate Issues

3. Test Scope Narrowed

Severity: MEDIUM | Files: .github/workflows/build.yaml (line 592)

The Newton test job now includes include-files: "test_newton_schemas.py", which limits test execution to only this file. This significantly reduces test coverage for the isaaclab_newton package.


📋 Process Issues

Issue Status
PR Description ❌ Template not filled out
Linked Issue ❌ No issue referenced
Contribution Checklist ❌ All items unchecked
PR Title ⚠️ "Test" is not descriptive

💡 Recommendations

  1. Do not merge this PR to develop - The CI disabling changes would break the repository's testing infrastructure
  2. If debugging is needed, consider:
    • Creating a draft PR
    • Adding [WIP] or [DO NOT MERGE] to the title
    • Using a separate branch for CI experiments
  3. Once debugging is complete, create a new PR with only the actual fix
  4. Remove faulthandler code before any merge to main branches

This is an automated review. Please address the critical issues before requesting human review.


Update (d537628): New commit adds -vv flag to pytest in tools/conftest.py for extra verbosity. This is consistent with the debugging nature of the PR and doesn't introduce new issues. Previous concerns remain unaddressed:

  • ❌ CI pipeline still disabled
  • ❌ Debug faulthandler code still present
  • ❌ Test scope still narrowed

Update (70b2a25): New commit comments out omni.kit.telemetry extension in both apps/isaaclab.python.headless.kit and apps/isaaclab.python.kit. This appears to be another debugging/troubleshooting change (possibly to rule out telemetry as a cause of hangs). No new critical issues, but this is another change that should not be merged to develop without justification.

Previous concerns status:

  • ❌ CI pipeline still disabled (22 jobs)
  • ❌ Debug faulthandler code still present
  • ❌ Test scope still narrowed
  • ⚠️ New: Telemetry disabled without documented reason

@github-actions github-actions Bot added the isaac-sim Related to Isaac Sim team label May 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

infrastructure isaac-lab Related to Isaac Lab team isaac-sim Related to Isaac Sim team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant