Skip to content

[Checkpoint] Add a logger to dedup_tensors #89503

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wz337 wants to merge 2 commits into from
Closed

[Checkpoint] Add a logger to dedup_tensors #89503

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e62b80c
update dedup tensors to include logging for dedup keys
wz337 Nov 22, 2022
fa7414a
update
wz337 Nov 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 21 additions & 3 deletions torch/distributed/checkpoint/dedup_tensors.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,13 +1,30 @@
# Copyright (c) Meta Platforms, Inc. and affiliates

from typing import Dict, List
import dataclasses
import logging
from typing import Dict, List

from torch.distributed.checkpoint.metadata import MetadataIndex
from torch.distributed.checkpoint.planner import SavePlan

__all__ = ["dedup_tensors"]


def init_logger() -> logging.Logger:
logger = logging.getLogger(__name__)
level = logging.INFO
logger.setLevel(level)
console = logging.StreamHandler()
formatter = logging.Formatter(
"%(asctime)s %(filename)s:%(lineno)s %(levelname)s p:%(processName)s t:%(threadName)s: %(message)s"
)
console.setFormatter(formatter)
console.setLevel(level)
logger.addHandler(console)
logger.propagate = False
return logger

logger = init_logger()

# TODO add docstring for dedup_tensors
def dedup_tensors(all_plans: List[SavePlan]) -> List[SavePlan]:
all_plans = list(all_plans)
Expand All @@ -18,12 +35,13 @@ def dedup_tensors(all_plans: List[SavePlan]) -> List[SavePlan]:

replicated_items = {k: v for k, v in key_to_plan.items() if len(v) > 1}

# Remove deplicates by always keeping the first entry.
# Remove duplicates by always keeping the first entry.
# Compute the per-rank remove set.
plan_to_keys: Dict[int, List[MetadataIndex]] = {}
for key, plans in replicated_items.items():
for plan_idx in plans[1:]:
plan_to_keys.setdefault(plan_idx, []).append(key)
logger.info(f"Duplicate keys to remove: {plan_to_keys}")

for plan_idx, keys in plan_to_keys.items():
key_set = set(keys)
Expand Down