Skip to content

Commit 1b60a6d

Browse files
pfalconpfalcon
committed
py: Divide "split" and "cat" phases of qstr extraction for better efficiency.
E.g. for stmhal, accumulated preprocessed output may grow large due to bloated vendor headers, and then reprocessing tens of megabytes on each build make take couple of seconds on fast hardware (=> potentially dozens of seconds on slow hardware). So instead, split once after each change, and only cat repetitively (guaranteed to be fast, as there're thousands of lines involved at most).
1 parent 8dd704b commit 1b60a6d

2 files changed

Lines changed: 14 additions & 9 deletions

File tree

‎py/makeqstrdefs.py‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,8 @@ def cat_together():
8787
if __name__ == "__main__":
8888
parser = argparse.ArgumentParser(description='Generates qstr definitions from a specified source')
8989

90+
parser.add_argument('command',
91+
help='Command (split/cat)')
9092
parser.add_argument('input_filename',
9193
help='Name of the input file (when not specified, the script reads standard input)')
9294
parser.add_argument('output_dir',
@@ -100,11 +102,9 @@ def cat_together():
100102
except OSError:
101103
pass
102104

103-
if args.input_filename:
104-
infile = open(args.input_filename, 'r')
105-
else:
106-
infile = sys.stdin
105+
if args.command == "split":
106+
with open(args.input_filename) as infile:
107+
process_file(infile)
107108

108-
file_data = process_file(infile)
109-
infile.close()
110-
cat_together()
109+
if args.command == "cat":
110+
cat_together()

‎py/mkrules.mk‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -90,9 +90,14 @@ $(HEADER_BUILD)/qstr.i.last: $(SRC_QSTR) | $(HEADER_BUILD)/mpversion.h
9090
$(CPP) $(QSTR_GEN_EXTRA_CFLAGS) $(CFLAGS) $? >$(HEADER_BUILD)/qstr.i.last; \
9191
fi
9292

93-
$(QSTR_DEFS_COLLECTED): $(HEADER_BUILD)/qstr.i.last
93+
$(HEADER_BUILD)/qstr.split: $(HEADER_BUILD)/qstr.i.last
9494
$(ECHO) "GEN $@"
95-
$(Q)$(PYTHON) $(PY_SRC)/makeqstrdefs.py $(HEADER_BUILD)/qstr.i.last $(HEADER_BUILD)/qstr $(QSTR_DEFS_COLLECTED)
95+
$(Q)$(PYTHON) $(PY_SRC)/makeqstrdefs.py split $(HEADER_BUILD)/qstr.i.last $(HEADER_BUILD)/qstr $(QSTR_DEFS_COLLECTED)
96+
$(Q)touch $@
97+
98+
$(QSTR_DEFS_COLLECTED): $(HEADER_BUILD)/qstr.split
99+
$(ECHO) "GEN $@"
100+
$(Q)$(PYTHON) $(PY_SRC)/makeqstrdefs.py cat $(HEADER_BUILD)/qstr.i.last $(HEADER_BUILD)/qstr $(QSTR_DEFS_COLLECTED)
96101

97102
# $(sort $(var)) removes duplicates
98103
#

0 commit comments

Comments
 (0)