Add an option for an alternative STEPcode generated output API - #480
Merged
Conversation
Opt in with either: exp2cxx --api-version 2 schema.exp or: cmake -DSC_EXP2CXX_API_VERSION=2 ... Version 1 remains the default, and the regression test verifies default output is byte-identical to explicit v1 output. API v2 now provides: - Compact SELECT classes backed by shared descriptor-driven runtime code instead of repeated generated parsing, assignment, and serialization bodies. - Generic Set* and typed getter APIs in include/clstepcore/sdaiSelect.h:109. - Dependency-bounded entity headers and source-owned dependencies in src/exp2cxx/classes_entity.c:968. - Out-of-line entity creators. - Unity chunks that no longer include the all-entity header. - C++11 output; C++17 did not offer enough additional benefit for this tranche. The command-line handling is in src/exp2cxx/fedex_main.c:85, while compact SELECT generation begins in src/exp2cxx/selects.c:702. Measured using the BRL-CAD ap242e4.exp, Release builds, eight parallel jobs, and 64:8 unity chunks: Metric API v1 Final API v2 Reduction ━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━ ━━━━━━━━━━━ Generated C++ lines 1,210,640 522,445 56.8% ──────────────────────── ─────────────── ────────────── ─────────── Generated bytes 57,566,051 27,202,644 52.7% ──────────────────────── ─────────────── ────────────── ─────────── Schema build wall time 183.22 s 88.31 s 51.8% ──────────────────────── ─────────────── ────────────── ─────────── Compiler CPU time 641.53 s 378.49 s 41.0% ──────────────────────── ─────────────── ────────────── ─────────── Peak RSS 1,035,112 KiB 916,412 KiB 11.5% ──────────────────────── ─────────────── ────────────── ─────────── Object-file bytes 70,472,952 40,031,088 43.2% ──────────────────────── ─────────────── ────────────── ─────────── Shared-library text 34,361,400 19,420,467 43.5% ──────────────────────── ─────────────── ────────────── ─────────── Generation wall time 1.52 s 0.69 s 54.6%
…mode
- API v2 table-driven schema, entity, type, attribute, and inheritance
initialization via include/clstepcore/ schemaInit.h.
- --late-bound/-T, producing descriptor-backed entities without
generated early-bound entity classes.
- CMake controls:
- SC_EXP2CXX_API_VERSION=2
- SC_EXP2CXX_LATE_BOUND=ON
- Runtime late-bound construction and owned attribute storage in
src/clstepcore/entityDescriptor.cc and src/clstepcore/STEPattribute.cc.
- Documentation and regression coverage, including incompatible option
handling.
AP242e4 generated-output results:
Mode Lines Bytes
━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━ ━━━━━━━━━━
Default v1 1,210,640 57.57 MB
──────────────────── ─────────── ──────────
API v2 early-bound 509,001 25.20 MB
──────────────────── ─────────── ──────────
API v2 late-bound 254,123 13.94 MB
The late-bound AP242 library compiled under C++11 and successfully
processed the BRL-CAD NIST sample: 10,609 instances
created and validated, with zero errors and warnings.
Add stable entity, type, and attribute IDs through SchemaModule. Make late-bound compatibility names opt-in, table-drive rules and complex metadata, and emit entity metadata directly into bounded chunks. AP242e4 late-bound output changed as follows: * generated files: 5,719 to 905 (-84.2%) * generated lines: 254,220 to 123,552 (-51.4%) * generated bytes: 13,942,523 to 9,804,680 (-29.7%) * generation wall time: 2.74s to 0.56s (-79.6%) * compile/link wall time: 157.68s to 37.02s (-76.5%) * compile/link peak RSS: 593,320 KiB to 199,156 KiB (-66.4%) * shared library size: 9,956,640 to 7,716,096 bytes (-22.5%) * read/write wall time: 0.26s to 0.25s ELF relocations increased from 28,843 to 39,719 due to descriptor-slot tables. The packed schema image stage is intended to remove this cost. The real AP242 sample created and validated 10,609 instances with zero errors and warnings. Rewritten output matched except for its timestamp.
Resolve each flattened inherited attribute layout once after schema initialization. Late-bound instance construction now follows descriptor and index records directly instead of recursively traversing supertypes and repeating derived and redefined attribute name searches. Keep the cache outside EntityDescriptor so its object layout and API v1 ABI remain unchanged. Invalidate cached subtype layouts when descriptor graphs are edited, and retain lazy preparation for non-module callers. For the AP242e4 sample's most common entity, constructing 1,531,000 DIRECTION instances improved from 0.84s to 0.63s (-25.0%); user time improved from 0.82s to 0.61s (-25.6%). Peak RSS increased by 0.9% and libstepcore grew by 1.6%. Twenty full read/write runs remained at 4.9s. The real file still created and validated 10,609 instances with zero errors and warnings. Its 949,685-byte rewrite changed only by its timestamp.
Replace generated SchemaModuleSlot pointer arrays with a versioned, 16-byte SchemaModuleImage. Shared initialization helpers capture entity, type, and attribute descriptors in stable generated-ID order and transfer them to SchemaModule without one relocated pointer per descriptor. Retain the slot-based initialization entry point for existing API v2 output, and leave API v1 generation unchanged. For AP242e4, generated output falls from 123,552 to 117,046 lines (-5.3%) and from 9,804,680 to 9,060,249 bytes (-7.6%). The schema module source falls from 808,907 to 64,367 bytes (-92.0%). The schema DSO falls from 7,728,384 to 7,470,304 bytes (-3.3%), while ELF relocations fall from 39,719 to 33,221 (-16.4%). Comparable clean compile times are unchanged, and paired read/write runs improve by 0.6%. The real file still creates and validates 10,609 instances with zero errors and warnings. Its 949,685-byte rewrite differs only by its timestamp.
Replace late-bound descriptor procedures with validated, offset-based schema and complex metadata images. Use an explicit per-load context for reentrant module construction while retaining the version 1 compatibility path. AP242e4 generated output falls from 9,060,249 to 4,865,657 bytes and clean compile/link time falls from 41.49 to 18.71 seconds. The schema DSO falls from 7,470,304 to 3,503,352 bytes and ELF relocations fall from 33,221 to 9,620.
Construct SELECT, enumeration, and aggregate values from descriptors so compact output needs no generated type classes or factories. Preserve enumeration elements and renamed schema items in the packed image. Make each public schema header independently includable through the small SchemaModule interface. Keep runtime and model declarations internal. AP242e4 output falls from 865 to 10 files and clean compile/link time falls from 18.71 to 1.21 seconds. The schema DSO falls from 3,503,352 to 2,178,296 bytes, with ELF relocations falling from 9,620 to 49.
Keep full descriptor metadata as the default and add an opt-in structural profile for conversion-oriented late-bound API v2 users. Strip descriptive and constraint text while retaining the complete runtime descriptor graph. Pass the profile through CMake and correct the scanner's compact late-bound file lists. AP242e4 full-to-structural measurements: Generated output falls from 54,839 lines and 3,304,007 bytes to 29,993 lines and 1,525,897 bytes. The packed schema source shrinks 68.1%. The schema DSO falls from 2,178,296 to 720,120 bytes, and peak compile RSS drops 6.8%. File count and 49 ELF relocations are unchanged. The NIST AP242e4 sample still creates and validates all 10,609 instances with no errors or warnings and writes the same 949,685-byte semantic result. Twenty read/write runs improve from 4.76 to 4.71 seconds and peak RSS falls from 82,276 to 78,900 KiB.
Write generated files to sibling temporary files and replace a target only when content differs. Track owned paths in a versioned manifest and remove obsolete generated files without touching unrelated files. Strip generator source paths from output and add regression coverage. Treat nonliteral aggregate bounds as expression records instead of reading an inactive integer union member. This also removes process addresses that made packed AP242 output nondeterministic. On AP242e4, an identical generation changes zero of ten generated source/header timestamps and leaves the 390-byte manifest unchanged. The following CMake build falls from 2.01 to 0.06 seconds and peak RSS from 118,108 to 18,432 KiB. Generation remains 0.40 seconds. The NIST AP242e4 sample still creates and validates all 10,609 instances with no errors or warnings and writes the same 949,685-byte semantic result.
- Correct ownership of stack-backed and allocated entity attributes. - Typed accessor delegation for redefined attributes. - Correct destruction when an attribute is redefined to a different type. - Complete aggregate descriptor initialization. - The API-v2 exp2python link dependency. - Regression tests covering those cases.
Provide explicit ASan, UBSan, and TSan options with compile and link probes. Keep thread instrumentation in an incompatible, standalone configuration and preserve frame pointers for useful diagnostics.
Reference validation already reports targets absent from DATA sections. Do not add those IDs to a materialization closure, where they cannot be loaded and only produce duplicate or misleading limit errors. Cover the closure behavior with the existing missing-reference fixture.
Judy nodes pack variable-width keys at byte offsets which are not guaranteed to meet judyvalue alignment. Access these keys and caller byte buffers through memcpy so lazy indexing is portable and visible to optimizers without undefined behavior. Also mask the radix iterator shift at a word boundary. Without the mask, the final byte requests a shift by 64 on 64-bit builds.
Preserve the component keyword vector already parsed while indexing a complex Part 21 instance, and expose it through a read-only lazy manager API. This lets clients interpret complex instances without reparsing raw source text or materializing the SDAI graph.
Disable leak detection only while the instrumented exp2cxx build tool generates schema sources. LeakSanitizer cannot run under common tracers and sandboxes, which otherwise makes an ASan build fail before its tests can start.
Materializing each instance repeated the same FILE_SCHEMA warning. Keep a thread-safe claim flag on the lazy file reader so concurrent loads and multiple DATA sections report the ambiguity once per input file.
When an embedding application installs a structured diagnostic callback, do not also print the same materialization errors to stderr. This keeps large batches of unsupported companion-schema entities from flooding converter output while preserving legacy standalone diagnostics.
Member
Author
|
Issue with BRL-CAD is upstream. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Was hitting scaling issues trying to use the AP242 schemeas in a converter, so adding an opt-in capability to generate a different series of output files with a different API. Optimized for performance rather than worrying about any sort of standard behavior compliance - its focus is on supporting geometry import/export for large files with the new, large schemas.