Skip to content

Add an option for an alternative STEPcode generated output API - #480

Merged
starseeker merged 17 commits into
stepcode:developfrom
starseeker:improve_output
Aug 4, 2026
Merged

Add an option for an alternative STEPcode generated output API#480
starseeker merged 17 commits into
stepcode:developfrom
starseeker:improve_output

Conversation

@starseeker

Copy link
Copy Markdown
Member

Was hitting scaling issues trying to use the AP242 schemeas in a converter, so adding an opt-in capability to generate a different series of output files with a different API. Optimized for performance rather than worrying about any sort of standard behavior compliance - its focus is on supporting geometry import/export for large files with the new, large schemas.

Opt in with either:

exp2cxx --api-version 2 schema.exp

or:

cmake -DSC_EXP2CXX_API_VERSION=2 ...

Version 1 remains the default, and the regression test verifies default
output is byte-identical to explicit v1
output.

API v2 now provides:

- Compact SELECT classes backed by shared descriptor-driven runtime code
  instead of repeated generated parsing, assignment, and serialization
  bodies.

- Generic Set* and typed getter APIs in
  include/clstepcore/sdaiSelect.h:109.
- Dependency-bounded entity headers and source-owned dependencies in
  src/exp2cxx/classes_entity.c:968.
- Out-of-line entity creators.
- Unity chunks that no longer include the all-entity header.
- C++11 output; C++17 did not offer enough additional benefit for this
  tranche.

The command-line handling is in src/exp2cxx/fedex_main.c:85, while
compact SELECT generation begins in src/exp2cxx/selects.c:702.

Measured using the BRL-CAD ap242e4.exp, Release builds, eight parallel
jobs, and 64:8 unity chunks:

 Metric                           API v1    Final API v2    Reduction
━━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━  ━━━━━━━━━━━
 Generated C++ lines           1,210,640         522,445        56.8%
────────────────────────  ───────────────  ──────────────  ───────────
 Generated bytes              57,566,051      27,202,644        52.7%
────────────────────────  ───────────────  ──────────────  ───────────
 Schema build wall time         183.22 s         88.31 s        51.8%
────────────────────────  ───────────────  ──────────────  ───────────
 Compiler CPU time              641.53 s        378.49 s        41.0%
────────────────────────  ───────────────  ──────────────  ───────────
 Peak RSS                  1,035,112 KiB     916,412 KiB        11.5%
────────────────────────  ───────────────  ──────────────  ───────────
 Object-file bytes            70,472,952      40,031,088        43.2%
────────────────────────  ───────────────  ──────────────  ───────────
 Shared-library text          34,361,400      19,420,467        43.5%
────────────────────────  ───────────────  ──────────────  ───────────
 Generation wall time             1.52 s          0.69 s        54.6%
…mode

- API v2 table-driven schema, entity, type, attribute, and inheritance
  initialization via include/clstepcore/ schemaInit.h.

- --late-bound/-T, producing descriptor-backed entities without
  generated early-bound entity classes.

- CMake controls:
    - SC_EXP2CXX_API_VERSION=2
    - SC_EXP2CXX_LATE_BOUND=ON

- Runtime late-bound construction and owned attribute storage in
  src/clstepcore/entityDescriptor.cc and src/clstepcore/STEPattribute.cc.

- Documentation and regression coverage, including incompatible option
  handling.

AP242e4 generated-output results:

 Mode                      Lines       Bytes
━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━  ━━━━━━━━━━
 Default v1            1,210,640    57.57 MB
────────────────────  ───────────  ──────────
 API v2 early-bound      509,001    25.20 MB
────────────────────  ───────────  ──────────
 API v2 late-bound       254,123    13.94 MB

The late-bound AP242 library compiled under C++11 and successfully
processed the BRL-CAD NIST sample: 10,609 instances
created and validated, with zero errors and warnings.
Add stable entity, type, and attribute IDs through SchemaModule.  Make
late-bound compatibility names opt-in, table-drive rules and complex
metadata, and emit entity metadata directly into bounded chunks.

AP242e4 late-bound output changed as follows:

* generated files: 5,719 to 905 (-84.2%)
* generated lines: 254,220 to 123,552 (-51.4%)
* generated bytes: 13,942,523 to 9,804,680 (-29.7%)
* generation wall time: 2.74s to 0.56s (-79.6%)
* compile/link wall time: 157.68s to 37.02s (-76.5%)
* compile/link peak RSS: 593,320 KiB to 199,156 KiB (-66.4%)
* shared library size: 9,956,640 to 7,716,096 bytes (-22.5%)
* read/write wall time: 0.26s to 0.25s

ELF relocations increased from 28,843 to 39,719 due to descriptor-slot
tables.  The packed schema image stage is intended to remove this cost.

The real AP242 sample created and validated 10,609 instances with zero
errors and warnings.  Rewritten output matched except for its timestamp.
Resolve each flattened inherited attribute layout once after schema
initialization.  Late-bound instance construction now follows descriptor
and index records directly instead of recursively traversing supertypes
and repeating derived and redefined attribute name searches.

Keep the cache outside EntityDescriptor so its object layout and API v1
ABI remain unchanged.  Invalidate cached subtype layouts when descriptor
graphs are edited, and retain lazy preparation for non-module callers.

For the AP242e4 sample's most common entity, constructing 1,531,000
DIRECTION instances improved from 0.84s to 0.63s (-25.0%); user time
improved from 0.82s to 0.61s (-25.6%).  Peak RSS increased by 0.9% and
libstepcore grew by 1.6%.  Twenty full read/write runs remained at 4.9s.

The real file still created and validated 10,609 instances with zero
errors and warnings.  Its 949,685-byte rewrite changed only by its
timestamp.
Replace generated SchemaModuleSlot pointer arrays with a versioned,
16-byte SchemaModuleImage.  Shared initialization helpers capture
entity, type, and attribute descriptors in stable generated-ID order
and transfer them to SchemaModule without one relocated pointer per
descriptor.

Retain the slot-based initialization entry point for existing API v2
output, and leave API v1 generation unchanged.

For AP242e4, generated output falls from 123,552 to 117,046 lines
(-5.3%) and from 9,804,680 to 9,060,249 bytes (-7.6%).  The schema
module source falls from 808,907 to 64,367 bytes (-92.0%).

The schema DSO falls from 7,728,384 to 7,470,304 bytes (-3.3%), while
ELF relocations fall from 39,719 to 33,221 (-16.4%).  Comparable clean
compile times are unchanged, and paired read/write runs improve by 0.6%.

The real file still creates and validates 10,609 instances with zero
errors and warnings.  Its 949,685-byte rewrite differs only by its
timestamp.
Replace late-bound descriptor procedures with validated, offset-based
schema and complex metadata images.  Use an explicit per-load context
for reentrant module construction while retaining the version 1
compatibility path.

AP242e4 generated output falls from 9,060,249 to 4,865,657 bytes and
clean compile/link time falls from 41.49 to 18.71 seconds.  The schema
DSO falls from 7,470,304 to 3,503,352 bytes and ELF relocations fall
from 33,221 to 9,620.
Construct SELECT, enumeration, and aggregate values from descriptors so
compact output needs no generated type classes or factories.  Preserve
enumeration elements and renamed schema items in the packed image.

Make each public schema header independently includable through the small
SchemaModule interface.  Keep runtime and model declarations internal.

AP242e4 output falls from 865 to 10 files and clean compile/link time
falls from 18.71 to 1.21 seconds.  The schema DSO falls from 3,503,352
to 2,178,296 bytes, with ELF relocations falling from 9,620 to 49.
Keep full descriptor metadata as the default and add an opt-in
structural profile for conversion-oriented late-bound API v2 users.
Strip descriptive and constraint text while retaining the complete
runtime descriptor graph.  Pass the profile through CMake and correct
the scanner's compact late-bound file lists.

AP242e4 full-to-structural measurements:

Generated output falls from 54,839 lines and 3,304,007 bytes to
29,993 lines and 1,525,897 bytes.  The packed schema source shrinks
68.1%.  The schema DSO falls from 2,178,296 to 720,120 bytes, and peak
compile RSS drops 6.8%.  File count and 49 ELF relocations are
unchanged.

The NIST AP242e4 sample still creates and validates all 10,609
instances with no errors or warnings and writes the same 949,685-byte
semantic result.  Twenty read/write runs improve from 4.76 to 4.71
seconds and peak RSS falls from 82,276 to 78,900 KiB.
Write generated files to sibling temporary files and replace a target
only when content differs.  Track owned paths in a versioned manifest
and remove obsolete generated files without touching unrelated files.
Strip generator source paths from output and add regression coverage.

Treat nonliteral aggregate bounds as expression records instead of
reading an inactive integer union member.  This also removes process
addresses that made packed AP242 output nondeterministic.

On AP242e4, an identical generation changes zero of ten generated
source/header timestamps and leaves the 390-byte manifest unchanged.
The following CMake build falls from 2.01 to 0.06 seconds and peak RSS
from 118,108 to 18,432 KiB.  Generation remains 0.40 seconds.

The NIST AP242e4 sample still creates and validates all 10,609
instances with no errors or warnings and writes the same 949,685-byte
semantic result.
- Correct ownership of stack-backed and allocated entity attributes.
- Typed accessor delegation for redefined attributes.
- Correct destruction when an attribute is redefined to a different type.
- Complete aggregate descriptor initialization.
- The API-v2 exp2python link dependency.
- Regression tests covering those cases.
Provide explicit ASan, UBSan, and TSan options with compile and
link probes.  Keep thread instrumentation in an incompatible,
standalone configuration and preserve frame pointers for useful
diagnostics.
Reference validation already reports targets absent from DATA sections.
Do not add those IDs to a materialization closure, where they cannot
be loaded and only produce duplicate or misleading limit errors.

Cover the closure behavior with the existing missing-reference fixture.
Judy nodes pack variable-width keys at byte offsets which are not
guaranteed to meet judyvalue alignment. Access these keys and caller
byte buffers through memcpy so lazy indexing is portable and visible
to optimizers without undefined behavior.

Also mask the radix iterator shift at a word boundary. Without the
mask, the final byte requests a shift by 64 on 64-bit builds.
Preserve the component keyword vector already parsed while indexing a
complex Part 21 instance, and expose it through a read-only lazy manager
API.  This lets clients interpret complex instances without reparsing raw
source text or materializing the SDAI graph.
Disable leak detection only while the instrumented exp2cxx build tool
generates schema sources.  LeakSanitizer cannot run under common tracers
and sandboxes, which otherwise makes an ASan build fail before its tests
can start.
Materializing each instance repeated the same FILE_SCHEMA warning.
Keep a thread-safe claim flag on the lazy file reader so concurrent
loads and multiple DATA sections report the ambiguity once per input
file.
When an embedding application installs a structured diagnostic callback,
do not also print the same materialization errors to stderr.  This keeps
large batches of unsupported companion-schema entities from flooding
converter output while preserving legacy standalone diagnostics.
@starseeker
starseeker merged commit c5d6b10 into stepcode:develop Aug 4, 2026
4 of 6 checks passed
@starseeker

Copy link
Copy Markdown
Member Author

Issue with BRL-CAD is upstream.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant