feat: add CoreS3 duplex Realtime voice and fluid face rendering - #634
feat: add CoreS3 duplex Realtime voice and fluid face rendering#634meganetaaan wants to merge 32 commits into
Conversation
This reverts commit bc93598.
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (9)
🚧 Files skipped from review as they are similar to previous changes (4)
📝 WalkthroughWalkthroughThe PR adds Gray16 face rendering with eased emotion transitions, CoreS3 duplex audio with echo cancellation, OpenAI Realtime conversation transport, validation workflows, smoke testing, benchmarks, and CoreS3 firmware tooling. ChangesFace rendering and emotion transitions
CoreS3 duplex voice
Firmware tooling
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0ed9846952
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Cloudflare PR previewOpen the latest preview for commit Immutable deployment: https://7555b43e.stack-chan-pr-preview.pages.dev Warning Pull request previews contain untrusted web and firmware code. Review the changes before granting WebSerial/Bluetooth permissions or flashing a device. |
There was a problem hiding this comment.
Actionable comments posted: 7
Note
Due to the large number of review comments, Critical, Major severity comments were prioritized as inline comments.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
firmware/host/modules/conversation/manifest.json (1)
35-45: 🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick winAdd
ChatAudioIOBaseresolve entries for the non-esp32 platforms, or remove the base import from the reachable modules.
./chat-audioio-stuband the mac path only exposeChatAudioIO, while the global conversation modules leaveChatAudioIOBaseempty../chat-audioio/worker-stackimportsChatAudioIOBase, so a build using the mac or default platform can fail instead of getting the intended unavailable implementation.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/conversation/manifest.json` around lines 35 - 45, Update the non-esp32 platform entries in the conversation manifest to resolve ChatAudioIOBase alongside ChatAudioIO, including the mac include path and default stub path, or remove the ChatAudioIOBase import from reachable modules such as ./chat-audioio/worker-stack. Ensure mac and default builds no longer leave ChatAudioIOBase unresolved while preserving the intended unavailable implementation.
🟡 Minor comments (17)
firmware/host/modules/testing/module-structure.architecture.ts-373-374 (1)
373-374: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winTest the draw boundary instead of the source expression.
Lines 373-374 require one exact bitwise expression and one helper absence. An equivalent numeric color-packing implementation fails even when
drawTexturereceives the same packed RGBA value. Assert the numeric argument at the drawing boundary with an XS-driven or instrumented rendering test.As per path instructions, “Flag tests that read production source as text only to re-assert constants, configuration values, or implementation fragments.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/testing/module-structure.architecture.ts` around lines 373 - 374, Replace the source-text assertions in the emoticon test with an instrumented or XS-driven rendering test that observes the numeric color argument passed to drawTexture. Validate the packed RGBA value at the draw boundary rather than requiring the specific bitwise expression or checking for colorString absence.Sources: Coding guidelines, Path instructions
firmware/host/modules/audio/__tests__/audio-duplex-device/manifest.json-3-6 (1)
3-6: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winAdd the ESP-IDF I2C master dependency for
amp-readback.
amp-readback.cuses theesp_driver_i2cmaster APIs, but this manifest only declaresesp_driver_i2s. Add anidfdependency onesp_driver_i2cso the ESP32 build links the required I2C master driver.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/audio/__tests__/audio-duplex-device/manifest.json` around lines 3 - 6, Update the manifest’s amp-readback module configuration to declare the ESP-IDF dependency on esp_driver_i2c alongside its existing dependencies, ensuring the build links the I2C master APIs used by amp-readback.c.firmware/host/modules/conversation/chat-audioio/duplex-chat-audioio.js-288-291 (1)
288-291: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winGuard
sourcePreRollBytesagainst rounding to zero.
sourcePreRollBytesroundspreRollBytesfrom the transport sample rate down to the hardware rate. For smallpreRollByteswith a highinputSampleRate(for examplepreRollBytes = 2at 48000 Hz), the expression rounds to0.CircularByteHistorythen throwsRangeError('history capacity must be positive')inside the worker message handler, so the gracefulChatAudioIOBase.FAILEDpath above is bypassed. Clamp the value to at least one sample frame.🛡️ Proposed clamp
- const sourcePreRollBytes = - Math.round((preRollBytes * HARDWARE_SAMPLE_RATE) / this.inputSampleRate / PCM_BYTES_PER_SAMPLE) * - PCM_BYTES_PER_SAMPLE + const sourcePreRollBytes = Math.max( + PCM_BYTES_PER_SAMPLE, + Math.round((preRollBytes * HARDWARE_SAMPLE_RATE) / this.inputSampleRate / PCM_BYTES_PER_SAMPLE) * + PCM_BYTES_PER_SAMPLE, + )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/conversation/chat-audioio/duplex-chat-audioio.js` around lines 288 - 291, Update the sourcePreRollBytes calculation in the duplex audio initialization flow to clamp the rounded value to at least one PCM sample frame before constructing CircularByteHistory. Preserve the existing sample-rate conversion and byte alignment, ensuring small preRollBytes values cannot produce zero capacity and bypass the ChatAudioIOBase.FAILED path.firmware/host/modules/conversation/chat-audioio/manifest.json-15-23 (1)
15-23: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winDocument or include TLS roots for all
wss://Realtime endpoints.
providerIDaccepts anywss://host, butchat-audioioonly bundlesca35,ca109,ca222,ca233, andca236. If the device can be configured for an OpenAI Realtimewss://api.openai.com/v1/realtimesession or another trusted upstream, document the expected root. If it can connect to an arbitrary server-backed endpoint, add the operator-selected roots and explain why.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/conversation/chat-audioio/manifest.json` around lines 15 - 23, Update the manifest data roots for chat-audioio to cover every supported wss:// Realtime endpoint: add the required OpenAI or other trusted upstream root certificates, or document the expected roots and configuration when operators select arbitrary server-backed endpoints. Keep the existing CA entries and align the documentation with the accepted providerID behavior.firmware/mods/examples/chat_audioio/mod.js-21-24 (1)
21-24: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winAlign the device smoke timeout with the host runner timeout.
DEFAULT_SMOKE_TIMEOUT_MSis 300000 ms.firmware/scripts/run-chat-smoke.mjsdefaults its run timeout to 120000 ms. In a default run the host aborts first, so[ChatSmoke] FAIL reason=timeoutnever reaches the log and the device never runs its own cleanup path. Lower the device default below the host default, or document the requiredsmokeTimeoutMsvalue for the runner.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/mods/examples/chat_audioio/mod.js` around lines 21 - 24, Update DEFAULT_SMOKE_TIMEOUT_MS in the chat audio smoke configuration to a value below the host runner’s 120000 ms default, ensuring the device emits its timeout failure and executes cleanup before the host aborts.firmware/mods/examples/chat_audioio/mod.js-569-579 (1)
569-579: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winDocument or enforce
autoStopfor smoke runs.The evaluator in
firmware/scripts/lib/chat-smoke-log.mjs(lines 78 and 141) upgrades a PASS topassedonly afteronStateChanged: disconnected. This code stops the chat only whenautoStopis true. If a smoke configuration enablesautoStartwithoutautoStop, every run fails with "chat did not disconnect within ... ms after PASS" even though the conversation succeeded. Stop the chat after a recorded PASS in smoke mode, or state theautoStoprequirement where the smoke configuration is documented.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/mods/examples/chat_audioio/mod.js` around lines 569 - 579, Ensure smoke runs always disconnect after recording a PASS by updating the response-complete handling around smokeResultRecorded and stopChat, rather than depending solely on autoStop. Preserve the delayed AUTO_STOP_DELAY_MS timer behavior where applicable, and update the related smoke configuration documentation if autoStop remains a required setting.firmware/scripts/capture-aec-waveforms.mjs-33-33 (1)
33-33: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winValidate the capture timeout.
Number.parseIntreturns NaN for a non-numericSTACKCHAN_AEC_CAPTURE_TIMEOUT_MS.setTimeout(fn, NaN)runs on the next tick, so the capture aborts right after the device is flashed and the failure reason reads "NaN ms以内に...".firmware/scripts/run-chat-smoke.mjsvalidates the same kind of value withpositiveInteger. Apply the same validation here.🐛 Proposed fix
-const timeoutMs = Number.parseInt(process.env.STACKCHAN_AEC_CAPTURE_TIMEOUT_MS ?? '120000', 10) +const rawTimeoutMs = process.env.STACKCHAN_AEC_CAPTURE_TIMEOUT_MS ?? '120000' +if (!/^[1-9][0-9]*$/.test(rawTimeoutMs)) { + console.error(`STACKCHAN_AEC_CAPTURE_TIMEOUT_MS は正の整数で指定してください: ${rawTimeoutMs}`) + process.exit(1) +} +const timeoutMs = Number.parseInt(rawTimeoutMs, 10)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/scripts/capture-aec-waveforms.mjs` at line 33, Validate STACKCHAN_AEC_CAPTURE_TIMEOUT_MS using the existing positiveInteger pattern from run-chat-smoke.mjs before assigning timeoutMs. Ensure non-numeric, zero, and negative values are rejected or replaced with the default so setTimeout always receives a valid positive integer and the timeout message never contains NaN.firmware/scripts/run-chat-smoke.mjs-25-29 (1)
25-29: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winResolve the default artifact directory against the firmware directory.
path.resolveapplies the process cwd to the relative fallback. If a caller runs the script from the repository root, logs land in<repo>/dist/chat-smoke, outsidefirmware/dist/, where the firmware ignore rules do not apply.firmware/scripts/capture-aec-waveforms.mjsanchors its default output tofirmwareDirectory. Use the same anchor.🐛 Proposed fix
const outputRoot = path.resolve( readOption(rawArguments, 'output') ?? process.env.STACKCHAN_CHAT_SMOKE_OUTPUT ?? - path.join('dist/chat-smoke', timestamp()), + path.join(firmwareDirectory, 'dist/chat-smoke', timestamp()), )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/scripts/run-chat-smoke.mjs` around lines 25 - 29, Update the outputRoot initialization to resolve the default artifact path against the existing firmwareDirectory, matching capture-aec-waveforms.mjs, while preserving explicitly supplied output options and environment values.firmware/host/modules/audio/__tests__/audio-duplex-waveform/README.md-20-20 (1)
20-20: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winRemove the account-specific setup command.
Line 20 names a path in one developer account. The documented capture workflow fails in other checkouts. Document a repository-supported setup prerequisite instead of a user home path.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/audio/__tests__/audio-duplex-waveform/README.md` at line 20, Replace the account-specific source command in the audio-duplex-waveform README capture workflow with the repository-supported setup prerequisite, using a portable repository-relative instruction that works across developer environments.firmware/host/modules/audio/__tests__/audio-duplex-waveform/manifest.json-9-18 (1)
9-18: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winAlign
captureSampleswith the requested capture duration.At 16 kHz, both manifests request longer capture windows than
captureSamplesallows, andmain.jsusesmaxSamples: CAPTURE_SAMPLESfor the diagnostics buffer. Increase the configured values to covercaptureMs, or document this as intentional bounded capture.
firmware/host/modules/audio/__tests__/audio-duplex-waveform/manifest.json:captureSamples= 16384 covers only 1024 ms; request iscaptureMs= 1200.firmware/host/modules/audio/__tests__/audio-duplex-waveform/manifest.doubletalk.json:captureSamples= 131072 covers only 8192 ms; request iscaptureMs= 8500.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/audio/__tests__/audio-duplex-waveform/manifest.json` around lines 9 - 18, Increase captureSamples in firmware/host/modules/audio/__tests__/audio-duplex-waveform/manifest.json (lines 9-18) to cover the configured 1200 ms captureMs, and increase it in firmware/host/modules/audio/__tests__/audio-duplex-waveform/manifest.doubletalk.json (lines 4-12) to cover the configured 8500 ms captureMs. Keep the diagnostics buffer in main.js fully sized for each requested capture duration.Source: Coding guidelines
firmware/scripts/prepare-openai-realtime.mjs-38-42 (1)
38-42: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winReject an option when its value is another option.
readOption()accepts--interactiveas the value of a preceding bare--output. The generator then writes the manifest to a file named--interactiveinstead of stopping with a usage error. Reject missing detached values and detached values that start with--.Proposed parser change
function readOption(values, name) { const prefix = `--${name}=` const index = values.indexOf(`--${name}`) - if (index >= 0) return values[index + 1] - return values.find((value) => value.startsWith(prefix))?.slice(prefix.length) + if (index >= 0) { + const value = values[index + 1] + if (!value || value.startsWith('--')) { + throw new Error(`--${name} requires a value`) + } + return value + } + + const argument = values.find((value) => value.startsWith(prefix)) + if (!argument) return undefined + const value = argument.slice(prefix.length) + if (!value) throw new Error(`--${name} requires a value`) + return value }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/scripts/prepare-openai-realtime.mjs` around lines 38 - 42, Update readOption() so bare options require a following value: return no value when the next argument is missing or starts with "--", while preserving the existing inline --name=value parsing. Ensure callers treat the rejected result as a usage error rather than using another option as a filename.firmware/host/modules/conversation/chat.ts-14-17 (1)
14-17: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winAdd the missing constructor-option contract test.
The existing test covers
endpointforwarding throughproviderID, but it does not cover fallback precedence forproviderIDorendpoint. Add an injected-constructor behavior test that asserts explicitendpointtakes precedence over existingproviderID, while existingproviderIDfalls back whenendpointis absent.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/conversation/chat.ts` around lines 14 - 17, Extend the ChatConfig constructor-option tests to cover fallback precedence: verify an explicit endpoint is forwarded instead of an existing providerID, and verify providerID is used when endpoint is absent. Use the existing injected-constructor test setup and preserve the current endpoint-forwarding assertion.firmware/host/modules/ui/components/face/parts/dog/mouth.ts-26-41 (1)
26-41: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winDerive the curve depth from the mouth height instead of the constant 20.
Lines 27-29 size the mask from
maxHeight, but lines 40-41 place the control points aty + 20regardless ofmaxHeight. With the defaults the curve fits. If a caller passes a smallermaxHeight, the curve extends past the mask and the native coverage writers discard the out-of-range pixels, so the mouth is silently clipped. Scale the control offset withmouthHeight, or clamp the mask height to the drawn extent.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/ui/components/face/parts/dog/mouth.ts` around lines 26 - 41, Update updateMask so the cubic-curve control-point depth is derived from the current mouthHeight instead of the fixed 20 value, ensuring the drawn mouth remains within the mask for smaller maxHeight values while preserving the existing curve shape.firmware/host/modules/ui/components/face/parts/gray16-mask-raster.c-287-295 (1)
287-295: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winCombine the top and bottom coverage additively when both edges fall on one row.
When the eyelid closes,
topandbottomconverge, sotopWholeequalsbottomWhole. Line 290 sets that row's coverage totop - topWhole, and line 292 then callsstackchanGray16AddCoverage, which keeps the larger coverage instead of the sum. A fully closed eyelid therefore keeps one partially transparent row across the iris. Sum the two contributions for the shared row.🐛 Proposed fix
- if ((topWhole >= 0) && (topWhole < mask.height)) - stackchanGray16SetCoverage(&mask, x, topWhole, top - topWhole); - if ((bottomWhole >= 0) && (bottomWhole < mask.height)) - stackchanGray16AddCoverage(&mask, x, bottomWhole, 1 - (bottom - bottomWhole)); + if (topWhole == bottomWhole) { + if ((topWhole >= 0) && (topWhole < mask.height)) + stackchanGray16SetCoverage( + &mask, x, topWhole, (top - topWhole) + (1 - (bottom - bottomWhole))); + } + else { + if ((topWhole >= 0) && (topWhole < mask.height)) + stackchanGray16SetCoverage(&mask, x, topWhole, top - topWhole); + if ((bottomWhole >= 0) && (bottomWhole < mask.height)) + stackchanGray16AddCoverage(&mask, x, bottomWhole, 1 - (bottom - bottomWhole)); + }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/ui/components/face/parts/gray16-mask-raster.c` around lines 287 - 295, Update the coverage handling around stackchanGray16SetCoverage and stackchanGray16AddCoverage so that when topWhole equals bottomWhole, the top and bottom edge contributions are summed on that shared row rather than merged by the existing max-preserving operation. Preserve the current behavior when the edges fall on different rows.firmware/host/modules/ui/components/face/parts/gray16-mask-raster.c-22-29 (1)
22-29: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winCompute the buffer-size guard without signed overflow.
Line 27 multiplies
mask->strideWidth * mask->heightinint. Large dimensions overflow before the cast, so the guard passes and the coverage writers then index pastbyteLength. The per-pixel checks instackchanGray16SetCoverageandstackchanGray16AddCoveragetest width and height only, so they do not stop the out-of-range offset. Widen the arithmetic and reject values that cannot fit.🛡️ Proposed fix
if ( (mask->width <= 0) || (mask->height <= 0) || (mask->strideWidth < mask->width) || (mask->strideWidth & 1) || - (((xsUnsignedValue)(mask->strideWidth * mask->height) >> 1) > mask->byteLength) + (mask->strideWidth > 0x7FFF) || + (mask->height > 0x7FFF) || + ((((uint32_t)mask->strideWidth * (uint32_t)mask->height) >> 1) > (uint32_t)mask->byteLength) ) xsRangeError("invalid Gray16 mask");🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/ui/components/face/parts/gray16-mask-raster.c` around lines 22 - 29, Update the buffer-size guard in the mask validation block to perform the stride-by-height multiplication in a sufficiently wide unsigned type before shifting or comparing with byteLength, preventing signed overflow. Ensure dimensions whose required buffer size cannot be represented or exceeds mask->byteLength are rejected before stackchanGray16SetCoverage and stackchanGray16AddCoverage can write.firmware/host/modules/ui/components/face/__tests__/face-rendering/face-rendering.test.ts-479-479 (1)
479-479: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winUpdate the stale assertion message.
The assertion now compares eyelid mask revisions. The message still says "eyelid outlines".
🐛 Proposed fix
-assert(blinkChanged, 'default FaceBehavior should change eyelid outlines for blinking') +assert(blinkChanged, 'default FaceBehavior should change the eyelid mask for blinking')🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/ui/components/face/__tests__/face-rendering/face-rendering.test.ts` at line 479, Update the assertion message in the face-rendering test to refer to eyelid mask revisions rather than eyelid outlines, while preserving the existing blinkChanged assertion and its behavior.firmware/benchmarks/face-rendering/manifest.json-5-9 (1)
5-9: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winRemove the unused i2s decoder modules from the face-rendering manifests.
firmware/benchmarks/face-rendering/manifest.jsonandmanifest.visual.jsondirectly includesbc_decoderanddvi_adpcm_decode, but face-rendering sources only measure face/effects rendering and instrumentation counters, andhost/modules/ui/manifest.jsondoes not transitively reference them. Remove these entries unless a non-tracked dependency intentionally requires them.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/benchmarks/face-rendering/manifest.json` around lines 5 - 9, Remove the unused $(MODULES)/pins/i2s/sbc_decoder and $(MODULES)/pins/i2s/dvi_adpcm_decode entries from the "*" dependency list in the face-rendering manifest, and apply the same change to manifest.visual.json. Preserve the instrumentation-control entry and verify no non-tracked dependency requires the decoder modules.
🧹 Nitpick comments (12)
firmware/host/modules/conversation/chat-audioio/openai-realtime-model.js (1)
12-18: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDocument or separate the two meanings of barrier index 1.
INPUT_ACK_BARRIER_INDEXandINPUT_RING_HEAD_BARRIER_INDEXare both 1. Line 130 storesmessage.sequenceat index 1, and line 146 reads index 1 as the ring head. Today the two uses cannot overlap: the constructor always requestsmode: 'shared-ring'at line 69, andpumpInputAudiocallssendAudiowithout asequence, so line 129 skips the store. If a caller ever sends audio with asequencewhile shared-ring mode is active, the store overwrites the ring head andpumpInputAudiocomputes a wrongavailableand offset.Add a comment that states the two barrier layouts, or gate the ack store on the active transport mode.
♻️ Proposed clarification
-const INPUT_ACK_BARRIER_INDEX = 1 -const INPUT_RING_HEAD_BARRIER_INDEX = 1 +/* + * The barrier layout depends on the input transport mode. + * message transport: [reserved, ack sequence] + * shared-ring transport: [reserved, ring head, ring tail, gate] + * Index 1 therefore carries the ack sequence only in message transport. + */ +const INPUT_ACK_BARRIER_INDEX = 1 +const INPUT_RING_HEAD_BARRIER_INDEX = 1🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/conversation/chat-audioio/openai-realtime-model.js` around lines 12 - 18, Add an explanatory comment beside INPUT_ACK_BARRIER_INDEX and INPUT_RING_HEAD_BARRIER_INDEX documenting that index 1 is an acknowledgement sequence in direct transport but the ring head in shared-ring transport, and that these layouts must not be mixed. Keep the existing shared-ring behavior unchanged.firmware/host/modules/testing/fakes/audio-in.ts (1)
36-47: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRetain the unread remainder on a partial buffer read.
When the queued chunk is larger than
target, the fake copies the prefix and discards the rest. A real driver keeps the remaining bytes for the nextread(). With the current fake, a test that queues a chunk larger than the caller's target cannot detect a lost-offset bug inMicrophone.record. Push the remainder back to the front of the queue.♻️ Proposed change
const source = new Uint8Array(chunk) const byteLength = Math.min(source.byteLength, target.byteLength) target.set(source.subarray(0, byteLength)) + if (byteLength < source.byteLength) { + chunks.unshift(source.slice(byteLength).buffer) + } return byteLength🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/testing/fakes/audio-in.ts` around lines 36 - 47, Update the read method to preserve partial-read data: when target is a Uint8Array and byteLength is smaller than source.byteLength, push the unconsumed suffix back to the front of chunks before returning. Keep the existing prefix copy and return behavior unchanged, and do not alter numeric-target reads.firmware/host/modules/conversation/manifest.json (1)
13-34: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueCollapse the identical esp32 platform blocks with manifest
includerather than comma-separated keys.The Moddable manifest
platformsobject uses one key per platform, soesp32/m5stack_cores3,esp32/m5stackchan_cores3,esp32/stackchan_rtis not valid. Keep the three platform entries only if the intent is to target these specific subplatforms; otherwise, move the shared platform configuration to an included manifest.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/conversation/manifest.json` around lines 13 - 34, Replace the duplicated platform entries in the platforms configuration with a shared included manifest, using the manifest include mechanism rather than comma-separated platform keys. Preserve the ChatAudioIO and ChatAudioIOBase mappings for esp32/m5stack_cores3, esp32/m5stackchan_cores3, and esp32/stackchan_rt, or retain separate valid entries if platform-specific targeting is required.firmware/host/modules/audio/__tests__/audio-duplex-acoustic/main.js (1)
111-119: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRestore a safe amplifier attenuation before the test ends.
The test arms the AW88298 at −60 dB and never restores a quiet level. The waveform test restores
SAFE_HARDWARE_ATTENUATION_DBinstopSafely(). If a later firmware image or a reset starts playback, the device stays loud. Set the safe attenuation after the streams stop, and set it also on the failure path before the throw.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/audio/__tests__/audio-duplex-acoustic/main.js` around lines 111 - 119, Update the duplex test cleanup around the Timer callback and stopSafely() so the amplifier is set to SAFE_HARDWARE_ATTENUATION_DB after both streams stop, before diagnostics or close; also apply the same attenuation on the failure path before throwing. Reuse the existing safe-attenuation constant and preserve the current shutdown and analysis flow.firmware/host/modules/audio/__tests__/audio-duplex-waveform/main.js (1)
60-83: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueExtract the shared device audio test helpers. Both CoreS3 device tests carry their own copy of the probe generator and the AW88298 register packing. The root cause is a missing shared fixture module for these tests, so the two copies can drift from the production packing in
firmware/host/modules/audio/platforms/m5stackchan-cores3/audio-duplex.js.
firmware/host/modules/audio/__tests__/audio-duplex-waveform/main.js#L60-L83: importbuildProbe,amplifierVolumeByte, andamplifierVolumeRegisterfrom a shared helper module, and keep only the localPROBE_PEAKand attenuation arguments here.firmware/host/modules/audio/__tests__/audio-duplex-acoustic/main.js#L21-L50: remove the localamplifierVolumeRegisterandbuildProbecopies and import the same helper module, adding it to this test's manifest.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/audio/__tests__/audio-duplex-waveform/main.js` around lines 60 - 83, Extract the shared audio test helpers into a fixture module based on the production AW88298 packing in audio-duplex.js. In firmware/host/modules/audio/__tests__/audio-duplex-waveform/main.js lines 60-83, remove the local probe generator and import buildProbe, amplifierVolumeByte, and amplifierVolumeRegister, retaining only local PROBE_PEAK and attenuation arguments. In firmware/host/modules/audio/__tests__/audio-duplex-acoustic/main.js lines 21-50, remove the local amplifierVolumeRegister and buildProbe implementations, import the shared helpers, and add the helper module to that test’s manifest.firmware/scripts/capture-aec-waveforms.mjs (1)
130-142: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winImport the shared xsbug decoder.
This function is identical to
decodeXsbugLoginfirmware/scripts/lib/chat-smoke-log.mjs. Two copies of the entity table will drift. Import the shared helper instead.♻️ Proposed refactor
+import { decodeXsbugLog } from './lib/chat-smoke-log.mjs' import { startXsbugServer } from './lib/xsbug-log-server.js'-function decodeXsbugLog(log) { - return Array.from(log.matchAll(/<log(?:\s[^>]*)?>([\s\S]*?)<\/log>/g), ([, text]) => text) - .join('') - .replaceAll('&', '&') - .replaceAll('&`#10`;', '\n') - .replaceAll('&`#13`;', '\r') - .replaceAll('&`#34`;', '"') - .replaceAll('&`#39`;', "'") - .replaceAll('<', '<') - .replaceAll('>', '>') - .replaceAll('"', '"') - .replaceAll(''', "'") -}🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/scripts/capture-aec-waveforms.mjs` around lines 130 - 142, Remove the local decodeXsbugLog implementation and import the shared decodeXsbugLog helper from firmware/scripts/lib/chat-smoke-log.mjs. Update all existing call sites in capture-aec-waveforms.mjs to use the imported function without changing decoding behavior.firmware/host/modules/ui/components/effects/music-notes.ts (1)
85-109: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueExtract the shared sprite-position computation.
onDrawandinvalidateSpriteboth deriveprogress,rise, and the sprite top from#elapsed. The invalidated rectangle must match the drawn rectangle exactly. If one copy changes later, the note leaves residue on screen because the parent skin repaints only the invalidated region. A single private helper removes that coupling.♻️ Proposed refactor
+ `#spriteTop`(): number { + const progress = this.#elapsed / FADE_DURATION_MS + const rise = Math.round(RISE_PIXELS * progress * (2 - progress)) + return RISE_PIXELS - rise + } + onDraw(port: PiuPort, x = 0, y = 0, width = SPRITE_SIZE, height = NOTE_HEIGHT) { if (this.#elapsed >= FADE_DURATION_MS) return - const progress = this.#elapsed / FADE_DURATION_MS - const rise = Math.round(RISE_PIXELS * progress * (2 - progress)) - const spriteY = RISE_PIXELS - rise + const progress = this.#elapsed / FADE_DURATION_MS + const spriteY = this.#spriteTop() if (x >= SPRITE_SIZE || x + width <= 0 || y >= spriteY + SPRITE_SIZE || y + height <= spriteY) return @@ private invalidateSprite(port: PiuPort): void { if (this.#elapsed >= FADE_DURATION_MS) return - const progress = this.#elapsed / FADE_DURATION_MS - const rise = Math.round(RISE_PIXELS * progress * (2 - progress)) - port.invalidate(0, RISE_PIXELS - rise, SPRITE_SIZE, SPRITE_SIZE) + port.invalidate(0, this.#spriteTop(), SPRITE_SIZE, SPRITE_SIZE) }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/ui/components/effects/music-notes.ts` around lines 85 - 109, Extract the shared progress, rise, and sprite-top calculation from onDraw and invalidateSprite into a single private helper, then use its returned position in both methods. Preserve the existing fade-duration early returns and ensure invalidateSprite uses exactly the same sprite top and dimensions as onDraw.firmware/benchmarks/face-rendering/manifest.visual.json (1)
5-8: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDocument why the visual benchmark includes the i2s decoder modules.
The visual benchmark does not reference
sbc_decoder,dvi_adpcm_decode,AudioOut,AudioIn, or audio startup behavior invisual.tsorfirmware/host/modules/ui/manifest.json. If these modules are required by a shared module/import path for this build target, add a document-level reason next to the entries; otherwise remove them to keep the benchmark build small.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/benchmarks/face-rendering/manifest.visual.json` around lines 5 - 8, Review the module entries in the visual benchmark manifest and either remove sbc_decoder and dvi_adpcm_decode if they are unused, or retain them with a document-level explanation of the shared import/build dependency requiring both modules. Keep the benchmark manifest limited to modules necessary for this build target.firmware/host/modules/ui/components/face/parts/gray16-mask.ts (1)
32-39: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRemove the extra
Bitmapconstructor argument.
commodetto/Bitmapis defined with constructor argumentswidth,height,format,buffer, andoffset; this call passesthis.bytes.byteLengthas a sixth argument. Drop it to keep the constructor call aligned with the documented SDK API.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/ui/components/face/parts/gray16-mask.ts` around lines 32 - 39, Update the Bitmap constructor call in the gray16 mask initialization to remove the trailing this.bytes.byteLength argument, retaining width, height, format, buffer, and byteOffset in the documented five-argument order.firmware/host/modules/ui/components/face/parts/__tests__/unit-steps.test.ts (1)
6-11: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd the clamping and non-finite cases.
quantizeUnitreturns 0 forNaNand for negative input, andunitFromStepclamps steps outside0..UNIT_OPEN_STEPS. Those branches drive every mask update, and the current test does not reach them.♻️ Proposed additional assertions
assert.equal(unitFromStep(quantizeUnit(0.5)), 0.5) }) + +test('out-of-range and non-finite ratios clamp onto the unit scale', () => { + assert.equal(quantizeUnit(Number.NaN), 0) + assert.equal(quantizeUnit(-0.5), 0) + assert.equal(quantizeUnit(2), UNIT_OPEN_STEPS) + assert.equal(unitFromStep(-1), 0) + assert.equal(unitFromStep(UNIT_OPEN_STEPS + 1), 1) +})🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/ui/components/face/parts/__tests__/unit-steps.test.ts` around lines 6 - 11, Extend the `animation ratios quantize onto the bounded unit scale` test to cover `quantizeUnit` returning 0 for `NaN` and negative inputs, and `unitFromStep` clamping steps below 0 and above `UNIT_OPEN_STEPS` to the corresponding unit bounds.firmware/host/modules/ui/state/face-state.architecture.ts (1)
89-92: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueNarrow the allocation matchers so comments do not fail the test.
/\bnew\b/matches the wordnewin a comment or string inside the extractedonFaceStateblock./\.\.\./also matches a rest parameter in a nested function. A future comment such as// reuse instead of new objectsthen fails the test with no regression present. Strip line comments from each block, or match the construction formnew <Identifier>(and the object-spread form.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/ui/state/face-state.architecture.ts` around lines 89 - 92, Narrow the allocation checks in the block-validation loop around onFaceState so comments and strings do not trigger failures. Replace the broad new and spread patterns with matchers for actual constructor expressions and object-spread syntax, or strip line comments before matching; preserve detection of allocations while allowing nested rest parameters.firmware/host/modules/ui/components/face/__tests__/face-rendering/face-rendering.test.ts (1)
132-145: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winDocument
fillOutsideApertureas resetting the mask before drawing.
fillOutsideAperturestarts withmemset(..., 0xFF, ...), so it clears existing raster output, includingfillCircle(4, 4, 2). Add a short API note or useclear()/separate mask semantics in the test so future code does not rely on an inferred reset behavior.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/ui/components/face/__tests__/face-rendering/face-rendering.test.ts` around lines 132 - 145, Document in the test or API-facing definition of fillOutsideAperture that it resets the mask before drawing, clearing prior raster output such as fillCircle results. Add a concise note or explicitly clear/use separate mask state before invoking fillOutsideAperture, without relying on inferred reset behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@firmware/host/modules/audio/platforms/m5stackchan-cores3/audio-duplex.c`:
- Around line 1295-1298: Guard AEC notifications in both the input-task branch
around audioDuplexPostInput and audioDuplexOutputTask: under audio->mutex, copy
audio->aecTask to a local TaskHandle_t, release the mutex, and call
xTaskNotifyGive only when the copied handle is non-NULL. Apply the same pattern
to both notify paths while preserving the existing non-AEC behavior.
In `@firmware/host/modules/conversation/chat-audioio/duplex-chat-audioio.js`:
- Around line 585-614: Update processInputAudio to resample the passed source
buffer rather than this.inputResampleSource in every resampleAndQueueInput call.
Derive the correct buffer and offset from source so pumpInputProbe uses its
probe view, while preserving the existing sourceSamples length and gate
behavior.
In `@firmware/host/modules/conversation/chat-audioio/openai-realtime-model.js`:
- Around line 79-87: Update the open-audioio module’s base-model import to use
the manifest-exposed stackchanOpenAIRealtimeModel symbol so the worker loads
successfully. In the probe configuration block, assign silence_duration_ms
through the session path initialized by the base class’s configure() method,
removing any unnecessary existence guard while preserving the resolved silence
duration.
In
`@firmware/host/modules/conversation/chat-audioio/server-chat-websocket-worker.js`:
- Around line 52-59: Reject or prevent connection setup in the websocket
configuration flow when an apiKey is present and the selected endpoint is
insecure ws://; enforce wss:// for Bearer-authenticated connections. Update the
logic around ServerOpenAIRealtimeModel.configure and the WebSocketClient setup
so Authorization is never sent through device.network.ws, while preserving
unauthenticated ws:// support if intended.
In
`@firmware/host/modules/conversation/chat-audioio/server-openai-realtime-model.js`:
- Around line 55-62: Remove the local input_audio_buffer.commit call from the
audio/VAD handling associated with the server_vad turn_detection configuration.
Keep local VAD state tracking intact, but ensure this.sendJSON({ type:
'input_audio_buffer.commit' }) is not invoked while server VAD is active.
In `@firmware/host/modules/ui/state/face-state.architecture.ts`:
- Around line 102-104: Replace the gray16-mask.ts source-text assertions in the
architecture test with behavior-based verification using the importable
Gray16Mask API, preserving the existing PocoGrayBitmapDraw and
PiuViewDrawContent checks. Remove assertions that only match the Gray16Mask
declaration and fillOutsideAperture/fillRoundRect method names, relying on
existing pixel-behavior coverage where applicable.
In `@firmware/scripts/lib/idf-dependencies.mjs`:
- Around line 60-69: The lockIsStale check currently only verifies that each
dependency name exists in the lock file but does not validate that the locked
version matches the manifest requirement. Update the staleness check to parse
the version value from each dependency lock entry and compare it against the
corresponding version constraint from the dependencies array, marking the lock
as stale when either the name is missing or the version differs. Keep the
existing invalidation logic for sdkconfigHeaderPath unchanged so it removes the
stale configuration file when a version mismatch is detected.
---
Outside diff comments:
In `@firmware/host/modules/conversation/manifest.json`:
- Around line 35-45: Update the non-esp32 platform entries in the conversation
manifest to resolve ChatAudioIOBase alongside ChatAudioIO, including the mac
include path and default stub path, or remove the ChatAudioIOBase import from
reachable modules such as ./chat-audioio/worker-stack. Ensure mac and default
builds no longer leave ChatAudioIOBase unresolved while preserving the intended
unavailable implementation.
---
Minor comments:
In `@firmware/benchmarks/face-rendering/manifest.json`:
- Around line 5-9: Remove the unused $(MODULES)/pins/i2s/sbc_decoder and
$(MODULES)/pins/i2s/dvi_adpcm_decode entries from the "*" dependency list in the
face-rendering manifest, and apply the same change to manifest.visual.json.
Preserve the instrumentation-control entry and verify no non-tracked dependency
requires the decoder modules.
In `@firmware/host/modules/audio/__tests__/audio-duplex-device/manifest.json`:
- Around line 3-6: Update the manifest’s amp-readback module configuration to
declare the ESP-IDF dependency on esp_driver_i2c alongside its existing
dependencies, ensuring the build links the I2C master APIs used by
amp-readback.c.
In `@firmware/host/modules/audio/__tests__/audio-duplex-waveform/manifest.json`:
- Around line 9-18: Increase captureSamples in
firmware/host/modules/audio/__tests__/audio-duplex-waveform/manifest.json (lines
9-18) to cover the configured 1200 ms captureMs, and increase it in
firmware/host/modules/audio/__tests__/audio-duplex-waveform/manifest.doubletalk.json
(lines 4-12) to cover the configured 8500 ms captureMs. Keep the diagnostics
buffer in main.js fully sized for each requested capture duration.
In `@firmware/host/modules/audio/__tests__/audio-duplex-waveform/README.md`:
- Line 20: Replace the account-specific source command in the
audio-duplex-waveform README capture workflow with the repository-supported
setup prerequisite, using a portable repository-relative instruction that works
across developer environments.
In `@firmware/host/modules/conversation/chat-audioio/duplex-chat-audioio.js`:
- Around line 288-291: Update the sourcePreRollBytes calculation in the duplex
audio initialization flow to clamp the rounded value to at least one PCM sample
frame before constructing CircularByteHistory. Preserve the existing sample-rate
conversion and byte alignment, ensuring small preRollBytes values cannot produce
zero capacity and bypass the ChatAudioIOBase.FAILED path.
In `@firmware/host/modules/conversation/chat-audioio/manifest.json`:
- Around line 15-23: Update the manifest data roots for chat-audioio to cover
every supported wss:// Realtime endpoint: add the required OpenAI or other
trusted upstream root certificates, or document the expected roots and
configuration when operators select arbitrary server-backed endpoints. Keep the
existing CA entries and align the documentation with the accepted providerID
behavior.
In `@firmware/host/modules/conversation/chat.ts`:
- Around line 14-17: Extend the ChatConfig constructor-option tests to cover
fallback precedence: verify an explicit endpoint is forwarded instead of an
existing providerID, and verify providerID is used when endpoint is absent. Use
the existing injected-constructor test setup and preserve the current
endpoint-forwarding assertion.
In `@firmware/host/modules/testing/module-structure.architecture.ts`:
- Around line 373-374: Replace the source-text assertions in the emoticon test
with an instrumented or XS-driven rendering test that observes the numeric color
argument passed to drawTexture. Validate the packed RGBA value at the draw
boundary rather than requiring the specific bitwise expression or checking for
colorString absence.
In
`@firmware/host/modules/ui/components/face/__tests__/face-rendering/face-rendering.test.ts`:
- Line 479: Update the assertion message in the face-rendering test to refer to
eyelid mask revisions rather than eyelid outlines, while preserving the existing
blinkChanged assertion and its behavior.
In `@firmware/host/modules/ui/components/face/parts/dog/mouth.ts`:
- Around line 26-41: Update updateMask so the cubic-curve control-point depth is
derived from the current mouthHeight instead of the fixed 20 value, ensuring the
drawn mouth remains within the mask for smaller maxHeight values while
preserving the existing curve shape.
In `@firmware/host/modules/ui/components/face/parts/gray16-mask-raster.c`:
- Around line 287-295: Update the coverage handling around
stackchanGray16SetCoverage and stackchanGray16AddCoverage so that when topWhole
equals bottomWhole, the top and bottom edge contributions are summed on that
shared row rather than merged by the existing max-preserving operation. Preserve
the current behavior when the edges fall on different rows.
- Around line 22-29: Update the buffer-size guard in the mask validation block
to perform the stride-by-height multiplication in a sufficiently wide unsigned
type before shifting or comparing with byteLength, preventing signed overflow.
Ensure dimensions whose required buffer size cannot be represented or exceeds
mask->byteLength are rejected before stackchanGray16SetCoverage and
stackchanGray16AddCoverage can write.
In `@firmware/mods/examples/chat_audioio/mod.js`:
- Around line 21-24: Update DEFAULT_SMOKE_TIMEOUT_MS in the chat audio smoke
configuration to a value below the host runner’s 120000 ms default, ensuring the
device emits its timeout failure and executes cleanup before the host aborts.
- Around line 569-579: Ensure smoke runs always disconnect after recording a
PASS by updating the response-complete handling around smokeResultRecorded and
stopChat, rather than depending solely on autoStop. Preserve the delayed
AUTO_STOP_DELAY_MS timer behavior where applicable, and update the related smoke
configuration documentation if autoStop remains a required setting.
In `@firmware/scripts/capture-aec-waveforms.mjs`:
- Line 33: Validate STACKCHAN_AEC_CAPTURE_TIMEOUT_MS using the existing
positiveInteger pattern from run-chat-smoke.mjs before assigning timeoutMs.
Ensure non-numeric, zero, and negative values are rejected or replaced with the
default so setTimeout always receives a valid positive integer and the timeout
message never contains NaN.
In `@firmware/scripts/prepare-openai-realtime.mjs`:
- Around line 38-42: Update readOption() so bare options require a following
value: return no value when the next argument is missing or starts with "--",
while preserving the existing inline --name=value parsing. Ensure callers treat
the rejected result as a usage error rather than using another option as a
filename.
In `@firmware/scripts/run-chat-smoke.mjs`:
- Around line 25-29: Update the outputRoot initialization to resolve the default
artifact path against the existing firmwareDirectory, matching
capture-aec-waveforms.mjs, while preserving explicitly supplied output options
and environment values.
---
Nitpick comments:
In `@firmware/benchmarks/face-rendering/manifest.visual.json`:
- Around line 5-8: Review the module entries in the visual benchmark manifest
and either remove sbc_decoder and dvi_adpcm_decode if they are unused, or retain
them with a document-level explanation of the shared import/build dependency
requiring both modules. Keep the benchmark manifest limited to modules necessary
for this build target.
In `@firmware/host/modules/audio/__tests__/audio-duplex-acoustic/main.js`:
- Around line 111-119: Update the duplex test cleanup around the Timer callback
and stopSafely() so the amplifier is set to SAFE_HARDWARE_ATTENUATION_DB after
both streams stop, before diagnostics or close; also apply the same attenuation
on the failure path before throwing. Reuse the existing safe-attenuation
constant and preserve the current shutdown and analysis flow.
In `@firmware/host/modules/audio/__tests__/audio-duplex-waveform/main.js`:
- Around line 60-83: Extract the shared audio test helpers into a fixture module
based on the production AW88298 packing in audio-duplex.js. In
firmware/host/modules/audio/__tests__/audio-duplex-waveform/main.js lines 60-83,
remove the local probe generator and import buildProbe, amplifierVolumeByte, and
amplifierVolumeRegister, retaining only local PROBE_PEAK and attenuation
arguments. In
firmware/host/modules/audio/__tests__/audio-duplex-acoustic/main.js lines 21-50,
remove the local amplifierVolumeRegister and buildProbe implementations, import
the shared helpers, and add the helper module to that test’s manifest.
In `@firmware/host/modules/conversation/chat-audioio/openai-realtime-model.js`:
- Around line 12-18: Add an explanatory comment beside INPUT_ACK_BARRIER_INDEX
and INPUT_RING_HEAD_BARRIER_INDEX documenting that index 1 is an acknowledgement
sequence in direct transport but the ring head in shared-ring transport, and
that these layouts must not be mixed. Keep the existing shared-ring behavior
unchanged.
In `@firmware/host/modules/conversation/manifest.json`:
- Around line 13-34: Replace the duplicated platform entries in the platforms
configuration with a shared included manifest, using the manifest include
mechanism rather than comma-separated platform keys. Preserve the ChatAudioIO
and ChatAudioIOBase mappings for esp32/m5stack_cores3, esp32/m5stackchan_cores3,
and esp32/stackchan_rt, or retain separate valid entries if platform-specific
targeting is required.
In `@firmware/host/modules/testing/fakes/audio-in.ts`:
- Around line 36-47: Update the read method to preserve partial-read data: when
target is a Uint8Array and byteLength is smaller than source.byteLength, push
the unconsumed suffix back to the front of chunks before returning. Keep the
existing prefix copy and return behavior unchanged, and do not alter
numeric-target reads.
In `@firmware/host/modules/ui/components/effects/music-notes.ts`:
- Around line 85-109: Extract the shared progress, rise, and sprite-top
calculation from onDraw and invalidateSprite into a single private helper, then
use its returned position in both methods. Preserve the existing fade-duration
early returns and ensure invalidateSprite uses exactly the same sprite top and
dimensions as onDraw.
In
`@firmware/host/modules/ui/components/face/__tests__/face-rendering/face-rendering.test.ts`:
- Around line 132-145: Document in the test or API-facing definition of
fillOutsideAperture that it resets the mask before drawing, clearing prior
raster output such as fillCircle results. Add a concise note or explicitly
clear/use separate mask state before invoking fillOutsideAperture, without
relying on inferred reset behavior.
In `@firmware/host/modules/ui/components/face/parts/__tests__/unit-steps.test.ts`:
- Around line 6-11: Extend the `animation ratios quantize onto the bounded unit
scale` test to cover `quantizeUnit` returning 0 for `NaN` and negative inputs,
and `unitFromStep` clamping steps below 0 and above `UNIT_OPEN_STEPS` to the
corresponding unit bounds.
In `@firmware/host/modules/ui/components/face/parts/gray16-mask.ts`:
- Around line 32-39: Update the Bitmap constructor call in the gray16 mask
initialization to remove the trailing this.bytes.byteLength argument, retaining
width, height, format, buffer, and byteOffset in the documented five-argument
order.
In `@firmware/host/modules/ui/state/face-state.architecture.ts`:
- Around line 89-92: Narrow the allocation checks in the block-validation loop
around onFaceState so comments and strings do not trigger failures. Replace the
broad new and spread patterns with matchers for actual constructor expressions
and object-spread syntax, or strip line comments before matching; preserve
detection of allocations while allowing nested rest parameters.
In `@firmware/scripts/capture-aec-waveforms.mjs`:
- Around line 130-142: Remove the local decodeXsbugLog implementation and import
the shared decodeXsbugLog helper from firmware/scripts/lib/chat-smoke-log.mjs.
Update all existing call sites in capture-aec-waveforms.mjs to use the imported
function without changing decoding behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
There was a problem hiding this comment.
🧹 Nitpick comments (1)
firmware/host/modules/testing/module-structure.architecture.ts (1)
467-478: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winReplace the source-text assertion with a behavior test.
Lines 468 and 472-477 require one exact implementation expression. A valid refactor can change that expression without changing behavior. Test
processInputAudiowith a distinct producer buffer. Assert that the queued resampled output derives from that buffer.As per coding guidelines, tests must verify observable behavior or relational invariants and must not re-assert production source literals. As per path instructions, use behavior- or relational-invariant assertions instead of source-text checks.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/testing/module-structure.architecture.ts` around lines 467 - 478, Replace the source-text checks in the test for duplex ChatAudioIO resampling with a behavioral test that invokes processInputAudio using a producer buffer distinct from the microphone/source buffer. Assert that the queued resampled output is derived from that producer buffer, preserving the invariant that each input producer’s own samples are resampled and queued.Sources: Coding guidelines, Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@firmware/host/modules/testing/module-structure.architecture.ts`:
- Around line 467-478: Replace the source-text checks in the test for duplex
ChatAudioIO resampling with a behavioral test that invokes processInputAudio
using a producer buffer distinct from the microphone/source buffer. Assert that
the queued resampled output is derived from that producer buffer, preserving the
invariant that each input producer’s own samples are resampled and queued.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 23ab4277-c20c-46e3-b54a-3fd2e758419d
📒 Files selected for processing (6)
firmware/host/app/main.tsfirmware/host/modules/conversation/chat-audioio/duplex-chat-audioio.jsfirmware/host/modules/conversation/manifest.jsonfirmware/host/modules/testing/module-structure.architecture.tsfirmware/scripts/firmware.mjsfirmware/scripts/lib/firmware-command.test.mjs
🚧 Files skipped from review as they are similar to previous changes (3)
- firmware/host/app/main.ts
- firmware/host/modules/conversation/manifest.json
- firmware/host/modules/conversation/chat-audioio/duplex-chat-audioio.js
|
@codex review |
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
firmware/host/modules/conversation/chat-audioio/server-openai-realtime-model.js (1)
189-199: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winCreate a response after a function result.
sendFunctionResultsendsconversation.item.createforfunction_call_output, but the client session never receivesresponse.create. The Realtime API requiresresponse.createafter tool output to resume model inference.Proposed fix
sendFunctionResult(message) { this.sendJSON({ type: 'conversation.item.create', item: { type: 'function_call_output', call_id: message.call, output: JSON.stringify(message.result), }, event_id: this.generateId('event_'), }) + this.sendJSON({ type: 'response.create' }) }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/host/modules/conversation/chat-audioio/server-openai-realtime-model.js` around lines 189 - 199, Update sendFunctionResult to send a response.create event after creating the function_call_output item, so the client resumes model inference after receiving tool results. Preserve the existing conversation.item.create payload and event ID generation.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@firmware/benchmarks/face-rendering/manifest.json`:
- Line 5: Update the manifest instrumentation mapping to use the concrete
"instrumentation" module selector for "./instrumentation-control" instead of the
global "*" selector, unless the global registration is intentional and
explicitly documented.
In `@firmware/scripts/lib/idf-dependencies.mjs`:
- Around line 63-66: Update the managed dependency handling before the
lock-staleness check so an existing espressif/esp-sr entry with a constraint
differing from the configured constraint is replaced, not merely recognized by
name. Ensure the subsequent lock invalidation uses the updated constraint; add a
regression fixture covering an old manifest and matching lock, verifying the
manifest constraint changes and sdkconfig.h is removed.
---
Outside diff comments:
In
`@firmware/host/modules/conversation/chat-audioio/server-openai-realtime-model.js`:
- Around line 189-199: Update sendFunctionResult to send a response.create event
after creating the function_call_output item, so the client resumes model
inference after receiving tool results. Preserve the existing
conversation.item.create payload and event ID generation.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 48a0cc04-d3ed-41ff-9ce5-feca81fba949
📒 Files selected for processing (42)
firmware/benchmarks/face-rendering/manifest.jsonfirmware/benchmarks/face-rendering/manifest.visual.jsonfirmware/docs/chat-audioio-integration.mdfirmware/host/modules/audio/__tests__/audio-duplex-acoustic/main.jsfirmware/host/modules/audio/__tests__/audio-duplex-acoustic/manifest.jsonfirmware/host/modules/audio/__tests__/audio-duplex-device/manifest.jsonfirmware/host/modules/audio/__tests__/audio-duplex-waveform/README.mdfirmware/host/modules/audio/__tests__/audio-duplex-waveform/main.jsfirmware/host/modules/audio/__tests__/audio-duplex-waveform/manifest.doubletalk.jsonfirmware/host/modules/audio/__tests__/audio-duplex-waveform/manifest.jsonfirmware/host/modules/audio/__tests__/fixtures/audio-duplex-test-helpers.jsfirmware/host/modules/audio/__tests__/microphone.test.tsfirmware/host/modules/audio/platforms/m5stackchan-cores3/audio-duplex.cfirmware/host/modules/conversation/__tests__/chat-service/chat-service.test.tsfirmware/host/modules/conversation/__tests__/duplex-chat-audio-buffer.test.tsfirmware/host/modules/conversation/chat-audioio/duplex-chat-audio-buffer.tsfirmware/host/modules/conversation/chat-audioio/duplex-chat-audioio.jsfirmware/host/modules/conversation/chat-audioio/manifest.jsonfirmware/host/modules/conversation/chat-audioio/openai-realtime-model.jsfirmware/host/modules/conversation/chat-audioio/server-openai-realtime-model.jsfirmware/host/modules/conversation/chat-audioio/stackchan-openai-realtime.jsfirmware/host/modules/conversation/manifest.jsonfirmware/host/modules/testing/fakes/audio-in.tsfirmware/host/modules/testing/module-structure.architecture.tsfirmware/host/modules/ui/components/effects/__tests__/emoticon-pulse.test.tsfirmware/host/modules/ui/components/effects/emoticon-pulse.tsfirmware/host/modules/ui/components/effects/emoticon.tsfirmware/host/modules/ui/components/effects/music-notes.tsfirmware/host/modules/ui/components/face/__tests__/face-rendering/face-rendering.test.tsfirmware/host/modules/ui/components/face/parts/__tests__/unit-steps.test.tsfirmware/host/modules/ui/components/face/parts/dog/mouth.tsfirmware/host/modules/ui/components/face/parts/gray16-mask-raster.cfirmware/host/modules/ui/components/face/parts/gray16-mask.tsfirmware/host/modules/ui/state/face-state.architecture.tsfirmware/mods/examples/chat_audioio/mod.jsfirmware/scripts/capture-aec-waveforms.mjsfirmware/scripts/lib/idf-dependencies.mjsfirmware/scripts/lib/idf-dependencies.test.mjsfirmware/scripts/lib/openai-realtime-manifest.mjsfirmware/scripts/lib/openai-realtime-manifest.test.mjsfirmware/scripts/prepare-openai-realtime.mjsfirmware/scripts/run-chat-smoke.mjs
💤 Files with no reviewable changes (1)
- firmware/host/modules/testing/module-structure.architecture.ts
🚧 Files skipped from review as they are similar to previous changes (23)
- firmware/host/modules/conversation/chat-audioio/stackchan-openai-realtime.js
- firmware/host/modules/ui/components/face/parts/tests/unit-steps.test.ts
- firmware/host/modules/ui/state/face-state.architecture.ts
- firmware/host/modules/audio/tests/audio-duplex-waveform/README.md
- firmware/host/modules/ui/components/face/parts/gray16-mask.ts
- firmware/scripts/lib/openai-realtime-manifest.mjs
- firmware/host/modules/audio/tests/audio-duplex-acoustic/manifest.json
- firmware/host/modules/conversation/chat-audioio/manifest.json
- firmware/host/modules/ui/components/effects/music-notes.ts
- firmware/host/modules/ui/components/face/parts/gray16-mask-raster.c
- firmware/host/modules/ui/components/effects/emoticon.ts
- firmware/host/modules/conversation/manifest.json
- firmware/scripts/capture-aec-waveforms.mjs
- firmware/host/modules/ui/components/face/tests/face-rendering/face-rendering.test.ts
- firmware/scripts/lib/openai-realtime-manifest.test.mjs
- firmware/benchmarks/face-rendering/manifest.visual.json
- firmware/host/modules/conversation/chat-audioio/openai-realtime-model.js
- firmware/host/modules/ui/components/face/parts/dog/mouth.ts
- firmware/host/modules/testing/fakes/audio-in.ts
- firmware/scripts/run-chat-smoke.mjs
- firmware/host/modules/audio/platforms/m5stackchan-cores3/audio-duplex.c
- firmware/host/modules/audio/tests/audio-duplex-waveform/main.js
- firmware/mods/examples/chat_audioio/mod.js
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 79f5f72b65
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
@coderabbitai review |
✅ Action performedReview finished.
|
|
@codex review |
|
@coderabbitai review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
firmware/mods/examples/chat_audioio/mod.js (1)
531-547: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick winMake smoke completion idempotent and run teardown.
smokeResultRecordedis the terminal flag, but theFAILEDbranch and the no-promptSPEAKINGbranch do not check it. A laterFAILEDevent can logFAILafterPASSand clear the pending stop timer. A repeatedSPEAKINGevent can schedule another stop timer. TheFAILEDbranch also only clears timers; it does not runstopChat(true), so it skips microphone restoration and the reset ofsmokeActive. Route terminal outcomes through one guarded finalizer that records once, clears timers, and tears down the chat.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@firmware/mods/examples/chat_audioio/mod.js` around lines 531 - 547, Update the smoke terminal handling around smokeResultRecorded so both the FAILED branch and the no-prompt SPEAKING branch use one guarded finalizer. Ensure it records and logs only the first terminal outcome, clears existing smoke timers, and calls stopChat(true) for teardown, preventing later events from logging or scheduling additional stop timers.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@firmware/mods/examples/chat_audioio/__tests__/chat-audioio-config/chat-audioio-config.test.ts`:
- Around line 32-49: Extend the assertions in the withChatDefaults test cases to
verify type preservation: assert providerDefaults.type remains 'deepgramAgent'
and explicitWorker.type remains 'openAIRealtime', alongside the existing
specifier, voiceID, and instructions checks.
---
Outside diff comments:
In `@firmware/mods/examples/chat_audioio/mod.js`:
- Around line 531-547: Update the smoke terminal handling around
smokeResultRecorded so both the FAILED branch and the no-prompt SPEAKING branch
use one guarded finalizer. Ensure it records and logs only the first terminal
outcome, clears existing smoke timers, and calls stopChat(true) for teardown,
preventing later events from logging or scheduling additional stop timers.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 84b843a6-63ab-40dc-afd7-05274397b648
📒 Files selected for processing (14)
firmware/benchmarks/face-rendering/README.mdfirmware/benchmarks/face-rendering/manifest.jsonfirmware/host/modules/conversation/__tests__/realtime-worker-entrypoints/direct-model.tsfirmware/host/modules/conversation/__tests__/realtime-worker-entrypoints/manifest.test.jsonfirmware/host/modules/conversation/__tests__/realtime-worker-entrypoints/realtime-worker-entrypoints.test.tsfirmware/host/modules/conversation/__tests__/realtime-worker-entrypoints/server-model.tsfirmware/host/modules/conversation/chat-audioio/manifest.jsonfirmware/host/modules/conversation/chat-audioio/stackchan-openai-realtime.jsfirmware/host/modules/conversation/chat-audioio/stackchan-server-openai-realtime.jsfirmware/mods/examples/chat_audioio/__tests__/chat-audioio-config/chat-audioio-config.test.tsfirmware/mods/examples/chat_audioio/config.jsfirmware/mods/examples/chat_audioio/mod.jsfirmware/scripts/lib/idf-dependencies.mjsfirmware/scripts/lib/idf-dependencies.test.mjs
🚧 Files skipped from review as they are similar to previous changes (6)
- firmware/benchmarks/face-rendering/README.md
- firmware/benchmarks/face-rendering/manifest.json
- firmware/host/modules/conversation/chat-audioio/stackchan-openai-realtime.js
- firmware/host/modules/conversation/chat-audioio/manifest.json
- firmware/scripts/lib/idf-dependencies.test.mjs
- firmware/scripts/lib/idf-dependencies.mjs
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9f9b6b3ff6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@coderabbitai review |
|
@codex review |
✅ Action performedReview finished.
|
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
Summary
developchanges, including the startup volume preference fix and lazy USB dock support.What Changed
Release Impact
minor: new CoreS3 Realtime duplex voice and face-rendering capabilities.Verification
cd firmware && npm run formatcd firmware && npm run lintcd firmware && npm run test(415 tests)cd firmware && npm run check:legacy-namescd firmware && npm run check:architecture(75 tests)cd firmware && npm run check:manifest(6 targets)cd firmware && npm run build:m5stackchan_cores3/dev/ttyACM0: host/MOD flashing, persisted volume0.1, and five long TTS turns without rebootAffected Areas
Breaking Changes
Related Issues
Summary by CodeRabbit