Contact Rollups: remove legacy pre-2017 PD attendance extraction - #74614
Open
carl-codeorg wants to merge 1 commit into
Open
Contact Rollups: remove legacy pre-2017 PD attendance extraction#74614carl-codeorg wants to merge 1 commit into
carl-codeorg wants to merge 1 commit into
Conversation
Before Sept 2017, PD workshop attendance was modeled as sections with a non-null section_type plus followers rows; commit 7d3b6ab switched entirely to the pd_attendances model and nothing has written a workshop section since. The nightly contact rollups job still scanned users x followers x sections (with no index on section_type) every run to re-extract those frozen rows. Marketing has confirmed that contacts whose only PD attendance predates the 2017 model are no longer relevant, so this deletes the extraction and its processing rather than snapshotting it: - drop extract_professional_learning_attendance_old_attendance_model and rename the "new model" extraction to extract_professional_learning_attendance, since there is only one now - drop the dashboard.followers handling (and SECTION_TYPE_INVERTED_MAP) in extract_professional_learning_attended and extract_roles Expected Pardot effect: teachers whose only attendance came from the old model get one update shortening db_Professional_Learning_Attended on the next run. Because Pardot multi-value fields are synced as indexed keys (db_X_0, db_X_1, ...) and the delta calculation only sends keys present in new data, a removed trailing value is not cleared in Pardot; the remaining stale tail values are accepted as a known one-time artifact of dropping this data. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
carl-codeorg
marked this pull request as ready for review
August 13, 2026 18:10
sureshc
approved these changes
Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Deletes the nightly Contact Rollups extraction over the pre-2017 PD attendance model, plus its processing code.
Before Sept 2017, PD workshop attendance was modeled as
sectionsrows with a non-nullsection_typejoined throughfollowers. Commit 7d3b6ab ("Remove all workshop.section references, switch completely to the new attendance model", 2017-09-06) moved attendance entirely topd_attendances, and nothing has written a workshop section since — the matching rows have been frozen for eight years. The nightly job still re-scannedusers⨝followers⨝sections(no index onsection_type;followersis one of the largest tables in the database) every single run to re-extract them.Marketing confirmed that contacts whose only PD attendance predates the 2017 model are no longer relevant, so this is a delete, not a snapshot.
extract_professional_learning_attendance_old_attendance_modeland its call incollect_contacts.extract_professional_learning_attendance_new_attendance_model→extract_professional_learning_attendance, since there is only one model now.dashboard.followershandling inContactRollupsProcessed: thesection_type→ course mapping (SECTION_TYPE_INVERTED_MAP) inextract_professional_learning_attended, and thedashboard.followerskey check inextract_roles' Teacher rule.Expected Pardot effect (one-time)
db_Professional_Learning_Attendedon the next nightly run.db_X_0,db_X_1, …) and the delta calculation only sends keys present in new data, so when a list shrinks, trailing indexed values are not cleared in Pardot. Those stale tail values are a known, accepted one-time artifact of dropping this data.Teacherrole; any surviving role change syncs the same way.Performance
Removes one of the heaviest collect-phase queries (full scan of
sectionsfiltered on unindexedsection_type, joined tofollowersandusers) and the corresponding raw-row volume from every downstream step (aggregation, parsing, sync joins).Testing
contact_rollups_processed_test.rb:extract_professional_learning_attendedcases now exercisepd_attendancesonly (including dedup/compact/sort);extract_rolesno longer expects a followers-based Teacher role.contact_rollups_raw_test.rb(teacher-verification/...pngmissing from the local test asset pipeline) is a pre-existing local-environment issue, unchanged by this diff.🤖 Generated with Claude Code