[{reqId}] {exception_url} InvalidArgumentException: $aspect must use one of the XXX_USAGE constants, "A" given!A batch of 690 of these showed up in the last 15 minutes.
A spike of these errors occurred during wmf.18 group1 promotion today but strangely all instances of the error were from 1.45.0-wmf.17.
Mentioned in SAL (#wikimedia-operations) [2025-09-10T18:28:11Z] <dduvall> elevated error rate during wmf.18 group1 promotion. all were $aspect must use one of the XXX_USAGE constants error occurring from wmf.17 (cc T404238)
At a first glance…seems potentially related to https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/1176670? cc @seanleong-WMDE @JoelyRooke-WMDE
Yes, it seems like it is related to our patch. But this will only occur if we go from wmf.18 to wmf.17, but it will not happen when we are going from wmf.17 to wmf.18. This is because the latter version already knows about "A" aspects, but when the version is going backwards, it doesn't know about the "A" aspects.
We are not sure how this could happen, as we are currently going from wmf.17 to wmf.18. We also couldn't find the log in Logstash using the reqId in the description. Do you have any wikis that experience this error, or ideas on where we can see the logs?
All of the production Wikis (group0 and group1) that we've checked are working fine as of now.
Thanks @dancy ! All of the exception traces are on version wmf.17. But we included these changes for wmf.18. Can you think of any reason that some systems were on previous version during the deployment? #
All the pages I have checked seems fine now (for example this). Maybe it was something during the migration phase only. I suspect during the deployment there was a point that they went backwards or some parts were at wmf.17 (and this is when the error would occur) then they went to wmf.18 again.
I don't think this would any more cause a problem since all are in wmf.18 now. Do you think we need to take an action about it?
While the train is rolling forward, there is definitely a period of time where a given wiki is accessible via the wmf.17 and wmf.18 code. This means that the wmf.18 code can create database entries that might be read by the wmf.17 code.
I don't think this would any more cause a problem since all are in wmf.18 now. Do you think we need to take an action about it?
I think the only outcome of this ticket is to remember that both old and new code can be running for a wiki for a non-zero amount of time during train deployments, so code should be written with this possibility in mind. Related to this is the possibility of having to roll back the train for some reason or another. When new code adds database records that old code can't process, it creates problems/confusion during a rollback.
I'm assuming that this can probably be closed, given the comments above that suggest it will have just been a temporary error while switching versions? (Please correct me if I am wrong!)