Triaging to UBN! as this is blocking activity for at least two hypotheses.

Anyone wanting to know what that tomcat is running can just use github.

In T404826#11304584, @bd808 wrote:

The generated VCL code for wmfuniq-based rate-limiting looks like this:

For instance, I see some of the IP ranges from GCP are part of a global on-wiki block, so I don't think you'll be able to create edits from that range.

Hi, I'm not sure this task is tagged with the right tags.

For python3-conftool and dervied packages, you can simply patch the .gitlab-ci.yml file to generate debs also for trixie, and then get them from apt-staging; assuming apt-staging is setup for trixie.

I think you have misunderstood what this task was about: it's about specifically removing that ACL from varnish, not about removing the concept, which is central to how we're doing traffic filtering.

In T389932#10701560, @jhathaway wrote:

In T389932#10697436, @Joe wrote:

In T389932#10694961, @jhathaway wrote:

One issue with using just the FQDN is that is breaks tools which rely on matching other hostnames, for instance the dcl tool reuses the site.pp defs for matching hosts of the form sretest1002.eqiad.default.svc.k8s.lan. I'm not aware of other tooling that is dependent on the regexes. On the other hand, if we moved the logic into an ENC script we could add some facility to return the correct role for dcl's hostnames as well.

Maybe I'm missing something, but one of the reasons I wanted to go with that format was also to make it easier for the puppet-compiler and dcl to parse which hosts belong to a node, without parsing regexes in site.pp. There's many ways to get a match without the TLD, starting from .startswith() in python.

Sorry I didn't realize that was on of the aims. I was focused on the regex complexity piece, which I thought was the primary concern of this task. In the case of dcl, the tool stands up a puppetserver, so the parsing of site.pp is done via puppet, the same as in production. For the puppet-compiler, I wasn't aware there were issues with the current python parsing breaking?

In T389932#10701585, @jhathaway wrote:

In proposing possible solutions, I would love to understand a bit more why our site.pp uses complex regexes. From looking through the git log it appears that the main use case is moving servers in and and out of their insetup role. What are the other reasons people use complex regexes?

Marking as resolved as this task is obsolete at this point IMO.

In T404291#11201131, @akosiaris wrote:

I 've already replied in T394982#11201112, but I find it improbable that SRE will be implementing such a behavior to accommodate for the change in node.js fetch() API. The HTTP Host header is pretty important across the infrastructure. Rewriting other HTTP headers to it might make debugging and reasoning more difficult than needed.

In T402959#11132802, @CDanis wrote:

Hi @Lydia_Pintscher , SRE can make some exception here. It seems warranted given the status quo in the broader sparql ecosystem.

But I want us to get the scope of the exception right:
Is federation traffic always against the /sparql endpoint?
Does federation traffic always set an Accept header like Accept: application/sparql-results[...] ?

Thanks!

Coming to @SLyngshede-WMF's concern, I think some of them are valid, like having disjoint configuration going besides the actual content of a file, including the puppet breakage.

I will tentatively close this task for now.

In T400119#11115049, @TheDJ wrote:

Yeah getting the swagger spec via curl https://api.wikimedia.org/core/v1/wikipedia/en/search/page?q=earth&limit=10 also no longer works I guess.

In T402546#11109851, @mickeybarber wrote:

Thx for reporting, i will try to check and fix it today.

Sorry, for reasons I don't understand my first request for that page got a x-cache-status: pass, I see now it's cacheable at the edge.

A couple of things we might want to consider:
* Is there any reason not to cache for some time the sitemap api URLs at the edge? Even a few hours would be beneficial I think if we ever publicize this more.

• Is there a reason to include User_talk: pages and similar in the sitemap? these pages are rarely cached and relatively expensive to render in some cases, do we care about them being indexed in search engines?

To clarify:

• If your error is coming from our filtering at the edge, you will get an error page containing "Error" in the <h1> tags
• Service Unavailable only happens when the negative response is coming from the backend.

In T402142#11093380, @XXBlackburnXx wrote:

In T402142#11092766, @Joe wrote:

I should also add - if this has happened in the last week, that might be connected to T400119

@Joe This seems to be related to {T400697} (private task). I don't believe the work on T400119 is connected to this issue, just an FYI.

In T402142#11092799, @Josve05a wrote:

I just noticed a pattern and wanted to surface it. For context: in the VRT (the volunteer helpdesk) we normally only see one such “Wikipedia inaccessible” email every couple of months, and almost always when there’s a confirmed outage. Over the past 1–2 weeks, however, we’ve received several separate reports, which is unusual enough that I thought it worth bringing up. Especially given the Reddit thread linked above.

I can’t share ISP/IP details directly here because of the VRT NDA, but I did try to collect as much information as possible from the reports (OS, browser versions, screenshots, etc.). In two of the tickets, users provided their IP and in one of them the entire user-agent (quoted above). What I can say without breaching confidentiality is that one was from an ISP in India and one from an ISP in the US. Those two seem to be using Chromium 126 related browsers. There were no other error message.

The one who solved their own issue, I can't confirm is related or not, since it is now fixed.

I should also add - if this has happened in the last week, that might be connected to T400119 - if users have any browser extension that modifies their user-agent, for example removing it, that would cause failures now.

This task seems to coalesce different issues; having said that, we've had quite a few abusers lately that used old versions of Chrome as their user agent, so we had to block traffic at times. Most of the rules should be disabled soon, and we are trying to improve our detection of actual browsers vs forged ones.

In T400119#11086977, @bd808 wrote:

In T400119#11084530, @Samwilson wrote:

Will GitLab CI be excluded from this policy?

I know you added the ignore edit to this, but as this thread is at least partly documentation at this point folks should be aware that the default gitlab-ci runners run in Digital Ocean's public cloud and are likely to be subject to public cloud restrictions. Adding tags: [wmcs] to your job specification will pin your jobs to a different pool of runners which are hosted in a Cloud VPS project.

In T400119#11076470, @Midleading wrote:

Please be more clear about the UA policy enforced here. I am always setting the Api-User-Agent header in my code with my Wikipedia username inside, but my bot still stopped working. My code always set the Api-User-Agent header rather than User-Agent header, because the same code may run in a browser or outside a browser. The most probable reason my code stopped working is that the Api-User-Agent header is null and void, just because User-Agent header is missing, right?

In T400119#11076853, @TheDJ wrote:

@Midleading you are always supposed to have a user-agent. Api-user-agent is just for situations where you are unable to MODIFY that agent to provide additional identifying information for your tool (browsers). It is not an alternative to having no user-agent, only an alternative for a non-modifiable user-agent (and is described as such in the policy in my opinion, though suggestions for improvement are always welcome on the talk page I guess).

We have 5M successful responses in the same period of time.

Sorry I hadn't realized the number of requests was this large, it reframes the problem a bit:

You missed changing the use of the abuse ACLs and removing the loading of netmaps for X-Public-Cloud and similar stuff as well.

In T400881#11057528, @Tgr wrote:

MediaWiki-Platform-Team will pick up the core part of this. Note that the soonest a change to the InstantCommons code could make a difference is after the next MediaWiki release (so in about 3 months). Many sites will only upgrade when the next LTS version is released (in about 15 months).

Maybe we can recommend some code that you can drop into your wiki configuration today to affect user agents.

I actually resolved the task by mistake: there's still some work to do on the varnish/haproxy side.

I fear this is a well known we've already encountered. You can see here https://grafana.wikimedia.org/d/zsdYRV7Vk/istio-sidecar?orgId=1&from=now-7d&to=now&timezone=utc&var-cluster=aWotKxQMz&var-namespace=revscoring-editquality-damaging&var-backend=$__all&var-response_code=503&var-quantile=0.5&var-quantile=0.95&var-quantile=0.99&viewPanel=panel-16 that most of the 503 have failure code UC which is envoy/istio for Upstream connection failed. In our experience, this can happen when a limit is not configured for the life of an upstream http connection.

I've taken a look from the side of the api.log mediawiki generates, and I was a bit surprised to find 16 identical calls over the span of an hour for the same revid that I found failing in logstash.

I thought about this a bit, and I think we need to distinguish between the grading system (which should express trust levels) and request feature (like bearing a valid session cookie, being identified as a bot...).

In T400881#11052795, @A_smart_kitten wrote:

In T400881#11052791, @Joe wrote:

In T400881#11050371, @Bawolff wrote:

Are you suggesting including the wiki user who caused the request to happen (as opposed to the server admin)? That feels like a privacy violation and i fail to see how it would be helpful in abuse fighting. Keep in mind the triggering event could just be a page view (whenever the page falls out of the appropriate cache) and may be an anonoymous user.

I was looking at this from the prespective of wiki operators. If we have the username in the UA, that allows us to rate-limit requests based on the UA string and only block abusive behaviour of single users, instead of need to punish an entire wiki. I'm not sure how that's a privacy violation, unless you assume that the username used on wiki X is somehow private identifying information, which I'm not convinced of.

I think the personal information might be the combined information that user X was viewing article Y/Commons images Y, rather than necessarily a username on its own (although, to be fair, possibly also just a username on its own). IANAL, but I believe that either of these may be covered by the GDPR's broad definition of "personal data", and would potentially cause data-protection headaches for wiki operators were we to implement a system that sent any sort of user data back to Wikimedia servers. On the face of the issue, I think I agree with @Bawolff here.

To give a bit of context, over the last day we saw:

• 62 million valid requests with no user-agent
• 24.5 million valid requests with user agent okhttp/*
• 17 million valid requests with user-agent python-{requests,urllib}*

In T400119#11052931, @TheDJ wrote:

In T400119#11052789, @Joe wrote:

Case in point, I can't find any request with that UA in the logs for the past few days. Indeed it's not in the list of "medium to large offenders" we're interested in blocking. Even if you are right and it can happen, it's not remotely on our radar as something to block.

It would be passed by Api-User-Agent, as User-Agent in the browser cannot be overridden by Javascript, but Api-User-Agent is a policy compliant fallback header to be set for that. Not sure if those are collated in the data you are looking at.

https://github.com/wikimedia/mediawiki/blob/master/resources/src/mediawiki.api/index.js
line 76 and 301

Please note, this solution is temporary: bots working from clouds will break repeatedly if they're not properly identified with us.

In T400881#11052701, @Bawolff wrote:

[Anyways, I adjusted the QuickInstantCommons part. That's the only part of this bug i plan to work on, so it should stay open for the stuff in MW core]

In T400881#11050371, @Bawolff wrote:

Are you suggesting including the wiki user who caused the request to happen (as opposed to the server admin)? That feels like a privacy violation and i fail to see how it would be helpful in abuse fighting. Keep in mind the triggering event could just be a page view (whenever the page falls out of the appropriate cache) and may be an anonoymous user.

In T400119#11051059, @Alien333 wrote:

Where does UAs like MediaWiki-JS/1.45.0-wmf.12, the defaults used by a plain new mw.Api() in an on-wiki script, stand with this?

(If they're going to be blocked, there are (from quick insource searches) at least a few thousand scripts that need fixing.)

In T400119#11047688, @DavidBrooks wrote:

@Joe I wasn't addressing AWB used as a bot, but as an interactive Windows app. Still, the rest of your comment seems applicable. The contact information would be the user's Wikipedia name (not the AWB authors'). We could prepend :p:ll:User: (p = project, ll = language code) if that would be better. I may need to take advice on bots' ids because I've never used that feature.

In T400119#11047556, @DavidBrooks wrote:

AutoWikiBrowser uses the MediaWiki API and User-Agent is WikiFunctions/n.n.n.n (Microsoft Windows NT n.n.n.n; .NET CLR 4.0.n.n). I don't know if that is distinctive enough. I guess a quick fix could be to add the logged-in user, although I'm not completely sure if any API request happens with nobody logged in yet.

Is AWB OK for now, or do we need that fix by Sep 1?

In T400119#11047176, @AntiCompositeNumber wrote:

external mw-related: requests with user-agent strings set by MediaWiki (like ForeignApiRepo) or by other mw-related software like WDQS Updater

Does this include things like ForeignFileRepos/InstantCommons?

In T400119#11047078, @RoySmith wrote:

Would it be possible to include a link to this phab ticket and/or the policy page in the HTTP error response?