Sounds good to me @Jdrewniak ! Thanks :)

For the record, I needed something similar but different for DNS Discovery records: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1198583

In T404826#11304584, @bd808 wrote:

For browser detection, we've created a private gitlab repository to host some code we don't want to be publicly available, to avoid offering an easy way to figure out evasion techniques to crawlers that try to mimic a browser.

This is a concern that should likely be discussed elsewhere, but to make it known I will state here that this concept being implemented feels like it has implications on https://wikitech.wikimedia.org/wiki/Edge_uniques/Canary and the promise of transparency in the CDN edge implementation being a community control against misuse of the edge uniques tokens.

tcp-proxy sounds good to me as a name.

In T407826#11293440, @Clement_Goubert wrote:

Fixed in debug, now to redeploy every service using the mesh...

Are there any early estimates of the expected %age increase in something like logged-in daily active users?

In T304373#11270367, @brouberol wrote:

@CDanis Sorry, I was on PTO. I can see from SAL that you deployed? How did it go?

BTW, after looking at a few weeks of data, I suggest increasing the failure sampling fraction for these services. 10% or 20% would be absolutely fine, given their usage, and would give much more signal.

In T221976#11248799, @Vgutierrez wrote:

The usual approach for HAProxy is to generate a UUID, append it to the request, and echo it back to the user, for example:

http-request set-var(txn.req_id) uuid()
http-request set-header X-Request-ID %[var(txn.req_id)]
http-response set-header X-Request-ID %[var(txn.req_id)]

@Prototyperspective Can you please post the output of the traceroute step in these instructions?

joal please deploy refinery patch at your leisure, then I'll deploy a Turnilo patch and we can call this done 🎉

Dashboards look good to me, let's call this finalized!

In T406212#11238224, @RLazarus wrote:

What's the best query to check the cluster is wiped? One option is just a command-line flag like --i_solemnly_swear_it_is_okay_to_obliterate_the_state_of_this_kubernetes_cluster. (It has the advantage that if charlie is interrupted mid-run it can be restarted in this mode even after recreating some objects. But that can be accomplished with a confirmation prompt too.) A kube api call or etcd read would be ideal -- I had a look at the wipe-cluster cookbook but nothing immediately jumped out at me. (CC @JMeybohm on this question too.)

@brouberol Let's deploy together Monday, my morning/your afternoon?

Will deploy Puppet patches on Monday.

OK, sounds good to me! I'll merge and deploy my patch now then, and can help you with the other one later.

@brouberol if you want, I'm happy to add this onto my existing patch against the same file: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1192572

In T401705#11221765, @cjming wrote:

The main takeaway from Chris that I got was that we should be looking at Logstash or the data lake for NEL data rather than Prometheus which is not advised for high cardinality like NEL data.

See also T402994, T391411

I stumbled across this post which has a few interesting takes, like using mdadm metadata v1.0 to keep the ESP partition bootable but still active in mdadm, and also, forcing a manual resync every boot in case UEFI writes to its active ESP pre-boot (since it doesn't know anything about mdadm, of course). I'm not 100% convinced but it doesn't sound too unreasonable?

In T405360#11216876, @amastilovic wrote:

Other than copying to multiple targets using one command/operation, am I wrong in assuming that rclone would indeed be the Swiss Army knife in this scenario? Fast, talks to all the different systems we need it to talk to, doesn't need a Java installation, would be super easy to implement using a Docker image.

approved, and merged

In T404742#11194509, @akosiaris wrote:

I still have the same concerns as voiced in T359067#9602091, but I also have to be pragmatic. I don't see us solving the bigger registry problems in the next 6 months

Furthermore, Dragonfly helps mitigate some of the concerns. Some more operational ones I have are:

• Experience has shown that VMs with a lot of memory are more difficult to migrate around in Ganeti, some to the point of stalling. However, I am thinking it shouldn't be any stalling (we 've seen that with VMs with high continuous memory churn and this use case isn't one of these), just taking longer.

In T404742#11194509, @akosiaris wrote:

I still have the same concerns as voiced in T359067#9602091, but I also have to be pragmatic. I don't see us solving the bigger registry problems in the next 6 months

Furthermore, Dragonfly helps mitigate some of the concerns. Some more operational ones I have are:

• Experience has shown that VMs with a lot of memory are more difficult to migrate around in Ganeti, some to the point of stalling. However, I am thinking it shouldn't be any stalling (we 've seen that with VMs with high continuous memory churn and this use case isn't one of these), just taking longer.

FWIW: I think the new considerations in the task description make it pretty reasonable to take ~1.1% of the total RAM already allocated to Ganeti globally and spend it on VMs that are a known chokepoint in our deployments and the administration and development of the site in general.

Luca, do you want an early test subject for the Sloth trial?

Thanks @taavi, good points as always.

Thank you @JAllemandou !

Oh, one other thing we might want to do, if mesh.tracing.service_name is unset, default it to .Release.Namespace.

FTR: This is data that might potentially be useful for SDS1.3.

Looks like we might not actually need this? https://phabricator.wikimedia.org/T325131#8480824

In T402959#11148003, @Kappakayala wrote:

Based on my conversation with Amy Tsay, Wikidata Query service team is looking into it. Hence, removing SRE related tags to avoid confusion.

Sorry, I must have missed it -- did we explicitly reject the idea of using a simple MW cron job to get data into Prometheus?

In T393950#11140677, @bvibber wrote:

If I poke around in there I do see a "mediawiki_Chart_added_to_pages_total" figure that I can graph but it's giving very low total values and I'm not sure it's correct or not.

Hi @Lydia_Pintscher , SRE can make some exception here. It seems warranted given the status quo in the broader sparql ecosystem.

Thank you, that helps a lot.

In T400119#11132534, @Rohitverma9625 wrote:

Hello everyone, I'm a contributor to the Wikimedia Commons Android App. The Commons app recently started facing some API issues with code 403 and message "Please set a user-agent and respect our robot policy https://w.wiki/4wJS. See also T400119".

Open questions / next steps:

• Is configuring the Puppet server with HDFS access acceptable?

I'm not sure if there's an easy way to re-use the logic already in Refine given the current setup of webrequest_sampled_live. If there is, that would be ideal from a DRY & business logic drift perspective. Can someone advise?

As one option for a workaround, running tunnelencabulator --tunnel-everything after applying the above patchset will allow you to bypass this temporarily.

Is that something we could hack up a Prometheus exporter for real quick? What's the method of fetching the sizes?

@JAllemandou now that we've removed client_port, how is the Druid storage usage of webrequest looking?

You got me curious, so I spent a little time digging on this -- as far as I can tell not much has changed in the real world since Daniel Stenberg's blog post.

@JAllemandou uploaded some patches, please take a look :)

Any updates on this work, or any guesses on when there might be time?

In T400360#11044508, @Ottomata wrote:

I'll update the task description, and maybe set a scheduled reminder message in slack? :)