@Clement_Goubert The current rdb* hosts are on Bullseye, and you listed Bookworm as the designated OS, if we move to a new OS, let's directly move to Trixie?

Also, I changed the names: ganeti1053/1053 were already added last year in https://phabricator.wikimedia.org/T401691.

@RobH Why did you create a racking task only for two servers, are these shipped out in batches and we only get two initially? The order is for four servers. I'll go ahead and create the site.pp/preseed config for all four of them already anyway.

Access has been granted to all who didn't have the access before

Sounds good. The maxbinderwmf account is now disabled, resolving the task

Ok! I've just enabled "wmf" for your "mbinder" account. Let me know if all works fine, then I'll go ahead and disable the "maxbinderwmf" account.

@Dzahn : I enabled Java 21 on build2002 (which is a Bookworm host) by merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/1245280 and it worked just fine:

@MBinder_WMF I did a little digging in account history: Your original mbinder account was created in 2020 in https://phabricator.wikimedia.org/T251349 to manage bulk jobs in Phabricator. Do you still use/need this?

In T418068#11654602, @Jelto wrote:

@MoritzMuehlenhoff or @SLyngshede-WMF this task sounds like an offboarding procedure.

You're using the wrong account: You shell access is for mbinder, but you requested access for the "wmf" group with "maxbinderwmf", that won't work as you need to use the same user for your for SSH and LDAP. Please login with your mbinder account and request the "wmf" permission for that user.

In T417655#11655864, @Ottomata wrote:

the former is now self-service via IDM.

I suppose that would be https://idm.wikimedia.org/. I don't totally see how one can edit their LDAP groups, but however it is done that is what needs to happen!

In T416582#11622819, @FCeratto-WMF wrote:

I installed orchestrator-client as well.

In T418448#11653297, @Jelto wrote:

@MoritzMuehlenhoff 18.9.1 finally contains openssl 3.4.4. I'll update the hosts to 18.8.5 first and then to 18.9.1.

In T418344#11653266, @Jelto wrote:

In T418344#11651279, @sbassett wrote:

@Jelto - Any objections to making this task public now? Do we not want to disclose the details within your last comment?

Its somewhat public what openssl versions are included in GitLab but given CVE-2025-15467 is a critical RCE vulnerability I'd like to keep that private until this is fixed upstream.

I also double-checked the openssl version in the newest release 18.9.1 and this version finally contains the fix:

docker run --rm --entrypoint /opt/gitlab/embedded/bin/openssl gitlab/gitlab-ce:18.9.1-ce.0 version
Unable to find image 'gitlab/gitlab-ce:18.9.1-ce.0' locally
18.9.1-ce.0: Pulling from gitlab/gitlab-ce
...
OpenSSL 3.4.4 27 Jan 2026 (Library: OpenSSL 3.4.4 27 Jan 2026)

I'll open a followup task, once this is resolved this one can be public too.
cc @MoritzMuehlenhoff ^

Right, I had totally missed that David, Francesco and Andrew were already part of the group.

@dcaro @fnegri @Raymond_Ndibe @Volans @Andrew @fgiunchedi Please log into https://idm.wikimedia.org, then select "Permissions" in the hamburger menu. In that menu, please click "Request this permission" for "Bitu-account-managers".

This will happen with the upcoming Trixie/MDB migration.

No longer relevant with Puppet 7

Starting with the Puppet 7 migration all Puppet 7 agents in production use SRV records. The CNAME mechanism is only used in Cloud VPS and Pontoon anymore.

This was mostly an issue with the automated restart (profile::auto_restarts::service) on older Debian releases. Since Bookworm these restarts are stable we only enable the automated restarts for Bookworm and later.

This happened with a very old slapd version and we haven't seen it since.

This is done

No longer relevant, relevant settings which are generally applicable have landed in the Debian kernel config over time or are not suitable for a general purpose kernel

This is no longer relevant with Mediawiki on wikikube

Access has been removed

In T376538#11635593, @hnowlan wrote:

That change has been superseded and master currently builds successfully

cergen is fully undeployed from our infrastructure: All certificates have been migrated to the PKI (or in some cases were obsoleted like some certs only relevant for baremetal Mediawiki), all old certificates were cleaned up (with the exception of two old Puppet CA certs which might still be needed to decrypt old backups) and cergen were removed from Puppet.

In T303744#11632353, @Clement_Goubert wrote:

The information about team -> contact is already in alertmanager, I don't see the point of adding it here.

Alternatively we could also rebuild linux-base for trixie-wikimedia and drop the rp-filter setting? It basically only changes for major releases, so we only need to revisit the settings every two years, but we keep the overall mechanism consistent.

In T417632#11627179, @JMeybohm wrote:

@ayounsi suggested we could remove linux-sysctl-defaults from our nodes and copy the useful setting from there to sysctl.pp. What do you think @MoritzMuehlenhoff (since you explicitly added the package in https://gerrit.wikimedia.org/r/c/operations/puppet/+/1141585)?

In T416582#11627663, @Marostegui wrote:

@FCeratto-WMF I am not sure if things are working properly. I hit "forget" on db2230 (test-s4 - which was broken anyway) and I reconnected it to db-test1003 (the current master) and it is not showing up in orchestrator (it should automatically show up after a few mins).
And I saw this on the logs:

2026-02-18T13:34:26.737755+00:00 dborch1002 orchestrator[2114906]: 2026-02-18 13:34:26 ERROR ReadTopologyInstance(db2230.codfw.wmnet:3306) show global status like 'Uptime': dial tcp 10.192.21.15:3306: i/o timeout
2026-02-18T13:34:26.737868+00:00 dborch1002 orchestrator[2114906]: ReadTopologyInstance(db2230.codfw.wmnet:3306) show global status like 'Uptime': dial tcp 10.192.21.15:3306: i/o timeout

Firewall rule somewhere for core hosts?:

All done

All done. We still keep two old Puppet CA certificates around which could potentially be needed to decrypt some old backups.

We're now on dnsmasq 2.92 across the fleet and I've upload the builds for bookworm and trixie to "main", i.e. they are now the default.

This seems to have been created in error

In T417465#11617433, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2026-02-14T01:12:01Z] <mutante> cumin1003 - race condition between puppet and systemd - puppet fails because userdel fails. userdel fails because there is still a pid used by the user. that process is /lib/systemdl/system --user - ran "loginctl terminate-user"; killed tmux PID; ran puppet again T417465

All resolved!