The smaller size was on purpose

This class exists (or more precisely, is added) in the feature's first commit (https://github.com/sindresorhus/refined-github/pull/1049/files#diff-041e2c7f28122df0efbf5dbb6ee3c2adR22), in which I don't see any reason except matching GitHub's style.

#3221 by @FloEdelmann, in the very first post

Missed that one. But that PR is made when design update is in progress, do you still think it looks fine with different button size?

do you still think it looks fine with different button size?

I do; you can see how the buttons also become wider and the space on that line might be limited already

Adjust the size with the following style:

.rgh-latest-tag-button,
.rgh-latest-tag-button + .btn,
.js-permalink-shortcut + div .btn-sm[data-pjax] {
    padding: 4px 16px 5px;
    font-size: 14px;
}