Edit 3/13/2025: The text size in ggplot for geom_text is specified in mm, not points. To convert the unit to points, you can use the shortcut of font size/.pt, for example 10 pt font would be size = 10/.pt. Note that in theme arguments (using element_text) the size is default in points.
The answer is : The unit is the points. It is the unit of fontsize in the grid package. In ?unit, we find the following definition
"points" Points. There are 72.27 points per inch.
(but note the closely related "bigpts" Big Points. 72 bp = 1 in.)
Internally ggplot2 will multiply the font size by a magic number ggplot2:::.pt, defined as 1/0.352777778.
Here a demonstration, I create a letter using grid and ggplot2 with same size:
library(grid)
library(ggplot2)
ggplot(data=data.frame(x=1,y=1,label=c('A'))) +
geom_text(aes(x,y,label=label),size=100)
## I divide by the magic number to get the same size.
grid.text('A',gp=gpar(fontsize=100/0.352777778,col='red'))

Addendum Thanks to @baptiste
The "magic number"(defined in aaa-constants.r as .pt <- 1 / 0.352777778) is really just the conversion factor between "points" and "mm", that is 1/72 * 25.4 = 0.352777778. Unfortunately, grid makes the subtle distinction between "pts" and "bigpts", which explains why convertUnit(unit(1, "pt"), "mm", valueOnly=TRUE) gives the slightly different value of 0.3514598.
pdf(),postscript()etc., theheightandwidthare given in inches, so would it be wrong to assume thatggsavealso uses inches?ggplot2” because there is more than onesizeargument. Thesizeargument togeom_text()does something very different from thesizeargument toelement_text(), and I have no idea how to reconcile the two or why they would be different.