Skip to content

Commit fe954ea

Browse files
rscrsc
committed
cmd/go: add notary simulation and GONOVERIFY support
As an experiment to better understand the impact of having an authoritative source of truth for module hashes before the real notary is available, this CL adds the basic notary authorization checks using a partial whitelist of known go.sum values for popular modules. In addition to the temporary whitelist, this CL adds code implementing $GONOVERIFY, a new 'go help modules-auth', and clearer error messages for verification mismatches. See golang#25530 for notary proposal. Filed golang#30601 to remove whitelist when notary lands. Change-Id: Ibcb6ac39c5e60455edf003d8c20af6932aeb7e88 Reviewed-on: https://go-review.googlesource.com/c/go/+/165380 Reviewed-by: Bryan C. Mills <bcmills@google.com>
1 parent a6436a5 commit fe954ea

File tree

11 files changed

+4038
-32
lines changed

11 files changed

+4038
-32
lines changed

‎src/cmd/go/alldocs.go‎

Lines changed: 85 additions & 13 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎src/cmd/go/internal/modfetch/fetch.go‎

Lines changed: 157 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -400,27 +400,64 @@ func checkOneSum(mod module.Version, h string) {
400400
defer goSum.mu.Unlock()
401401
if initGoSum() {
402402
checkOneSumLocked(mod, h)
403+
} else if useNotary(mod) {
404+
checkNotarySum(mod, h)
403405
}
404406
}
405407

406408
func checkOneSumLocked(mod module.Version, h string) {
407409
goSum.checked[modSum{mod, h}] = true
408410

409-
for _, vh := range goSum.m[mod] {
410-
if h == vh {
411-
return
411+
checkGoSum := func() bool {
412+
for _, vh := range goSum.m[mod] {
413+
if h == vh {
414+
return true
415+
}
416+
if strings.HasPrefix(vh, "h1:") {
417+
base.Fatalf("verifying %s@%s: checksum mismatch\n\tdownloaded: %v\n\tgo.sum: %v"+goSumMismatch, mod.Path, mod.Version, h, vh)
418+
}
412419
}
413-
if strings.HasPrefix(vh, "h1:") {
414-
base.Fatalf("verifying %s@%s: checksum mismatch\n\tdownloaded: %v\n\tgo.sum: %v", mod.Path, mod.Version, h, vh)
420+
return false
421+
}
422+
423+
if checkGoSum() {
424+
return
425+
}
426+
427+
if useNotary(mod) {
428+
goSum.mu.Unlock()
429+
checkNotarySum(mod, h) // dies if h is wrong
430+
goSum.mu.Lock()
431+
432+
// Because we dropped the lock, a racing goroutine
433+
// may have already added this entry to go.sum.
434+
// Check again.
435+
if checkGoSum() {
436+
return
415437
}
416438
}
439+
417440
if len(goSum.m[mod]) > 0 {
418-
fmt.Fprintf(os.Stderr, "warning: verifying %s@%s: unknown hashes in go.sum: %v; adding %v", mod.Path, mod.Version, strings.Join(goSum.m[mod], ", "), h)
441+
fmt.Fprintf(os.Stderr, "warning: verifying %s@%s: unknown hashes in go.sum: %v; adding %v"+hashVersionMismatch, mod.Path, mod.Version, strings.Join(goSum.m[mod], ", "), h)
419442
}
420443
goSum.m[mod] = append(goSum.m[mod], h)
421444
goSum.dirty = true
422445
}
423446

447+
// checkNotarySum checks the mod, h pair against the Go notary.
448+
// It calls base.Fatalf if the hash is to be rejected.
449+
func checkNotarySum(mod module.Version, h string) {
450+
hashes := notaryHashes(mod)
451+
for _, vh := range hashes {
452+
if h == vh {
453+
return
454+
}
455+
if strings.HasPrefix(vh, "h1:") {
456+
base.Fatalf("verifying %s@%s: checksum mismatch\n\tdownloaded: %v\n\tnotary: %v"+notarySumMismatch, mod.Path, mod.Version, h, vh)
457+
}
458+
}
459+
}
460+
424461
// Sum returns the checksum for the downloaded copy of the given module,
425462
// if present in the download cache.
426463
func Sum(mod module.Version) string {
@@ -539,3 +576,117 @@ func TrimGoSum(keep map[module.Version]bool) {
539576
}
540577
}
541578
}
579+
580+
const goSumMismatch = `
581+
582+
SECURITY ERROR
583+
This download does NOT match an earlier download recorded in go.sum.
584+
The bits may have been replaced on the origin server, or an attacker may
585+
have intercepted the download attempt.
586+
587+
For more information, see 'go help module-auth'.
588+
`
589+
590+
const notarySumMismatch = `
591+
592+
SECURITY ERROR
593+
This download does NOT match the expected download known to the notary.
594+
The bits may have been replaced on the origin server, or an attacker may
595+
have intercepted the download attempt.
596+
597+
For more information, see 'go help module-auth'.
598+
`
599+
600+
const hashVersionMismatch = `
601+
602+
SECURITY WARNING
603+
This download is listed in go.sum, but using an unknown hash algorithm.
604+
The download cannot be verified.
605+
606+
For more information, see 'go help module-auth'.
607+
608+
`
609+
610+
var HelpSum = &base.Command{
611+
UsageLine: "module-auth",
612+
Short: "module authentication using go.sum",
613+
Long: `
614+
The go command tries to authenticate every downloaded module,
615+
checking that the bits downloaded for a specific module version today
616+
match bits downloaded yesterday. This ensures repeatable builds
617+
and detects introduction of unexpected changes, malicious or not.
618+
619+
In each module's root, alongside go.mod, the go command maintains
620+
a file named go.sum containing the cryptographic checksums of the
621+
module's dependencies.
622+
623+
The form of each line in go.sum is three fields:
624+
625+
<module> <version>[/go.mod] <hash>
626+
627+
Each known module version results in two lines in the go.sum file.
628+
The first line gives the hash of the module version's file tree.
629+
The second line appends "/go.mod" to the version and gives the hash
630+
of only the module version's (possibly synthesized) go.mod file.
631+
The go.mod-only hash allows downloading and authenticating a
632+
module version's go.mod file, which is needed to compute the
633+
dependency graph, without also downloading all the module's source code.
634+
635+
The hash begins with an algorithm prefix of the form "h<N>:".
636+
The only defined algorithm prefix is "h1:", which uses SHA-256.
637+
638+
Module authentication failures
639+
640+
The go command maintains a cache of downloaded packages and computes
641+
and records the cryptographic checksum of each package at download time.
642+
In normal operation, the go command checks the main module's go.sum file
643+
against these precomputed checksums instead of recomputing them on
644+
each command invocation. The 'go mod verify' command checks that
645+
the cached copies of module downloads still match both their recorded
646+
checksums and the entries in go.sum.
647+
648+
In day-to-day development, the checksum of a given module version
649+
should never change. Each time a dependency is used by a given main
650+
module, the go command checks its local cached copy, freshly
651+
downloaded or not, against the main module's go.sum. If the checksums
652+
don't match, the go command reports the mismatch as a security error
653+
and refuses to run the build. When this happens, proceed with caution:
654+
code changing unexpectedly means today's build will not match
655+
yesterday's, and the unexpected change may not be beneficial.
656+
657+
If the go command reports a mismatch in go.sum, the downloaded code
658+
for the reported module version does not match the one used in a
659+
previous build of the main module. It is important at that point
660+
to find out what the right checksum should be, to decide whether
661+
go.sum is wrong or the downloaded code is wrong. Usually go.sum is right:
662+
you want to use the same code you used yesterday.
663+
664+
If a downloaded module is not yet included in go.sum and it is a publicly
665+
available module, the go command consults the Go notary server to fetch
666+
the expected go.sum lines. If the downloaded code does not match those
667+
lines, the go command reports the mismatch and exits. Note that the
668+
notary is not consulted for module versions already listed in go.sum.
669+
670+
The GONOVERIFY environment variable is a comma-separated list of
671+
patterns (in the syntax of Go's path.Match) of module path prefixes
672+
that should not be verified using the notary. For example,
673+
674+
GONOVERIFY=*.corp.example.com,rsc.io/private
675+
676+
disables notary verification for modules with path prefixes matching
677+
either pattern, including "git.corp.example.com/xyzzy", "rsc.io/private",
678+
and "rsc.io/private/quux".
679+
680+
As a special case, if GONOVERIFY is set to "off", or if "go get" was invoked
681+
with the -insecure flag, the notary is never consulted, but note that this
682+
defeats the security provided by the notary. A better course of action is
683+
to set a narrower GONOVERIFY and, in the case of go.sum mismatches,
684+
investigate why the code downloaded code differs from what was
685+
downloaded yesterday.
686+
687+
NOTE: Early in the Go 1.13 dev cycle, the notary is being simulated by
688+
a whitelist of known hashes for popular Go modules, to expose any
689+
problems arising from knowing the expected hashes.
690+
TODO(rsc): This note should be removed once the real notary is used instead. See #30601.
691+
`,
692+
}

0 commit comments

Comments
 (0)