stackrox · dashrews78 · Aug 10, 2023 · Aug 10, 2023 · Aug 11, 2023 · Aug 11, 2023
diff --git a/Makefile b/Makefile
@@ -776,4 +776,4 @@ mitre:
 
 .PHONY: bootstrap_migration
 bootstrap_migration:
-	$(SILENT)if [[ "x${DESCRIPTION}" == "x" ]]; then echo "Please set a description for your migration in the DESCRIPTION environment variable"; else go run tools/generate-helpers/bootstrap-migration/main.go --root . --description "${DESCRIPTION}" ;fi
+	$(SILENT)if [[ "x${DESCRIPTION}" == "x" ]]; then echo "Please set a description for your migration in the DESCRIPTION environment variable"; else go run tools/generate-helpers/bootstrap-migration/main.go --root . --description "${DESCRIPTION}" --storeObject "${STORE_OBJECT}" ;fi
@@ -7,8 +7,11 @@ are in the `migrations` subdirectory.
 
 Migrations are organized with sequence numbers and executed in sequence. Each migration is provided
 with a pointer to `types.Databases` where the pointed object contains all necessary database instances,
-including `Bolt`, `RocksDB`, `Postgres` and `GormDB` (datamodel for postgres). Depending on the version
-a migration is operating on, some database instances may be nil.
+including `Bolt`, `RocksDB`, `Postgres` and `GormDB` (datamodel for postgres) as well as a context `DBCtx`.
+The context allows for the migrations to be wrapped in a transaction so they can be committed as they
+are processed.  (Migrations moving data with `GormDB` will not be part of the outer transaction, as such,
+care should be taken when using `GormDB` to move data.)  Depending on the version a migration is
+operating on, some database instances may be nil.
 
 A migration can read from any of the databases, make changes to the data or to the datamodel
 (database schema when working with postgres), then persist these changes to the database.
@@ -44,22 +47,27 @@ A migration can read from any of the databases, make changes to the data or to t
 Script should correspond to single change. Script should be part of the same release as this change.
 Here are the steps to write migration script:
 
-1. Run `DESCRIPTION="xxx" make bootstrap_migration` with a proper description of what the migration will do
+1. Run `DESCRIPTION="xxx" STORE_OBJECT="storage.Object0,storage.Object1" make bootstrap_migration` with a proper description of what the migration will do
    in the `DESCRIPTION` environment variable.
+and OPTIONALLY with the objects being migrated to create `gen.go` files for each object being migrated
+    in the optional `STORE_OBJECT` environment variable.
 
-2. Determine if this change breaks a previous releases database.  If so increment the `MinimumSupportedDBVersionSeqNum` 
-   to the `CurrentDBVersionSeqNum` of the release immediately following the release that cannot tolerate the change. 
-   For example, in 4.2 a column `column_v2` is added to replace the `column_v1` column in 4.1.  All the code from 4.2
+2. If using the optional `STORE_OBJECT` parameter, generate the migration stores with the following command:
+    `gogen -run pg-table-bindings`
+
+3. Determine if this change breaks a previous releases database.  If so increment the `MinimumSupportedDBVersionSeqNum` to
+   the `CurrentDBVersionSeqNum` of the release immediately following the release that cannot tolerate the change. For
+   example, in 4.2 a column `column_v2` is added to replace the `column_v1` column in 4.1.  All the code from 4.2
    onward will not reference `column_v1`.  At some point in the future a rollback to 4.1 will not longer be supported
    and we want to remove `column_v1`.  To do so, we will upgrade the schema to remove the column
    and update the `MinimumSupportedDBVersionSeqNum` to be the value of `CurrentDBVersionSeqNum` in 4.2
    as 4.1 will no longer be supported.  The migration process will inform the user of an error when trying to migrate
    to a software version that can no longer be supported by the database.
 
-3. Write the migration code and associated tests in the generated `migration_impl.go` and `migration_test.go` files.
+4. Write the migration code and associated tests in the generated `migration_impl.go` and `migration_test.go` files.
    The files contain a number of TODOs to help with the tasks to complete when writing the migration code itself.
 
-4. To better understand how to write the `migration.go` and `migration_test.go` files, look at existing examples
+5. To better understand how to write the `migration.go` and `migration_test.go` files, look at existing examples
 in `migrations` directory, or at the examples listed below.
 
     - [#1](https://github.com/stackrox/rox/pull/8609)
@@ -90,6 +98,9 @@ in `migrations` directory, or at the examples listed below.
 
 ## Writing postgres migration tests
 
+Follow the TODOs listed in `migration_test.go`.  This includes a recommended test to verify the pre-migration SQL statements provide
+the expected results against the post-migration database in order to verify backwards compatiblity.
+
 ### Migrator limitations
 
 Migrator upgrades the data from a previous datamodel to the current one. In the case of data manipulation migrations,
@@ -116,7 +127,7 @@ to generate current schema which can be find in each Postgres store.
 pg-schema-migration-helper --type=<prototype> --search-category ...
 ```
 
-This tool also generates conversion tools for schema, you may remove the 
+This tool also generates conversion tools for schema, you may remove them.
 
 #### Create or upgrade the schema of a table.
 
@@ -144,7 +155,7 @@ In migrator, there are a multiple ways to access data.
 
     Raw SQL commands are always available to databases and it has good isolation from current release. It is used frequently in
     migrations before Postgres. Migrations with raw SQL command needs less maintenance but it may not be convenient
-    and it could be error-prone.
+    and it could be error-prone.  This model supports the transaction passed via the databases.DBCtx.
     We try to provide more convenient way to read and update the data.
 
 2. Gorm
@@ -193,12 +204,15 @@ In migrator, there are a multiple ways to access data.
        Serialized []byte    `gorm:"column:serialized;type:bytea"`
    }
    ```
+   Gorm cannot participate in the transaction passed via the context.  The risk and possible ramifications of that
+   should be considered before deciding to use Gorm to move the data.
+
+3. Stores
 
-3. Duplicate the Postgres Store
-   This method is used in version 73 and 74 to migrate all tables from RocksDB to Postgres. In addition to frozen schema,
-   the store to access the data are also frozen for migration. The migrations with this method are closely associated
-   with current release eg. search/delete with schema and the prototypes of the objects. This method is NOT recommended for
-   4.0 and beyond.
+    Generate a version of the Postgres Store based off the migrator's Generic Store via the bootstrap process.
+    The migrations with this method are closely associated with current release eg. search/delete with schema
+    and the prototypes of the objects. The search schema and the prototypes of the objects must be backwards
+    compatible.  The advantage of this approach is it would be more familiar for the engineers.
 
 #### Conversion tool