mtdev2 · pull · Dec 12, 2019 · Mar 13, 2020 · Sep 25, 2020 · Sep 25, 2020
diff --git a/source/conf.py b/source/conf.py
@@ -196,8 +196,8 @@
 
 # Custom sidebar templates, filenames relative to this file.
 html_sidebars = {
-    '**': ['localtoc.html', 'relations.html'],
-    'index': ['localtoc.html']
+    '**': ['globaltoc.html', 'relations.html'],
+    'index': ['globaltoc.html']
 }
 
 # Additional templates that should be rendered to pages, maps page names to

diff --git a/source/discussions/pip-vs-easy-install.rst b/source/discussions/pip-vs-easy-install.rst
@@ -6,12 +6,12 @@ pip vs easy_install
 ===================
 
 
-`easy_install` was released in 2004, as part of :ref:`setuptools`.  It was
+:ref:`easy_install <easy_install>` was released in 2004, as part of :ref:`setuptools`.  It was
 notable at the time for installing :term:`packages <Distribution Package>` from
 :term:`PyPI <Python Package Index (PyPI)>` using requirement specifiers, and
 automatically installing dependencies.
 
-:ref:`pip` came later in 2008, as alternative to `easy_install`, although still
+:ref:`pip` came later in 2008, as alternative to :ref:`easy_install <easy_install>`, although still
 largely built on top of :ref:`setuptools` components.  It was notable at the
 time for *not* installing packages as :term:`Eggs <Egg>` or from :term:`Eggs <Egg>` (but
 rather simply as 'flat' packages from :term:`sdists <Source Distribution (or

diff --git a/source/glossary.rst b/source/glossary.rst
@@ -44,7 +44,7 @@ Glossary
         A :term:`Built Distribution` format introduced by :ref:`setuptools`,
         which is being replaced by :term:`Wheel`.  For details, see `The
         Internal Structure of Python Eggs
-        <https://setuptools.readthedocs.io/en/latest/formats.html>`_ and
+        <https://setuptools.readthedocs.io/en/latest/deprecated/python_eggs.html>`_ and
         `Python Eggs <http://peak.telecommunity.com/DevCenter/PythonEggs>`_
 
     Extension Module

diff --git a/source/guides/analyzing-pypi-package-downloads.rst b/source/guides/analyzing-pypi-package-downloads.rst
@@ -68,35 +68,35 @@ the `BigQuery quickstart guide
 Data schema
 -----------
 
-Linehaul writes an entry in a ``the-psf.pypi.downloadsYYYYMMDD`` table for each
+Linehaul writes an entry in a ``the-psf.pypi.file_downloads`` table for each
 download. The table contains information about what file was downloaded and how
 it was downloaded. Some useful columns from the `table schema
-<https://console.cloud.google.com/bigquery?pli=1&p=the-psf&d=pypi&t=downloads&page=table>`__
+<https://console.cloud.google.com/bigquery?pli=1&p=the-psf&d=pypi&t=file_downloads&page=table>`__
 include:
 
-+------------------------+-----------------+-----------------------+
-| Column                 | Description     | Examples              |
-+========================+=================+=======================+
-| file.project           | Project name    | ``pipenv``, ``nose``  |
-+------------------------+-----------------+-----------------------+
-| file.version           | Package version | ``0.1.6``, ``1.4.2``  |
-+------------------------+-----------------+-----------------------+
-| details.installer.name | Installer       | pip, `bandersnatch`_  |
-+------------------------+-----------------+-----------------------+
-| details.python         | Python version  | ``2.7.12``, ``3.6.4`` |
-+------------------------+-----------------+-----------------------+
++------------------------+-----------------+-----------------------------+
+| Column                 | Description     | Examples                    |
++========================+=================+=============================+
+| timestamp              | Date and time   | ``2020-03-09 00:33:03 UTC`` |
++------------------------+-----------------+-----------------------------+
+| file.project           | Project name    | ``pipenv``, ``nose``        |
++------------------------+-----------------+-----------------------------+
+| file.version           | Package version | ``0.1.6``, ``1.4.2``        |
++------------------------+-----------------+-----------------------------+
+| details.installer.name | Installer       | pip, `bandersnatch`_        |
++------------------------+-----------------+-----------------------------+
+| details.python         | Python version  | ``2.7.12``, ``3.6.4``       |
++------------------------+-----------------+-----------------------------+
 
 
 Useful queries
 --------------
 
 Run queries in the `BigQuery web UI`_ by clicking the "Compose query" button.
 
-Note that the rows are stored in separate tables for each day, which helps
+Note that the rows are stored in a partitioned, which helps
 limit the cost of queries. These example queries analyze downloads from
-recent history by using `wildcard tables
-<https://cloud.google.com/bigquery/docs/querying-wildcard-tables>`__ to
-select all tables and then filter by ``_TABLE_SUFFIX``.
+recent history by filtering on the ``timestamp`` column.
 
 Counting package downloads
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -108,18 +108,17 @@ The following query counts the total number of downloads for the project
 
     #standardSQL
     SELECT COUNT(*) AS num_downloads
-    FROM `the-psf.pypi.downloads*`
+    FROM `the-psf.pypi.file_downloads`
     WHERE file.project = 'pytest'
       -- Only query the last 30 days of history
-      AND _TABLE_SUFFIX
-        BETWEEN FORMAT_DATE(
-          '%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))
-        AND FORMAT_DATE('%Y%m%d', CURRENT_DATE())
+      AND DATE(timestamp)
+        BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
+        AND CURRENT_DATE()
 
 +---------------+
 | num_downloads |
 +===============+
-| 2117807       |
+| 20531925      |
 +---------------+
 
 To only count downloads from pip, filter on the ``details.installer.name``
@@ -129,71 +128,94 @@ column.
 
     #standardSQL
     SELECT COUNT(*) AS num_downloads
-    FROM `the-psf.pypi.downloads*`
+    FROM `the-psf.pypi.file_downloads`
     WHERE file.project = 'pytest'
       AND details.installer.name = 'pip'
       -- Only query the last 30 days of history
-      AND _TABLE_SUFFIX
-        BETWEEN FORMAT_DATE(
-          '%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))
-        AND FORMAT_DATE('%Y%m%d', CURRENT_DATE())
+      AND DATE(timestamp)
+        BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
+        AND CURRENT_DATE()
 
 +---------------+
 | num_downloads |
 +===============+
-| 1829322       |
+| 19391645      |
 +---------------+
 
 Package downloads over time
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-To group by monthly downloads, use the ``_TABLE_SUFFIX`` pseudo-column. Also
-use the pseudo-column to limit the tables queried and the corresponding
-costs.
+To group by monthly downloads, use the ``TIMESTAMP_TRUNC`` function. Also
+filtering by this column reduces corresponding costs.
 
 ::
 
     #standardSQL
     SELECT
       COUNT(*) AS num_downloads,
-      SUBSTR(_TABLE_SUFFIX, 1, 6) AS `month`
-    FROM `the-psf.pypi.downloads*`
+      DATE_TRUNC(DATE(timestamp), MONTH) AS `month`
+    FROM `the-psf.pypi.file_downloads`
     WHERE
       file.project = 'pytest'
       -- Only query the last 6 months of history
-      AND _TABLE_SUFFIX
-        BETWEEN FORMAT_DATE(
-          '%Y%m01', DATE_SUB(CURRENT_DATE(), INTERVAL 6 MONTH))
-        AND FORMAT_DATE('%Y%m%d', CURRENT_DATE())
+      AND DATE(timestamp)
+        BETWEEN DATE_TRUNC(DATE_SUB(CURRENT_DATE(), INTERVAL 6 MONTH), MONTH)
+        AND CURRENT_DATE()
     GROUP BY `month`
     ORDER BY `month` DESC
 
-+---------------+--------+
-| num_downloads | month  |
-+===============+========+
-| 1956741       | 201801 |
-+---------------+--------+
-| 2344692       | 201712 |
-+---------------+--------+
-| 1730398       | 201711 |
-+---------------+--------+
-| 2047310       | 201710 |
-+---------------+--------+
-| 1744443       | 201709 |
-+---------------+--------+
-| 1916952       | 201708 |
-+---------------+--------+
-
-More queries
-~~~~~~~~~~~~
-
-- `Data driven decisions using PyPI download statistics
-  <https://langui.sh/2016/12/09/data-driven-decisions/>`__
-- `PyPI queries gist <https://gist.github.com/alex/4f100a9592b05e9b4d63>`__
-- `Python versions over time
-  <https://github.com/tswast/code-snippets/blob/master/2018/python-community-insights/Python%20Community%20Insights.ipynb>`__
-- `Non-Windows downloads, grouped by platform
-  <https://bigquery.cloud.google.com/savedquery/51422494423:ff1976af63614ad4a1258d8821dd7785>`__
++---------------+------------+
+| num_downloads | month      |
++===============+============+
+| 1956741       | 2018-01-01 |
++---------------+------------+
+| 2344692       | 2017-12-01 |
++---------------+------------+
+| 1730398       | 2017-11-01 |
++---------------+------------+
+| 2047310       | 2017-10-01 |
++---------------+------------+
+| 1744443       | 2017-09-01 |
++---------------+------------+
+| 1916952       | 2017-08-01 |
++---------------+------------+
+
+Python versions over time
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Extract the Python version from the ``details.python`` column. Warning: This
+query processes over 500 GB of data.
+
+::
+
+    #standardSQL
+    SELECT
+      REGEXP_EXTRACT(details.python, r"[0-9]+\.[0-9]+") AS python_version,
+      COUNT(*) AS num_downloads,
+    FROM `the-psf.pypi.file_downloads`
+    WHERE
+      -- Only query the last 6 months of history
+      DATE(timestamp)
+        BETWEEN DATE_TRUNC(DATE_SUB(CURRENT_DATE(), INTERVAL 6 MONTH), MONTH)
+        AND CURRENT_DATE()
+    GROUP BY `python_version`
+    ORDER BY `num_downloads` DESC
+
++--------+---------------+
+| python | num_downloads |
++========+===============+
+| 3.7    | 12990683561   |
++--------+---------------+
+| 3.6    | 9035598511    |
++--------+---------------+
+| 2.7    | 8467785320    |
++--------+---------------+
+| 3.8    | 4581627740    |
++--------+---------------+
+| 3.5    | 2412533601    |
++--------+---------------+
+| null   | 1641456718    |
++--------+---------------+
 
 Caveats
 =======
@@ -229,13 +251,12 @@ the official Python client library for BigQuery.
 
     query_job = client.query("""
     SELECT COUNT(*) AS num_downloads
-    FROM `the-psf.pypi.downloads*`
+    FROM `the-psf.pypi.file_downloads`
     WHERE file.project = 'pytest'
-    -- Only query the last 30 days of history
-    AND _TABLE_SUFFIX
-        BETWEEN FORMAT_DATE(
-            '%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))
-        AND FORMAT_DATE('%Y%m%d', CURRENT_DATE())""")
+      -- Only query the last 30 days of history
+      AND DATE(timestamp)
+        BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
+        AND CURRENT_DATE()""")
 
     results = query_job.result()  # Waits for job to complete.
     for row in results:

diff --git a/source/guides/distributing-packages-using-setuptools.rst b/source/guides/distributing-packages-using-setuptools.rst
@@ -508,9 +508,8 @@ entry_points
 Use this keyword to specify any plugins that your project provides for any named
 entry points that may be defined by your project or others that you depend on.
 
-For more information, see the section on `Dynamic Discovery of Services and
-Plugins
-<https://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discovery-of-services-and-plugins>`_
+For more information, see the section on `Advertising Behavior
+<https://setuptools.readthedocs.io/en/latest/userguide/entry_point.html#dynamic-discovery-of-services-and-plugins>`_
 from the :ref:`setuptools` docs.
 
 The most commonly used entry point is "console_scripts" (see below).

diff --git a/source/guides/github-actions-ci-cd-sample/publish-to-test-pypi.yml b/source/guides/github-actions-ci-cd-sample/publish-to-test-pypi.yml
@@ -13,28 +13,28 @@ jobs:
       uses: actions/setup-python@v1
       with:
         python-version: 3.7
-    - name: Install pep517
+    - name: Install pypa/build
       run: >-
         python -m
         pip install
-        pep517
+        build
         --user
     - name: Build a binary wheel and a source tarball
       run: >-
         python -m
-        pep517.build
-        --source
-        --binary
-        --out-dir dist/
+        build
+        --sdist
+        --wheel
+        --outdir dist/
         .
     # Actually publish to PyPI/TestPyPI
     - name: Publish distribution 📦 to Test PyPI
       uses: pypa/gh-action-pypi-publish@master
       with:
-        password: ${{ secrets.test_pypi_password }}
+        password: ${{ secrets.TEST_PYPI_API_TOKEN }}
         repository_url: https://test.pypi.org/legacy/
     - name: Publish distribution 📦 to PyPI
       if: startsWith(github.ref, 'refs/tags')
       uses: pypa/gh-action-pypi-publish@master
       with:
-        password: ${{ secrets.pypi_password }}
+        password: ${{ secrets.PYPI_API_TOKEN }}
diff --git a/source/guides/index.rst b/source/guides/index.rst
@@ -7,13 +7,20 @@ introduction to packaging, see :doc:`/tutorials/index`.
 
 .. toctree::
    :maxdepth: 1
+   :caption: Installing Packages:
 
-   tool-recommendations
    installing-using-pip-and-virtual-environments
    installing-stand-alone-command-line-tools
    installing-using-linux-tools
    installing-scientific-packages
    multi-version-installs
+   index-mirrors-and-caches
+   hosting-your-own-index
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Building and Publishing Projects:
+
    distributing-packages-using-setuptools
    using-manifest-in
    single-sourcing-package-version
@@ -23,10 +30,14 @@ introduction to packaging, see :doc:`/tutorials/index`.
    supporting-windows-using-appveyor
    packaging-namespace-packages
    creating-and-discovering-plugins
-   analyzing-pypi-package-downloads
-   index-mirrors-and-caches
-   hosting-your-own-index
    migrating-to-pypi-org
    using-testpypi
    making-a-pypi-friendly-readme
    publishing-package-distribution-releases-using-github-actions-ci-cd-workflows
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Miscellaneous:
+
+   tool-recommendations
+   analyzing-pypi-package-downloads
diff --git a/source/guides/installing-using-linux-tools.rst b/source/guides/installing-using-linux-tools.rst
@@ -98,7 +98,7 @@ To install pip, wheel, and setuptools, in a parallel, non-system environment
 (using yum) then there are two options:
 
 
-1. Use the "Sofware Collections" feature to enable a parallel collection that
+1. Use the "Software Collections" feature to enable a parallel collection that
    includes pip, setuptools, and wheel.
 
    * For Redhat, see here:

diff --git a/source/guides/installing-using-pip-and-virtual-environments.rst b/source/guides/installing-using-pip-and-virtual-environments.rst
@@ -7,7 +7,7 @@ for Python 2. These are the lowest-level tools for managing Python
 packages and are recommended if higher-level tools do not suit your needs.
 
 .. note:: This doc uses the term **package** to refer to a
-    :term:`Distribution Package`  which is different from a :term:`Import
+    :term:`Distribution Package`  which is different from an :term:`Import
     Package` that which is used to import modules in your Python source code.
 
 

diff --git a/source/guides/multi-version-installs.rst b/source/guides/multi-version-installs.rst
@@ -32,7 +32,7 @@ This can be worked around by setting all dependencies in
 ``__main__.__requires__`` before importing ``pkg_resources`` for the first
 time, but that approach does mean that standard command line invocations of
 the affected tools can't be used - it's necessary to write a custom
-wrapper script or use ``python -c '<commmand>'`` to invoke the application's
+wrapper script or use ``python -c '<command>'`` to invoke the application's
 main entry point directly.
 
 Refer to the `pkg_resources documentation

diff --git a/source/guides/packaging-binary-extensions.rst b/source/guides/packaging-binary-extensions.rst
@@ -112,7 +112,7 @@ profiling has identified the code where the speed increase is worth
 additional maintenance effort), a number of other alternatives should
 also be considered:
 
-* look for existing optimised alternatives. The CPython standard libary
+* look for existing optimised alternatives. The CPython standard library
   includes a number of optimised data structures and algorithms (especially
   in the builtins and the ``collections`` and ``itertools`` modules). The
   Python Package Index also offers additional alternatives. Sometimes, the