\documentclass[12pt]{article}

\usepackage[top=1in, bottom=1in, left=1in, right=1in]{geometry}

\usepackage{fontspec}

\setmainfont{Arial}

\usepackage{amssymb}

\usepackage{amsmath}

\usepackage{latexsym}

\usepackage{color}

\usepackage{graphicx}

\usepackage{threeparttable}

\usepackage{setspace}

\usepackage[labelfont=bf]{caption}

\usepackage{subcaption}

\usepackage{wrapfig}

\usepackage[numbers,super]{natbib}

\bibliographystyle{unsrtnat}

\usepackage[nottoc,numbib]{tocbibind}

\renewcommand{\tocbibname}{References\setcounter{page}{1}}

\usepackage{tocloft}

\renewcommand{\cftsecleader}{\cftdotfill{\cftdotsep}}

\usepackage{titling}

\setlength{\droptitle}{-50pt}

\pretitle{\begin{center}\Large\bfseries\vspace{0ex}}%

\posttitle{\end{center}\Large\vspace{-2ex}}%

\preauthor{\begin{center}\large}%

\postauthor{\end{center}\large\vspace{-3ex}}%

\predate{\begin{center}\large}%q

\postdate{\end{center}\large\vspace{-6ex}}%

\usepackage[noindentafter]{titlesec}

\titleformat{\section}{\large\bfseries}{\thesection}{1em}{}

\titlespacing{\section}{0pt}{18pt plus 2pt minus 2pt}{4pt plus 2pt minus 2pt}[0pt]

\titlespacing{\subsection}{0pt}{16pt plus 2pt minus 2pt}{4pt plus 2pt minus 2pt}[0pt]

\titlespacing{\subsubsection}{0pt}{14pt plus 2pt minus 2pt}{4pt plus 2pt minus 2pt}[0pt]

\usepackage{enumitem}

\setlist{nolistsep}

\setlist[2]{noitemsep}

\setlist[1]{noitemsep}

\usepackage{booktabs}

\usepackage{multirow, hhline}

\usepackage{array}

\usepackage[table]{xcolor}% http://ctan.org/pkg/xcolor

\usepackage{pgfgantt}

\usepackage{url}

\urlstyle{rm}

\usepackage{lineno}

\usepackage[rightcaption]{sidecap}

\sidecaptionvpos{figure}{t}

\usepackage{wrapfig}

\renewcommand{\floatpagefraction}{.8}%

\renewcommand{\thefootnote}{\fnsymbol{footnote}}

\numberwithin{page}{section}

\renewcommand{\thepage}{\thesection-\arabic{page}}

\renewcommand{\paragraph}{%

\@startsection{paragraph}{4}%

{\z@}{1.25ex \@plus 0ex \@minus .2ex}{-.5em}%

{\normalfont\normalsize\itshape\bfseries}%

}

\begin{document}

\title{Revamping Matplotlib for Modern Data Structures}

\author{Thomas A.\ Caswell}

\date{}

\setcounter{tocdepth}{2}

\thispagestyle{empty}

\section{Scientific/Technical/Management (S/T/M)}

\setcounter{page}{1}

\subsection{Matplotlib's role in scientific computing}

Matplotlib~\cite{Hunter:2007} is a comprehensive Python library for

creating static, animated, and interactive visualizations. It

provides the visualization for projects across all SMD divisions,

including flagship missions like the Hubble Space Telescope (HST) and

the James Webb Space Telescope~\cite{jwst_pipeline} (JWST), and

instruments like the Curiosity Rover

Mastcam~\cite{https://doi.org/10.1002/2016EA000219}.

Matplotlib is part of the Scientific Python Ecosystem (SPE), a loosely

defined community of projects and programmers with the common goal of

advancing science through the use of the Python programming language.

This development is largely volunteer work or work that is sponsored

implicitly by specific science projects. SPE, shown with a rough

schematic in Figure \ref{fig:ecosystem}, has a core of general purpose

domain-agnostic tools, like NumPy~\cite{Harris2020} and

SciPy~\cite{Virtanen2020}. Matplotlib is in the next ring of

specialized, but still domain agnostic tools, and is the most

prevalent data visualization library for the SPE. The outer rings are

increasingly domain-specific tools, like

AstroPy~\cite{astropy:2013,

astropy:2018} for astrophysics,

SunPy~\cite{sunpy_community2020} for heliophysics, and

MetPy~\cite{metpy} and cartopy~\cite{Cartopy} for earth sciences; all of these use

Matplotlib for specialized plotting. This layered approach gives

scientists and engineers convenient and powerful high-level tools

while enabling direct access to the underlying libraries when needed.

\begin{wrapfigure}{r}{0.5\textwidth}

\includegraphics[width=0.45\textwidth]{scipy-ecosystem}

\caption{\small A schematic of the Scientific Python ecosytem. At the

core we have the Python language itself with concentric rings of

domain agnostic to domain specific libraries. Both AstroPy (top

left) and SunPy (top right), used in the astrophysics and

heliophysics divisions respectively, rely on Matplotlib (center, circled)

Credit: Jake van der Plas, ``The Unexpected Effectiveness of Python

in Science'', PyCon 2017}

\label{fig:ecosystem}

\end{wrapfigure}

Soon after its start by John D. Hunter in 2002, Matplotlib began growing as an

open source plotting

library with a BSD-compatible license. The

first commits in the Matplotlib history date to early 2003. Over the

last 17 years Matplotlib has been actively developed and maintained by

a vibrant, primarily volunteer, community. Matplotlib has over 1,250

individual contributors to the code base, with many more individuals

having contributed in ways not easily tracked, such as answering user

questions on the mailing list and reporting issues.

Matplotlib has a high-level API for quick plotting, such as in

exploratory data analysis, plus a low-level API that gives full control for

fine-tuned publication-quality plots, for animations, for writing both

GUI-independent and GUI-specific tools for interactive data plotting, and

for output to a variety of vector and raster

formats including \texttt{svg}, \texttt{eps},

\texttt{pdf}, and \texttt{png}.

Matplotlib's generality and flexibility make it readily extendable to

many data visualization domains. A rich ecosystem of downstream packages use

Matplotlib as their base, providing domain-specific streamlining of

Matplotlib's API: including

yt~\cite{2011ApJS..192....9T}, AstroPy~\cite{astropy:2013,

astropy:2018}, ArviZ~\cite{arviz_2019},

seaborn~\cite{waskom2020seaborn}, xarray~\cite{hoyer2017xarray},

cartopy~\cite{Cartopy}, corner.py~\cite{corner} and

mpl-scatter-density~\cite{mpl-scatter-density}.

Matplotlib is ubiquitous in scientific computing. Conservative

estimates based on downloads and on usage of the documentation website

indicate that Matplotlib has over a million users. We expect that a

large fraction of NASA SMD-sponsored projects rely on this shared

infrastructure to some degree, but this is hard to quantify.

It is not usual to cite tool chains in scientific literature, so

citation counts are likely to vastly under-represent the scientific

use of Matplotlib. For example, a canonical reference for

Matplotlib~\cite{Hunter:2007} has over 12,750 citations overall and

over 4,500 citations in ADS~\cite{ads_mpl},

while a common reference of NumPy~\cite{walt2011numpy} has over 6,800

citations. These counts illustrate the problem in measuring usage via

citations: Matplotlib depends on NumPy, yet Matplotlib has almost

twice as many citations.

\subsection{Objectives}

Matplotlib is a primarily community-driven project, but we have grown

to the point where we need supported developers with the time to

organize, plan, and make decisions. Supported developers will allow

us to take on more complex projects than can be implemented by part-time

volunteer effort alone, while executing critical day-to-day

maintenance tasks required to keep the project healthy.

\textbf{We propose to split the effort on this grant equally between

maintenance}.

\subsubsection{Refactor the Internal Data Model}

\label{sec:ridm}

Matplotlib is based on visualizing data stored in Numpy arrays,

which is a simple yet flexible data model that has been the

underpinning of data visualization for the last few decades. However,

new data types are being developed and standardized to carry

structure and metadata along with the numerical values of the data.

Physical units are a key example of such metadata

that a plotting package should handle gracefully. Modern data

structures can also describe massive data sets and allow performing

operations without loading the whole data set into memory

(i.e. dask, xarray); Matplotlib should leverage these packages

to work efficiently with Big Data.

Eliding many of the implementation details, Matplotlib can be thought

of as having three primary layers~\cite{AOSA_mpl}: the user-facing plotting

methods; the \texttt{Artist} layer; and the backends. The core of the

architecture is the set of \texttt{Artist}s which can be understood as

objects that describe something for a backend to render to the output device.

Each individual

\texttt{Artist} is responsible for holding the user data and style

information for part of the final visualization, but the exact data

held varies among \texttt{Artist} classes. For example,

\texttt{matplotlib.text.Text} is responsible for drawing text and

holds, among other things, the location of the text, the font family

and size, the color, and the actual string. Internally Matplotlib

represents a \texttt{Figure} as a tree of \texttt{Artist}s. The

plotting methods in Matplotlib create these objects and add them to

the tree. To render the final output Matplotlib walks the

\texttt{Artist} tree depth-first and each each \texttt{Artist} is

responsible for rendering itself via the backend. Thus, by changing

the backend we can change the output format from raster to vector

without making any changes to the \texttt{Artist} tree.

Currently, each plotting method and \texttt{Artist} independently

handles sanitizing and storing data. Hence, each \texttt{Artist}

subclass holds its data in a different way and some common

functionality, such as handling data with attached units (e.g.,

degrees Celsius, dates), is repeated throughout the code base. This

leads to inconsistencies across the library and makes it difficult for

users to write code that updates interactive explorations or

animations. Another consequence of the data storage is that

Matplotlib also cannot exploit ``Structured data''--combining multiple

pieces of (possibly heterogeneous) data with labels and metadata into

single data structure; instead, users must extract data elements and

reassemble them as arguments to Matplotlib plotting functions.

We have on-going theoretical and design work\footnote{Supported by the

Chan Zuckerberg Initiative, see current support.} to re-architect

Matplotlib to better separate the data representation from the rest of

the library. We are developing a consistent data access API such that

each \texttt{Artist} will have a single \texttt{DataSource} object

responsible for providing access to all of the data that the

\texttt{Artist} needs. This design will allow us to overcome many of

the issues with the current design, opening the door to

new functionality. The current work is expected to produce design

documents and a proof-of-principal implementation by the end of 2021.

Based on this preliminary work \textbf{we propose refactoring all of

Matplotlib to use the new data model}. The proposed design is

consistent with the rest of the architecture; just as the

\texttt{Artist}s are currently agnostic to the backend used, the

\texttt{Artist}s will be agnostic to the internals of its \texttt{DataSource}.

An immediate motivation for this refactor is to improve

Matplotlib's handling of data with physical units attached. Units are

fundamental to science and engineering, but most numerical

software is unit-naive. It is the user's responsibility to

convert data to consistent units prior to any

computation. Failing to do so leads to the most insidious of bugs:

everything ``works'' but silently gives incorrect results! There are

several libraries which provide unit-aware data structures in Python

that can be used with Matplotlib for unit-aware plotting.

\textbf{This capability is currently used to support spaceflight

system}; NASA supported its initial development in Matplotlib.

The current support for physical units in Matplotib is based on

inspecting the incoming data type. When the the data structure supplied by

the user includes units, Matplotlib looks for a registered converter and sets

default axis labels, limits, and functions for locating and formatting

the ticks. Internally, Matplotlib uses the unit machinery to handle

datetime and string-categorical types. Third-party libraries are

able to register their own converters. With this machinery users can

control the units in which the data is plotted, which may differ from

the units in which it was supplied. Axis limits may be specified

in physical

units.

There are several problems with the current unit support in

Matplotlib. It is under-documented, making it hard for users to

understand and use, and hard for developers to extend. Test

coverage is incomplete, and inconsistencies have accumulated in the

code base. These range from subtle inconsistencies between plotting

methods to data with units simply not working with some methods.

\textbf{To address this we propose to use unit-handling as a

motivating feature while doing the \texttt{DataSource} refactoring}.

Unit logic will be consolidated in the \texttt{DataSource}.

There is a broad recognition across the SPE of the importance of

supporting physical units as an intrinsic component of computation.

There is on-going work in NumPy (NEP40-43), pandas, and xarray to add

support for physical units to their foundational data structures. The

proposed work will ensure that Matplotlib can take advantage

of such efforts, for seamless handling of data with units from computation

to visualization.

In addition to fixing problems with the current

support of physical units, this refactoring will lay the ground work for

\begin{itemize}[noitemsep]

\item smart down-sampling of plotted data based on view limits,

\item native consumption of structured data,

\item seamless updating of the underlying data, either interactively

or via streams,

\item and use of alternative data sources such as out-of-core data

structures, database queries or analytic functions.

\end{itemize}

\subsubsection{General maintenance}

While rewiring Matplotlib's data model,

we need to sustain Matplotlib as a vibrant project and

continue to support the hundred of thousands of users and hundreds of

downstream packages. To maintain Matplotlib's health we must:

\begin{itemize}[noitemsep]

\item fix critical bugs and regressions promptly as they are reported;

\item make improvements while maintaining backward compatibility and

documenting changes;

\item categorize Issues and PRs in terms of topic, difficulty, and

urgency;

\item entrain new contributors to sustain and diversify the community

developer team;

\item maintain the continuous-integration infrastructure;

\item manage the release process;

\item and manage community discussions about proposed enhancements, features,

and API changes.

\end{itemize}

These are tasks that are never ``done''; users find new ways to use

the library, they expose previously un-detected bugs, they request or propose new

features, and the world continues to changes around us. \textbf{Dedicated

manner, and provide the crucial roadmaps that will steer the future of the library}.

Historically, pull requests (PRs) and issues have been submitted

faster than they can be reviewed; Matplotlib has accumulated about 300

open PRs and 1300 open issues due to this imbalance. There are

critical bug reports and insightful feature requests among the issue

backlog, while among the PR backlog there are useful contributions or

bug fixes that would improve the libraries for direct users and

downstream packages. The large backlog is discouraging for new and

occasional contributors and distracting for core developers.

In 2020, with a grant from CZI supporting a Research Software Engineer

(RSE) working on Matplotlib for 10 month, we resolved 2,056 PRs and

999 issues. However, even at this rate we only reduced the PR backlog

by 70 open PRs and held the issue backlog even. As part of reducing

the backlog, we were able to fix several long-standing bugs. We have

demonstrated that supporting maintenance work has a positive effect on

the health of the project and \textbf{the additional resources

maintenance backlog}.

\subsection{Impact and Relevance to Science Mission Directorate}

As mentioned above, it is hard to estimate the prevalence of open

source software due to it being freely distributed and not

systematically cited. Matplotlib is in the top 100 most-downloaded

packages from PyPI~\cite{pypi_stats} and is packaged by every major

Linux and scientific Python distribution. Another metric to measure

impact is static analysis of publicly available code. According to

github over 299k repositories and over 12k packages hosted on GitHub

depend on Matplotlib~\cite{gh_deps:2021}. A recent study of the

almost 10M Jupyter notebooks on github found that over 3M of them have

a direct Matplotlib import~\cite{datalore:2020}.

These measures show broad impact but cover many uses outside of the

SMD community. If we restrict the static analysis to

\url{https://github.com/nasa}, NASA's official organization on GitHub,

we see that 53 of the 330 hosted repositories make reference to

Matplotlib. Soliciting on social media for examples of Matplotlib in SMD research

yielded a (non-exhaustive) list including:

\begin{itemize}[noitemsep]

to study

thunderstorms~\cite{https://doi.org/10.1002/2016JD025299,https://doi.org/10.1029/2019JD030874},

seasonal ocean winds~\cite{https://doi.org/10.1002/2017JD027516} and

tropical storms~\cite{Lang_2020};

\item in the first science paper from the Parker Solar

probe~\cite{Bale2019};

\item in the Martian science program in both

orbiter~\cite{https://doi.org/10.1029/2019JE006188} and

rover~\cite{https://doi.org/10.1002/2016EA000219} contexts;

\item as part of ground operations from the Mars Phoenix Lander;

\item with data from Kepler and K2 missions to study Trojan

asteroids~\cite{Nixon_2019} and

Titan~\cite{Ryan_2017,2019PASP..131h4505P};

\item on the New Horizons Kuiper belt extended

mission~\cite{Porter_2018};

\item for visualization of scheduling, safety and constraint checks,

and telemetry by the Swift science operations

team~\cite{swift_ops,2020ApJ...900...35T};

\item as part of the HST and JWST~\cite{jwst_pipeline} data processing

pipelines;

\item in Monte, JPL's mission design

and navigation software system to support spaceflight operations;

\item in fundamental research on graphene~\cite{PhysRevLett.120.236802};

\item and for work on nuclear powered rockets~\cite{leu_cerment}.

\end{itemize}

This list shows the breadth of Matplotlib usage across the SMD

divisions. Matplotlib falls under the umbrella of projects which the

National Academies of Science recommended for funding in the report

Sciences (2018)''~\cite{NAP25217} (Option B4).

\textbf{Small investments in Matplotlib will have returns across the

entire SMD portfolio}.

Maintaining Matplotlib's health and engineering for its future will

have wide-ranging impacts for the scientific community, and for NASA in

particular. As pointed out above, Matplotlib is the plotting layer

for much of the Python ecosystem. It has been used in thousands of

academic papers, plotting for news and sporting media, technical

dash-boarding, and most modern data science applications. As of

2018Q2 over a third of papers published to astro-ph on the Arxiv

contained a figure generated with

Matplotlib~\cite{arxvi_stats},

with a clear upward trend. Matplotlib has also been used in a number

of high-profile astronomy projects including the Nobel prize-winning

observation of black hole merger by the Laser Interferometer

Gravitational-Wave Observatory (LIGO), the 2020 Nobel prize-winning

discovery of a super massive compact object at the center of our

galaxy and the recent observation of Sagittarius A* by the Event

Horizon Telescope (EHT).

This proposal is another step towards Matplotlib's sustainability, and

ability to move forward. Given the usage and the scale of the library

it is not sustainable to continue to maintain Matplotlib as a

volunteer-only effort. A core team of full-time developers and

managers will better co-ordinate and nurture volunteer efforts, with

the goal of growing and sustaining a diverse community of volunteer

expert contributors.

Finally, with explicit support for their time, Matplotlib's core developers

will better be able to plan the project's future.

This requires vision, co-ordination with the rest of the scientific

Python community, including downstream libraries, and getting out in

the community to get feedback. It is very hard to provide that

leadership and vision while working a ``day job''.

\subsection{Risk Management}

There is minimal risk in the maintenance work. There is a large volume

of individually small tasks that need to be addressed: the issue and

PR backlogs. Any dedicated effort devoted to clearing the backlog

will have a positive impact on the project. We have demonstrated that

supporting developers for maintenance work is an effective and

efficient use of resources.

The data pipeline refactor carries more risk. Given the large user

base it is imperative that changes are as backwards compatible as

possible. While this large user base is on one hand a lever that

multiplies the value any improvements, it also makes changes more expensive. We

cannot simply change our public interface to track the internal

refactoring, instead we must also write code to translate the

old API to the new implementation. This engineering work makes the

refactor more challenging and time consuming than starting from

scratch. We will rely on, and expand, our extensive test suite to

catch any unintentional changes early in the development process.

There is also the risk than we are underestimating the amount of work

that will be required. There is always a possibility that during

refactoring the scope of work will rapidly snowball due to unexpected

internal dependencies or edge cases.

However, even in the event that we cannot complete the implementation of a new data

pipeline model within the performance period, throughout the process we will be working towards

improving library internals and improving our

documentation.

We anticipate that all of the work can be done incrementally and will

be merged to the default branch of Matplotlib throughout the

performance period. Frequently merging incremental work reduces the

risk, and is consistent with our standard development process. Work

done under this proposal will be reviewed in our normal fashion, and many

small PRs are easier to review and merge than a single large PR.

Large PRs also quickly drift out of sync with the master branch, and

develop insurmountable rebase requirements. The improvements, both to

the code and documentation, will be released to users as part of our

standard semiannual release cadence.

There is personnel risk in that we have not yet identified the RSE who will

be a key individual for carrying out this work. We are optimistic that we will

be able to recruit a qualified individual in reasonable time. In early

2020 we ran a similar search for a 1 year position and had a number of

qualified candidates, one of whom is currently employed via NumFOCUS as

a RSE for Matplotlib.

\subsection{Contributions of Principal Investigator and Key Personnel}

Dr.\ Thomas A.\ Caswell has the sole individual responsibility for

directing and supervising the execution of this work. He has

extensive experience developing and maintaining Python libraries for

scientists. He has been involved with Matplotlib from 2012 and had a

leadership role in the project from 2014. He is also a core developer

of h5py and has contributed to many of the other core projects of the

SPE. At Brookhaven National Laboratory he is the lead architect and

developer of the Bluesky Suite, an ecosystem of co-developed libraries

and applications for data acquisition and management, which is being

adopted for beamline operations at synchrotrons in the US and across

the world.

\subsection{Work Plan}

\begin{ganttchart}[

vgrid,

title height=1,

milestone inline label node/.append style={left=5mm},

x unit=.75cm,

%inline,

%expand chart=\textwidth,

%milestone/.append style={xscale=12*.75cm/\textwidth},

milestone/.append style={xscale=1},

bar height=.75,

bar top shift=.125,

y unit chart=.75cm,

y unit title=.75cm]{1}{12}

\gantttitle{2021}{2}

\gantttitle{2022}{4}

\gantttitle{2023}{4}

\gantttitle{2024}{2}\\

\gantttitle{Y1}{4}

\gantttitle{Y2}{4}

\gantttitle{Y3}{4} \\

\gantttitlelist[title list options=%

{var=\y, evaluate=\y as \x%

using " Q\y "}]{1,...,4}{1} \

\gantttitlelist[title list options=%

{var=\y, evaluate=\y as \x%

using " Q\y "}]{1,...,4}{1}

\gantttitlelist[title list options=%

{var=\y, evaluate=\y as \x%

using " Q\y "}]{1,...,4}{1} \\

\ganttmilestone{Hire RSE}{1} \ganttnewline

\ganttbar{Document Units (internal)}{1}{2} \ganttnewline

\ganttbar{Planning and Design}{2}{4} \ganttnewline

\ganttmilestone{\texttt{DataSource} API stable}{3} \ganttnewline

\ganttbar{Refactor Data Layer}{4}{12} \ganttnewline

\ganttmilestone{SciPy Talk}{4} \ganttmilestone{}{8} \ganttmilestone{}{12} \ganttnewline

\ganttbar[inline]{General Maintenance (ongoing)}{1}{12} \ganttnewline

\ganttmilestone{Matplotlib Releases}{2} \ganttmilestone{}{4} \ganttmilestone{}{6} \ganttmilestone{}{8} \ganttmilestone{}{10} \ganttmilestone{}{12}

\end{ganttchart}

\subsubsection{General Maintenance}

Throughout the project both Dr.\ Caswell and the RSE will work on

Matplotlib general maintenance and community engagement. This includes,

but is not limited to, fixing bugs, reviewing Pull Requests, answering

user questions, welcoming new contributors to the project, and

planning for the future. Dr.\ Caswell will particularly focus on

the project roadmap and community governance.

\subsubsection{Data Model Refactor}

\paragraph{Document Existing State of Data Pipeline}

We will document, both at the conceptual and API level, how the unit

dispatch system currently works in theory and practice. Given the

importance of backwards compatibility it is critical to understand

what we are refactoring before beginning. This documentation will be

aimed at Matplotlib developers and focus on the low-level details.

Unfortunately, the original authors of the unit-handling code are no

longer active with the project so we may have to reverse engineer some

aspects of the code and motivation. This documentation will be used

in further planning and as a reference for what is ``correct''

behavior when resolving inconsistencies.

\paragraph{Refactor Planning and Design}

The data pipeline refactor is a critical part of the library that

touches most artists, so we require a comprehensive plan. As part of

developing this plan we will reach out to upstream data-structure

libraries, such as pandas, numpy, xarray and the Consortium for Python

Data API Standards~\footnote{https://data-apis.org/}, to downstream

libraries, such as astropy, yt, and sunpy, and to users of the unit

machinery, such as \texttt{unyt}~\cite{Goldbaum2018},

\texttt{pint}~\cite{pint}, \texttt{astropy.units}~\cite{astropy:2013,

astropy:2018, astropyunits}, and the Monte developers at JPL. This

will be a unique chance to change the internal architecture of

Matplotlib and we need to ensure that we engage with all stakeholders.

\paragraph{API Design} The core of the refactor will be a newly developed \texttt{DataSource}

interface. This will provide a consistent interface between the

\texttt{Artists} and their data. This interface, and the ability to

provide alternate implementation of data storage and access, is what

will allow the features discussed in section \ref{sec:ridm}. This

builds on our theoretical and proof-of-concept work that will be

finished by the end of 2021. Active work on both projects will

overlap and we will actively engage to ensure that the theoretical

work is effectively translated. We will fully specify the

\texttt{DataSource} interface and a write a reference implementation

by the end of Y1Q3. Concurrently we will developed a detailed

schedule to refactor the approximately 100 \texttt{Artist} classes in

Matplotlib to use the new data model.

\paragraph{Refactor Data Layer} As mentioned above, it is critical that we maintain as much backwards

compatibility as possible in the API changes needed for the refactoring. The variation

among \texttt{Artist}s is both what we are trying to address and the

biggest complicating factor. Each class well need to be handled on a

case-by-case basis. We anticipate that there will be significant

collateral work, such as writing additional tests to ensure that we do

not accidentally break current behavior, and implementing

back-compatibility shims. As we

perform this refactor we will also add tests that each \texttt{Artist}

and plotting method handles data with units correctly

\textbf{This work will impact a majority of the library} and is the

largest proposed change to the library in over a decade.

This whole process will be extensively documented, both inward facing,

so that future Matplotlib developers understand why decisions were

made, and outward facing so that users can make the most of the new

and improved functionality.

A secondary goal is to extend the unit machinery to improve the user

experience. Currently much of the mechanism of unit-handling is

implicit and automatic based on type inference. We propose to extend

the interface to allow optional direct control of the unit machinery

by users and third-party library developers.

\subsubsection{Work Process}

The Research Software Engineer (RSE), yet to be identified,

will devote 100\% FTE to this proposal. They will split their time

evenly between general maintenance and the data model refactoring. If

the refactoring is completed faster than expected, the RSE will devote

100\% of their effort to general maintenance. Dr.\ Caswell will

devote 25\% FTE to this proposal. This time will be split between

general maintenance (10\%), the data model refactor (10\%), and

management of the grant and supervision of the RSE (5\%).

All work under this proposal will undergo the same review process

as any other contribution. We

encourage many small PRs to reduce risk. At least one maintainer not

supported on this proposal

will need to sign off on any changes before they are merged. This

disseminates understanding of the proposed work and ensures

community buy-in. Finally, by merging

changes to the default branch continuously throughout the period of

work the incremental improvements will be released as part of our

semiannual release cadence.

The RSE will present the status of the work at two conferences yearly,

expected to be SciPy in July and one of PyData regional conferences in

the winter.

\subsubsection{Grant Management}

Dr.\ Caswell of BNL and the Matplotlib Lead Developer is the PI of the

proposed development. He is responsible for the quality and

direction of the proposed work and the proper use of all awarded

funds. He is also responsible for all management, and budget issues

and is the final authority for this task.

\subsection{Matplotlib Project Management}

\subsubsection{Governance and Finances}

Matplotlib is a NumFOCUS Fiscally Sponsored Project.

Matplotlib is governed by a steering council, a Project Lead (PL), and

several Deputy Project Leads (DPL)~\cite{gov}. We are a community

project and try to make all decisions, both strategic and day-to-day,

by consensus. However, if consensus can not be reached, the

responsibility falls back to the DPLs and ultimately to the PL. The

current PL is Dr.\ Caswell.

As Matplotlib has matured we have moved from a single person,

originally Dr.\ John Hunter, to a ``BDFL'' governance model. The

leadership of the project passed to Dr.\ Micheal Droettboom in 2012.

In 2014 Dr.\ Caswell and Dr.\ Droettboom were co-leads and in 2016

Dr.\ Caswell took over as the sole project lead. In 2020 we formally

adopted the current governance model.

\subsubsection{License}

Matplotlib is licensed under the Matplotlib License~\cite{mpl_lic}

which is a derivative of the Python Software Foundation

license~\cite{psf_lic} and is a permissive license compatible with the

BSD license~\cite{jdh_bsd_opinions}.

\subsubsection{Sustainability Metrics}

There are many possible ways to measure the sustainability of an

open source project. We propose to focus on growing our developer

community, reducing our Issue and PR backlogs, and maintaining a

regular release cadence.

Matplotlib is a community driven project with a vast majority of the

work done by volunteers. The thing we are shortest on is the time of

maintainers who are able and willing to review Pull Requests.

Increasing the number of regular contributors and maintainers improves

the sustainability of the project in several ways; many hands make

light work and it makes the project resilient to individuals leaving

the project.

Quantitatively evaluating maintenance work can be tricky --- some

Issues or PRs take minutes to review while others can take days to

weeks of effort --- but we believe that there is value at looking at

the total number of open Issues and PRs and the rate at which they are

addressed. We aim to close Issues and PRs faster than they are opened

until a reasonable equilibrium is reached. We will aim to hit the

following metrics:

\begin{itemize}[noitemsep]

\item Initial response to all issues / new PRs within a week

\item Resolve majority of new issues / PRs within 1 month

\item Reduce backlog of issues by 25 / quarter

\item Reduce backlog of PRs by 25 / quarter

\end{itemize}

Finally, for the user community to benefit from our work we need to do

regular releases. We will continue to maintain a bi-yearly release

cycle for feature releases with 1-3 bug-fix releases as needed.

Holding to a time-based release schedule is advantageous because it

provides predictability to our users and reduce the lag between when a

new feature is merged to the default branch and when it is generally

available.

\subsubsection{Collaboration with Related Projects}

As noted above, we will be working closely with both upstream and

downstream projects on our refactor. This work is timely as Numpy,

Pandas, and xarray are all adding new data types with richer metadata

and unit support to the Python ecosystem. Matplotlib must meet the

challenge of supporting these new data types. This will require

constant communication with those projects. Similarly, our changes

will need to meet the needs of downstream domain-specific projects.

Many of these projects already have specialized data pipelines where

and are working around the limitations of Matplotlib's data model.

Matplotlib will aggressively engage with these projects during the

data pipeline refactor to ensure that we will actually meet their

needs.

Much of the communication is through the standard communication

channels, such as project issue trackers, mailing lists or discussions

forums, and submitting PRs. The strongest relationships, both

historical and current, are with projects where we have shared

developers. For example Ryan May and Elliot sales de Andre are both

core contributors to Matplotlib and Cartopy and David Stansby is a

maintainer on both Matplotlib and \texttt{SunPy}. Micheal Droettboom,

the previous Matplotlib Project lead, was a core developer on Astropy.

Additionally, through NumFOCUS and domain-specific conferences there

are regular in-person meetings.

We will work with down-stream projects to help them improve their test

coverage of plotting code and to add jobs to their continuous

integration that test development snap-shots of Matplotlib. This is

particularly useful during the planned refactor as we will be making

extensive changes to Matplotlib that we believe are backwards

compatible, but down-stream projects and users exercise the library in

ways the test suite does not. While we hope to introduce any

regressions, catching them before release is the next best thing.

As a NumFOCUS project, we recognize the importance of every project

that is part of our open source scientific computing community. Though

we would like for our work to be funded we are committed to supporting

and collaborating with other NumFOCUS projects that receive funding

regardless of our own outcome. We believe that this attitude is

crucial for the success of our community and the sustainability of

open source projects. It is our hope that this sentiment will be taken

into consideration when evaluating our proposal.

\subsubsection{Inclusive Community Development}

Matplotlib strives to be an inclusive and open project, anyone who is

willing and able to contribute to the project should feel welcome to

do so. We have adopted the Contributor Covenant V2.0 as

our Code of Conduct~\cite{CoC}.

Matplotlib has an open development model, all work is done in the open

and welcome contributions, in the from of bug reports, feature

requests, or pull requests from anyone. We maintain an extensive

contribution guide

(\url{https://matplotlib.org/devdocs/devel/index.html}) as part of our

documentation. To improve the development and retention of new

contributors we have recently started two efforts: an ``incubator''

channel on gitter and a Triage Team.

The hardest part of getting started to contributing to open source

projects is can be simply getting started. Open source communities,

particularly big ones, can be intimidating for first time

contributors. The incubator is a semi-closed chat room where new

contributors can get support on any aspect of contributing to

Matplotlib. This include the technical aspects of the code they are

working on, help with git/github, our review process, or the social

expectations and norms of the community. The goal is that by

providing this support to first time contributors we will retain more

of them as regular contributors and then maintainers.

The issue tracker is important to communication in the project because

it serves as the centralized location for making feature requests,

reporting bugs, identifying major projects to work on, and discussing

priorities. For this reason, it is important to curate the issue

list, adding labels to issues and closing issues that are resolved or

unresolvable. Triaging issues does not require any particular

expertise in the internals of Matplotlib but is extremely valuable to

the project. To this end we have created a ``Triage Team'' in the

organization who have power to tag, milestone, and close issues. In

addition to the direct benefit of improving the issue triage and

freeing the core-developers to spend more time reviewing PRs, this

role will bring more people into the developer community and may

provide a path way to becoming regular contributors and maintainers.

Ongoing work at NumFOCUS to develop metrics will help us evaluate the

efficacy of these efforts at diversifying our contributor base.

\subsubsection{Contribution Workflow}

Matplotlib is an established community driven project in the

``federation'' model as defined by Nadia Eghbal~\cite{eghbal_2020}.

We have a core group regular maintainers who take responsibility for

reviewing and merging proposed changes to the library and welcome Pull

Requests from anyone who is interesting in contributing. We strive

for consensus and rely on the collective judgment of our maintainers

to maintain the quality and functionality of the library.

Matplotlib uses a variation on the ``git flow'' process~\cite{ghflow}