USGS Fix, NMBGMR Fix, & Other Updates by jacob-a-brown · Pull Request #66 · DataIntegrationGroup/DataIntegrationEngine

jacob-a-brown · 2026-04-29T17:05:10Z

Why

This PR addresses the following problem / context:

The USGS is in the process of or has already deprecated water services APIs. We need to use their new APIs
We should be able to pass a tag of None to the _execute_json_request method when we need to use the full response object, such as features and links
BoR does not report bicarbonate data, so it should not be queryable when running the DIE for bicarbonate
NMBGMR needs a larger timeout for water level data
the _execute_json_request and _execute_text_request methods should raise errors if unsuccessful instead of returning empty datasets. This prevents partial data sets from being accidentally returned

How

Implementation summary - the following was changed / added / removed:

USGS

There are two issues that needed to be resolved when retrieving USGS data:

I needed to get all records, with or without pagination.
I needed to stay within the allowed request rate limit.

It turns out that both of these issues could be addressed with the same solution: set the limit query parameter to the maximum allowed (50,000). By doing this I would get all of the records back in one page and I would severely cut the number of necessary requests.

It also turns out, however, that if I use the limit query parameter in a POST request the cursor-based pagination breaks. I'm not sure why, but if limit is less than the expected number of records pagination returns all USGS records and doesn't apply the filters. There are ~32,000 wells in NM that have groundwater field measurements, and I can get measurements for up to 250 wells at a time, so using ?limit=50000 in my GET and POST requests should be safe for a very long time. Nevertheless, I have commented-out code that can handle pagination if/when it is fixed.

_execute_json_request tags

Previously if no tag was specified (i.e. set or defaulted to None) then the method _execute_json_request would default to tag to "data". This had the unintended consequence of disallowing multiple keys to be used to access data from the response object, such as "features" and "links" (the latter used to follow pagination). This default was removed, allowing the full response object to be returned. I then updated BoR and NMOSE ISC Seven Rivers to explicitly set tag="data" when invoking the _execute_json_request method.

Notes

Any special considerations, workarounds, or follow-up work to note?

Use bullet points here

Previously if the tag parameter was set to Null or was not provided it would default to "data". This caused issues for some endpoints that did not have a "data" key in their response or for which the full object was desired.

BoR does not report bicarbonate data

The USGS deprecated the old water level retrieval API endpoints, so this commit updates the NWISWaterLevelSource to use the new OGC API endpoints. Additionally, the chunk size for site retrieval is reduced to 5 to avoid URI length issues and httpx read timeouts.

Use the new USGS APIs to retrieve water level data.

Copilot

Pull request overview

Updates the USGS connector to use the newer USGS Water Data OGC API and adjusts _execute_json_request behavior so callers can optionally receive the full JSON response object (e.g., to access features/links).

Changes:

Added support for returning full JSON responses from _execute_json_request when tag is not provided, and updated BoR/ISC Seven Rivers callers to pass tag="data" explicitly.
Migrated USGS NWIS site/waterlevel retrieval to the USGS OGC API endpoints (including batching for complex queries and a large limit).
Extended datetime parsing to handle an additional +00:00 ISO-like format; updated parameter-to-agency mapping for bicarbonate.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
backend/transformer.py	Adds an additional datetime format for parsing timestamps returned by sources.
backend/source.py	Changes `_execute_json_request` to no longer default `tag` to `"data"`, allowing full response objects to be returned.
backend/connectors/usgs/transformer.py	Updates USGS site transformation to match OGC API field shapes/keys.
backend/connectors/usgs/source.py	Reworks USGS site/waterlevel retrieval to use OGC API endpoints and batching/limits.
backend/connectors/isc_seven_rivers/source.py	Passes `tag="data"` explicitly to maintain prior behavior after `_execute_json_request` change.
backend/connectors/bor/source.py	Passes `tag="data"` explicitly to maintain prior behavior after `_execute_json_request` change.
backend/config.py	Adds a bicarbonate-specific agency configuration branch.

Comments suppressed due to low confidence (2)

backend/config.py:309

BICARBONATE is now handled in its own branch, but it also still appears in the following multi-parameter branch, making that entry unreachable for bicarbonate and easy to misread later. Remove BICARBONATE from the later list (or consolidate the branches) so each parameter is mapped in exactly one place.

        elif self.parameter in [BICARBONATE]:
            config_agencies = ["nmbgmr_amp", "nmed_dwb", "nmose_isc_seven_rivers", "wqp"]
            false_agencies = [
                "bor",
                "bernco",
                "cabq",
                "ebid",
                "nmose_roswell",
                "nmose_pod",
                "nwis",
                "pvacd",
            ]
        elif self.parameter in [
            BICARBONATE,
            CALCIUM,
            CHLORIDE,
            FLUORIDE,
            MAGNESIUM,
            NITRATE,
            PH,
            POTASSIUM,
            SILICA,
            SODIUM,
            SULFATE,
            TDS,
        ]:

backend/connectors/usgs/transformer.py:50

The site transformer now sets source to "USGS", but NWISWaterLevelTransformer.source_tag is still "USGS-NWIS". Other connectors keep site source and parameter source_tag aligned (e.g., BOR-RISE). Consider standardizing these to the same value to avoid inconsistent source labels in outputs.

        rec = {
            "source": "USGS",
            "id": record["properties"]["monitoring_location_id"],
            "name": record["properties"]["monitoring_location_name"],
            "latitude": record["geometry"]["coordinates"][1],
            "longitude": record["geometry"]["coordinates"][0],
            "elevation": elevation,
            "elevation_units": "ft",
            "horizontal_datum": "WGS84",
            "vertical_datum": record["properties"]["vertical_datum"],
            "aquifer": record["properties"]["national_aquifer_code"],
            "well_depth": record["properties"]["well_constructed_depth"],
            "well_depth_units": "ft",
        }
        return rec


class NWISWaterLevelTransformer(WaterLevelTransformer):
    source_tag = "USGS-NWIS"

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

jacob-a-brown · 2026-04-29T18:48:19Z

@jirhiker @likithabommasani-sketch I'm setting timeout=1800 to have it timeout after 30 minutes. If that occurs it tries again. CoPilot suggested I not use timeout=None so that there are no indefinite hangups.

jacob-a-brown · 2026-04-29T18:52:35Z

Also, @jirhiker and @likithabommasani-sketch, should we continue to call it NWIS or should it just be USGS now?

this prevents code duplication and allows all sources to benefit from retry logic when making requests to external APIs.

Copilot

Pull request overview

Copilot reviewed 17 out of 18 changed files in this pull request and generated 8 comments.

previously only the usgs api key was reset. now all are reset because of load_dotenv()

Now that errors in calling _execute_json_request are raised if nothing is returned the health checkers need to be amended

It previously called out water levels when it should have called out sites

Don't close handlers that may be shared with other loggers. Track handlers created by this module and only close those.

…tes_only If sites_only is True then do not filter by parameter code, otherwise filter by parameter code 72019 (Groundwater level)

Convert to string so that the strip method when cleaning the data works as expected. This allows for None values to be properly handled and preventserrors when trying to clean numeric values.

Copilot

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated 7 comments.

…rking

sometimes make_site_list returns a single site id as a string, which causes the batch processing logic to break since it expects a list of site ids. This change ensures that if make_site_list returns a string, it is converted to a list with one element for consistency with the batch processing logic.

…rns empty response

Copilot

Pull request overview

Copilot reviewed 19 out of 20 changed files in this pull request and generated 13 comments.

instead of relying on get_records, which may not work if config is not fully initialized. This should prevent false negatives in health checks while still providing a meaningful check of service availability.

…ng waits when USGS is unresponsive

if pagination is required then an error will be raised and the tool can be updated. no need to maintain dead code for a feature that is not currently used and may never be used.

…e dict

do a health check using get_records instead

jacob-a-brown added 6 commits April 27, 2026 13:32

fix(source): enable tag paramter for json requests to be Null

c322402

Previously if the tag parameter was set to Null or was not provided it would default to "data". This caused issues for some endpoints that did not have a "data" key in their response or for which the full object was desired.

fix(config): update config/false agencies for bicarbonate

ec9f6e3

BoR does not report bicarbonate data

fix(usgs): update transformer for combined-metadata endpoint

036e9e7

fix(usgs): update USGS water level retrieval for new APIs

2425094

Use the new USGS APIs to retrieve water level data.

fix(usgs): set limit to 50000 to avoid pagination/rate limits

500a0b1

jacob-a-brown requested review from Copilot, jirhiker and likithabommasani21 April 29, 2026 17:05

Copilot started reviewing on behalf of jacob-a-brown April 29, 2026 17:06

View session

refactor(usgs): check usgs health with get request

d93869d

Copilot AI reviewed Apr 29, 2026

View reviewed changes

jacob-a-brown added 10 commits April 29, 2026 11:14

fix(config): set agencies for bicarbonate

12247f5

fix(nmbgmr): set timeout to 30 minutes and retry failures

4eaac1c

fix(usgs): add timeout to usgs api calls and retry logic if errors

b342922

fix(usgs): add retry logic to pagination retrieval

3f2d104

refactor(usgs): set datum in site transformer

6005f70

fix(usgs): return False if health check fails

dba8213

fix(usgs): shorten error message

b17bbde

fix(usgs): set limit of 1 for health check

5570c36

fix(usgs): remove unused imports

836243e

fix(nmbgmr): retry when status_code!=200

4fd3ac6

jacob-a-brown added 6 commits April 29, 2026 12:58

fix(nmbgmr): remove print debugging statement

1fcede3

fix(nmbgmr): call self.warn not warning

c1771d7

fix(usgs): call self.warn not warning

a51b13e

refactor(nmbgmr): change note style at top of doc

43e2457

refactor(retries): move rety logic to BaseSource

c17e806

this prevents code duplication and allows all sources to benefit from retry logic when making requests to external APIs.

refactor(nmbgmr): change timeout to 15 minutes

4a64c6f

jacob-a-brown requested a review from Copilot May 18, 2026 20:49

Copilot started reviewing on behalf of jacob-a-brown May 18, 2026 20:49

View session

Copilot AI reviewed May 18, 2026

View reviewed changes

jacob-a-brown added 7 commits May 18, 2026 15:04

fix(tests): reset env var in USGS tests

ade753a

previously only the usgs api key was reset. now all are reset because of load_dotenv()

fix(health): return False if exception is raised in health check

e75be0a

Now that errors in calling _execute_json_request are raised if nothing is returned the health checkers need to be amended

fix(usgs): correct site collection error message

49abe13

It previously called out water levels when it should have called out sites

fix(logger): Ensure only managed handlers are closed

bab5252

Don't close handlers that may be shared with other loggers. Track handlers created by this module and only close those.

fix(usgs): Add parameter code filter to USGS data retrieval if not si…

eafa40f

…tes_only If sites_only is True then do not filter by parameter code, otherwise filter by parameter code 72019 (Groundwater level)

feat(version): Update version to 0.10.1

0c29a8a

fix(usgs): update usgs parameter reader to enable None and number values

3d6d41a

Convert to string so that the strip method when cleaning the data works as expected. This allows for None values to be properly handled and preventserrors when trying to clean numeric values.

jacob-a-brown requested a review from Copilot May 18, 2026 21:39

Copilot started reviewing on behalf of jacob-a-brown May 18, 2026 21:39

View session

Copilot AI reviewed May 18, 2026

View reviewed changes

jacob-a-brown added 3 commits May 18, 2026 15:53

fix(nmbgmr): Check for features in response to determine if API is wo…

5a7c01f

…rking

fix(health): update health checks to return false if get_records retu…

b9ac492

…rns empty response

jacob-a-brown requested a review from Copilot May 18, 2026 22:11

Copilot started reviewing on behalf of jacob-a-brown May 18, 2026 22:11

View session

Copilot AI reviewed May 18, 2026

View reviewed changes

jacob-a-brown added 9 commits May 18, 2026 16:26

fix(st2 health): Update health checks to probe service directly

9560e0b

instead of relying on get_records, which may not work if config is not fully initialized. This should prevent false negatives in health checks while still providing a meaningful check of service availability.

fix(usgs health check): decrease timeout for health check to avoid lo…

e40aa64

…ng waits when USGS is unresponsive

fix(usgs): remove dead code for pagination

1d45571

if pagination is required then an error will be raised and the tool can be updated. no need to maintain dead code for a feature that is not currently used and may never be used.

update(README): added note about USGS API key for die sites

8b20e34

fix(doc): update source _execute_json_request return type to always b…

0c685e1

…e dict

fix(documentation): remove repeated word in comment

074c17f

fix(README): use "for their APIs" instead of "to their APIs"

45916b2

fix(nmenv): list method hangs up service in health check

de660d8

do a health check using get_records instead

fix(readme): indicate api key for site and water level data from usgs

1e3d418

jacob-a-brown merged commit e880976 into main May 19, 2026

jacob-a-brown deleted the jab-usgs-fix branch May 19, 2026 17:47

Conversation

jacob-a-brown commented Apr 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

How

USGS

_execute_json_request tags

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jacob-a-brown commented Apr 29, 2026

Uh oh!

jacob-a-brown commented Apr 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jacob-a-brown commented Apr 29, 2026 •

edited

Loading