Update vignette #779

ldecicco-USGS · 2025-06-04T13:45:45Z

No description provided.

Fix pkgdown See merge request water/dataRetrieval!445

fix NLDI test See merge request water/dataRetrieval!446

…te_vignette

…nto update_vignette

ldecicco-USGS · 2025-06-04T14:10:05Z

@mikemahoney218-usgs @jzemmels @ehinman @cnell-usgs
Here's the in-development rendered doc:
https://water.code-pages.usgs.gov/dataRetrieval/articles/read_USGS_functions.html

Any feedback would be very much appreciated. I'm hoping this document stays in active development, so this is just a first draft to introduce folks to the functions.

…te_vignette

vignettes/read_USGS_functions.Rmd

cnell-usgs · 2025-06-04T15:17:14Z

vignettes/read_USGS_functions.Rmd


+Discrete USGS water quality can be accessed via the `read_USGS_samples` function. While this is a new, modern USGS endpoint, it is not served in the same infrastructure as the rest of these new advertised functions. See [Samples Data](articles/samples_data.html)) for information on accessing USGS discrete water quality data.


Use read_USGS_samples below instead of “read_USGS_samples”

New functions will use a “snake case”, such as “read_USGS_samples”.

Co-authored-by: Cee Nell <cnell@usgs.gov>

jzemmels

Looks great. A few suggested changes.

jzemmels · 2025-06-04T14:55:33Z

vignettes/read_USGS_functions.Rmd

  fig.height = 4,
  fig.width = 7
 )
 ```


-As we bid adieu to the NWIS web services, we welcome a new web service offering, the USGS Water Data OGC APIs.
+As we bid adieu to the NWIS web services, we welcome a host of new web service offering: the [USGS Water Data OGC APIs](https://api.waterdata.usgs.gov/ogcapi/v0/). This is a modern access point for USGS water data. The USGS is planning to modernize **all** web services in the near future. For each of these updates, `dataRetrieval` will create a new function to access the new services. 


Explicitly mention that old services will be decommissioned also.

Suggested change

As we bid adieu to the NWIS web services, we welcome a host of new web service offering: the [USGS Water Data OGC APIs](https://api.waterdata.usgs.gov/ogcapi/v0/). This is a modern access point for USGS water data. The USGS is planning to modernize **all** web services in the near future. For each of these updates, `dataRetrieval` will create a new function to access the new services.

As we bid adieu to the NWIS web services, we welcome a host of new web service offering: the [USGS Water Data OGC APIs](https://api.waterdata.usgs.gov/ogcapi/v0/). This is a modern access point for USGS water data. The USGS is planning to modernize **all** web services in the near future. For each of these updates, `dataRetrieval` will create a new function to access the new services and deprecate functions for accessing the legacy services.

vignettes/read_USGS_functions.Rmd

jzemmels · 2025-06-04T15:19:55Z

vignettes/read_USGS_functions.Rmd

+The returned data includes a column "geometry" which is a collection of simple feature (sf) points. This allows for seamless integration with the `sf` package. Here are 2 quick examples of using the `sf` object in ggplot2 and leaflet:
+
+```{r}
+library(ggplot2)


SC: sf is suggested, so installation may be required for new users.

Suggested change

library(ggplot2)

# install.packages("sf")

library(ggplot2)

We added sf as an Import (before it was just a suggestion), so presumably any user who uses the new version should have it. So, I'm not sure this is a great place for that, but I should update the Dependency section below (when I first started writing this up we were still debating keeping it as suggest vs import.)

The issue I've seen with including commented out install instructions like this is that folks tend to uncomment it and have consistently have fresh installs of that one package (in this case "sf"), and then CRAZY outdated packages everywhere else.

jzemmels · 2025-06-04T15:22:24Z

vignettes/read_USGS_functions.Rmd


-The `read_USGS_monitoring_location` function will eventually replace the `readNWISsite` function. Generally, both functions look pretty similar:
+daily_modern <- read_USGS_daily(monitoring_location_id = "USGS-01491000",
+                          parameter_code = c("00060", "00010"),


SC: add extra whitespace to these lines so function arguments are horizontally aligned.

vignettes/read_USGS_functions.Rmd

Co-authored-by: Joe Zemmels (he/him) <jzemmels@gmail.com>

mikemahoney218-usgs

Super exciting seeing this!

mikemahoney218-usgs · 2025-06-04T14:24:31Z

README.Rmd

@@ -27,7 +27,7 @@ library(dataRetrieval)
 [![](http://cranlogs.r-pkg.org/badges/dataRetrieval)](https://cran.r-project.org/package=dataRetrieval)
 [![](http://cranlogs.r-pkg.org/badges/grand-total/dataRetrieval)](https://cran.r-project.org/package=dataRetrieval)

-The `dataRetrieval` package was created to simplify the process of loading hydrologic data into the R environment. It is designed to retrieve the major data types of U.S. Geological Survey (USGS) hydrology data that are available on the Web, as well as data from the Water Quality Portal (WQP), which currently houses water quality data from the Environmental Protection Agency (EPA), U.S. Department of Agriculture (USDA), and USGS. Direct USGS data is obtained from a service called the National Water Information System (NWIS). 
+The `dataRetrieval` package was created to simplify the process of loading hydrologic data into the R environment. It is designed to retrieve the major data types of U.S. Geological Survey (USGS) hydrology data that are available on the Web, as well as data from the Water Quality Portal (WQP), which currently houses water quality data from the Environmental Protection Agency (EPA), U.S. Department of Agriculture (USDA), and USGS. 


No opinion on whether this sentence is kept or removed, but the new APIs are still part of NWIS -- alongside the rest of the apis on api.waterdata.usgs.gov . We need to think through the branding better, frankly.

mikemahoney218-usgs · 2025-06-04T14:24:56Z

README.Rmd


-2. Get daily USGS discharge data. Start here: `?readNWISdv`
+2. Get daily USGS data (for example, mean daily discharge). Start here: `?read_USGS_daily`


mikemahoney218-usgs · 2025-06-04T14:25:45Z

README.Rmd


-6. Find Hydro Network-Linked Data Index (NLDI) data. Start here: `?findNLDI`
+6. Discover USGS time series data. Start here: `?read_USGS_ts_meta`


This might make more sense as Get metadata about USGS time series data, including instantaneous and daily data, perhaps?

mikemahoney218-usgs · 2025-06-04T14:30:06Z

vignettes/read_USGS_functions.Rmd

  fig.height = 4,
  fig.width = 7
 )
 ```


-As we bid adieu to the NWIS web services, we welcome a new web service offering, the USGS Water Data OGC APIs.
+As we bid adieu to the NWIS web services, we welcome a host of new web service offering: the [USGS Water Data OGC APIs](https://api.waterdata.usgs.gov/ogcapi/v0/). This is a modern access point for USGS water data. The USGS is planning to modernize **all** web services in the near future. For each of these updates, `dataRetrieval` will create a new function to access the new services. 


This spooks me a bit:

The USGS is planning to modernize all web services in the near future

Maybe:

The USGS will be modernizing all of the NWIS web services in the near future

mikemahoney218-usgs · 2025-06-04T15:58:24Z

vignettes/read_USGS_functions.Rmd

 ```
+API_USGS_PAT = "my_super_secret_token"
+```
+You can use `usethis::edit_r_environ()` to edit find and open your .Renviron file. You will need to restart R for that variable to be recognized. 


Maybe mention "you should not add this file to git projects or share your API key, generally. Anyone else using your API key will count against the number of requests available to you!" or something similar about keeping secrets secret

ehinman

This looks and reads very well, @ldecicco-USGS! I especially loved the CQL section. Seems very cool. Made a few minor wording suggestions, plus a question about the term "NWIS".

README.Rmd

ehinman · 2025-06-04T15:40:43Z

vignettes/read_USGS_functions.Rmd

  fig.height = 4,
  fig.width = 7
 )
 ```


-As we bid adieu to the NWIS web services, we welcome a new web service offering, the USGS Water Data OGC APIs.
+As we bid adieu to the NWIS web services, we welcome a host of new web service offering: the [USGS Water Data OGC APIs](https://api.waterdata.usgs.gov/ogcapi/v0/). This is a modern access point for USGS water data. The USGS is planning to modernize **all** web services in the near future. For each of these updates, `dataRetrieval` will create a new function to access the new services. 


Suggested change

As we bid adieu to the NWIS web services, we welcome a host of new web service offering: the [USGS Water Data OGC APIs](https://api.waterdata.usgs.gov/ogcapi/v0/). This is a modern access point for USGS water data. The USGS is planning to modernize **all** web services in the near future. For each of these updates, `dataRetrieval` will create a new function to access the new services.

As we bid adieu to the NWIS web services, we welcome a host of new web service offerings: the [USGS Water Data OGC APIs](https://api.waterdata.usgs.gov/ogcapi/v0/). This is a modern access point for USGS water data. The USGS is planning to modernize **all** web services in the near future. For each of these updates, `dataRetrieval` will create a new function to access the new services.

vignettes/read_USGS_functions.Rmd

ehinman · 2025-06-04T15:48:44Z

vignettes/read_USGS_functions.Rmd

  fig.height = 4,
  fig.width = 7
 )
 ```


-As we bid adieu to the NWIS web services, we welcome a new web service offering, the USGS Water Data OGC APIs.
+As we bid adieu to the NWIS web services, we welcome a host of new web service offering: the [USGS Water Data OGC APIs](https://api.waterdata.usgs.gov/ogcapi/v0/). This is a modern access point for USGS water data. The USGS is planning to modernize **all** web services in the near future. For each of these updates, `dataRetrieval` will create a new function to access the new services. 


May need to have a larger communication conversation about this. @mikemahoney218-usgs please chime in since you're way more in the know....will we be using the term NWIS to describe these services in the future? Are we really saying "goodbye" to NWIS web services, or just modernizing them? If dataRetrieval stops referring to NWIS but it is referred to elsewhere in USGS documentation...is that confusing?

A simple fix might be to replace NWIS with "WaterServices".

NWIS is the broadest possible umbrella containing almost the entire water observing system. Our group owns "NWIS Delivery", which covers WDFN, WQP, WaterAlert, STN, RT-FIM, NGWMN, NWD, comptools, the new APIs, and so on. But NWIS itself also includes AQTS, AQS, MLR, NIMS, TID, DECAP, DIRS, RLMS... there's a few dozen other acronyms included, everything that's required to keep the data flowing. We'll be continuing to call this system NWIS going forward (the replacement for the current NWISMod will be called NWIS). The purpose of api.waterdata.usgs.gov is to be "a catalog of all modernized NWIS APIs" going forward -- that's why it includes now RTFI-API, for instance.

That said, we're moving away from tagging each of the subcomponents as "NWIS" -- rather than "NWISWeb" and "NWISMapper", it's now "WDFN" and "Explore USGS Water Data" and "How to access USGS data in QGIS" and so on.

Maybe that answers the question?

ehinman · 2025-06-04T15:51:59Z

vignettes/read_USGS_functions.Rmd


-`r dataRetrieval:::get_description("daily")`
+* Any column that is returned can also be queried on! The main `dataRetrieval` functions will take vector inputs. Each query is appended as a boolean AND. For instance, if you ask for a vector of monitoring location ids and a vector of parameter codes, you will get data back for just those monitoring locations AND the specified parameter codes. Towards the bottom of this document, we'll discuss ways to make more specific queries using "Common Query Language" or CQL.


I find this statement a little bit confusing, since it seems like the example given here is how you would get specific sites/characteristics from USGS Samples, WQP, any of the services. You could provide a more novel example, like "For instance, you may only want daily observations above a certain value...."

Ah, now re-reading, I see how the "AND" part is unique...whereas with the other services, it would be an "OR"? Maybe here you could specify why this example is different from how Samples or legacy NWIS or the WQP works.

I'll work on a re-write to hopefully clarify. It's not necessarily unique to this service - but it is unique to the way dataRetrieval does it (because the API is expecting the CQL which is way more flexible than a bunch of ANDs)

Co-authored-by: Elise Hinman <121896266+ehinman@users.noreply.github.com>

Co-authored-by: Cee Nell <cnell@usgs.gov>

mikemahoney218-usgs · 2025-06-04T18:59:16Z

vignettes/read_USGS_functions.Rmd


-If you find yourself running into limits, you can request an API token here: <https://api.waterdata.usgs.gov/signup/>
+The new APIs can handle complex requests. For those queries, users will need to construct their own request using Common Query Language. There's an excellent article <https://api.waterdata.usgs.gov/docs/ogcapi/complex-queries/>.


So here's a weird one... we use CQL2. Which OGC always refers to as "Common Query Language" and never calls "version 2". Not sure if it's worth specifying "(CQL2)" after the full name is spelled out?

https://www.ogc.org/standards/cql2/

Merge branch 'update_vignette' of github.com:ldecicco-USGS/dataRetrieval into update_vignette # Conflicts: # vignettes/read_USGS_functions.Rmd

ldecicco-USGS added 11 commits June 3, 2025 08:46

Update readme with new flow

eb3e372

Merge branch 'add_docs' into 'develop'

ce5a007

Fix pkgdown See merge request water/dataRetrieval!445

Merge branch 'add_docs' into 'develop'

9c1db9b

fix NLDI test See merge request water/dataRetrieval!446

make sure this works on gitlab

8150512

Remove bbox from daily

eab31be

Merge branch 'develop' of github.com:DOI-USGS/dataRetrieval into upda…

a5394a7

…te_vignette

Add API key info to readme

5b011c8

add discrete stuff

7871762

Merge branch 'develop' of https://code.usgs.gov/water/dataRetrieval i…

b16744d

…nto update_vignette

put bbox back in daily

3bf91ce

Vignette updates

6eb3ca0

ldecicco-USGS added 2 commits June 4, 2025 09:18

Merge branch 'develop' of github.com:DOI-USGS/dataRetrieval into upda…

70c3719

…te_vignette

Add help section

77b5c23

cnell-usgs reviewed Jun 4, 2025

View reviewed changes

ldecicco-USGS and others added 3 commits June 4, 2025 10:23

Update vignettes/read_USGS_functions.Rmd

aaffa73

Co-authored-by: Cee Nell <cnell@usgs.gov>

Update vignettes/read_USGS_functions.Rmd

2b0ea51

Co-authored-by: Cee Nell <cnell@usgs.gov>

Update vignettes/read_USGS_functions.Rmd

69b5d04

Co-authored-by: Cee Nell <cnell@usgs.gov>

jzemmels reviewed Jun 4, 2025

View reviewed changes

ldecicco-USGS and others added 4 commits June 4, 2025 10:34

Update vignettes/read_USGS_functions.Rmd

095c5d5

Co-authored-by: Joe Zemmels (he/him) <jzemmels@gmail.com>

Update vignettes/read_USGS_functions.Rmd

da75aec

Co-authored-by: Joe Zemmels (he/him) <jzemmels@gmail.com>

Update vignettes/read_USGS_functions.Rmd

f478f2f

Co-authored-by: Joe Zemmels (he/him) <jzemmels@gmail.com>

Update vignettes/read_USGS_functions.Rmd

7bb6683

Co-authored-by: Joe Zemmels (he/him) <jzemmels@gmail.com>

mikemahoney218-usgs reviewed Jun 4, 2025

View reviewed changes

ehinman reviewed Jun 4, 2025

View reviewed changes

ldecicco-USGS and others added 3 commits June 4, 2025 13:41

Update README.Rmd

8943ec3

Co-authored-by: Elise Hinman <121896266+ehinman@users.noreply.github.com>

Update vignettes/read_USGS_functions.Rmd

4c090c0

Co-authored-by: Elise Hinman <121896266+ehinman@users.noreply.github.com>

Update vignettes/read_USGS_functions.Rmd

35c918e

Co-authored-by: Cee Nell <cnell@usgs.gov>

mikemahoney218-usgs reviewed Jun 4, 2025

View reviewed changes

some updates from reviewers

69c7d1b

ldecicco-USGS added 2 commits June 4, 2025 15:32

pull from PR

1c16310

Merge branch 'update_vignette' of github.com:ldecicco-USGS/dataRetrieval into update_vignette # Conflicts: # vignettes/read_USGS_functions.Rmd

vignette updates

4972ee0

ldecicco-USGS merged commit 67d69b2 into DOI-USGS:develop Jun 4, 2025
1 check passed


		Discrete USGS water quality can be accessed via the `read_USGS_samples` function. While this is a new, modern USGS endpoint, it is not served in the same infrastructure as the rest of these new advertised functions. See [Samples Data](articles/samples_data.html)) for information on accessing USGS discrete water quality data.


		2. Get daily USGS discharge data. Start here: `?readNWISdv`
		2. Get daily USGS data (for example, mean daily discharge). Start here: `?read_USGS_daily`


		6. Find Hydro Network-Linked Data Index (NLDI) data. Start here: `?findNLDI`
		6. Discover USGS time series data. Start here: `?read_USGS_ts_meta`

	As we bid adieu to the NWIS web services, we welcome a host of new web service offering: the [USGS Water Data OGC APIs](https://api.waterdata.usgs.gov/ogcapi/v0/). This is a modern access point for USGS water data. The USGS is planning to modernize all web services in the near future. For each of these updates, `dataRetrieval` will create a new function to access the new services.
	As we bid adieu to the NWIS web services, we welcome a host of new web service offerings: the [USGS Water Data OGC APIs](https://api.waterdata.usgs.gov/ogcapi/v0/). This is a modern access point for USGS water data. The USGS is planning to modernize all web services in the near future. For each of these updates, `dataRetrieval` will create a new function to access the new services.


		`r dataRetrieval:::get_description("daily")`
		* Any column that is returned can also be queried on! The main `dataRetrieval` functions will take vector inputs. Each query is appended as a boolean AND. For instance, if you ask for a vector of monitoring location ids and a vector of parameter codes, you will get data back for just those monitoring locations AND the specified parameter codes. Towards the bottom of this document, we'll discuss ways to make more specific queries using "Common Query Language" or CQL.


		If you find yourself running into limits, you can request an API token here: <https://api.waterdata.usgs.gov/signup/>
		The new APIs can handle complex requests. For those queries, users will need to construct their own request using Common Query Language. There's an excellent article <https://api.waterdata.usgs.gov/docs/ogcapi/complex-queries/>.

Update vignette #779

Update vignette #779

Uh oh!

Conversation

ldecicco-USGS commented Jun 4, 2025

Uh oh!

ldecicco-USGS commented Jun 4, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jzemmels left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mikemahoney218-usgs left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ehinman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!