Skip to content

Develop - #242

Merged
sedv8808 merged 9 commits into
productionfrom
develop
Nov 2, 2026
Merged

sedv8808 merged 9 commits into
productionfrom
develop

Conversation

@sedv8808

@sedv8808 sedv8808 commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

This pull request improves input validation, enhances API documentation for query parameters, and updates test data to ensure more robust and predictable API behavior. The most significant changes are grouped below.

API Input Validation and Error Handling:

  • Improved the commaSep function in src/neotomaapi.js to better handle trailing or doubled commas and ensure only valid integer lists are parsed, preventing database errors from malformed input.
  • Added allIntegers, badRequest, and integerParam utility functions to validate integer list parameters and return standardized HTTP 400 errors for invalid input, preventing internal server errors and improving API reliability. [1] [2]
  • Updated the catch-all route in app.js to return a 404 JSON error for non-GET requests to /v2.0 routes without handlers, instead of redirecting to HTML documentation, which avoids JSON parsing failures in API clients.

API Documentation and OpenAPI Specification:

  • Clarified and extended descriptions for datasettypeQuery and limitParam parameters in OpenAPI specs (openapi.yaml, parameters.yml), detailing how single and multiple dataset types are handled and how limits work for various endpoints. [1] [2] [3] [4]
  • Added a new datasettypeListQuery OpenAPI parameter for endpoints that accept multiple dataset types, and updated relevant routes in OpenAPI and YAML path files to use it. [1] [2] [3] [4] [5] [6]

Test Data Updates:

  • Updated various test files to use current or valid example IDs and query parameters, ensuring tests are meaningful and do not fail due to outdated data. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11]

Copilot AI balanced review requested due to automatic review settings November 2, 2026 21:40

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The dataset-type schemas exclude supported values, while ID validation accepts out-of-range integers and silently clears malformed mixed db filters, allowing queries for unrelated datasets.

3 open issues
What changed in this PR

This PR improves Neotoma API request handling and pagination, expands dataset-type filtering, and updates the API documentation and test inputs.

Changes:

  • Adds integer validation, standardized HTTP 400 responses, and JSON errors for unsupported non-GET routes.
  • Revises site and dataset pagination, download limits, chronology selection, and asynchronous error handling.
  • Documents dataset-type lists and pagination behavior, updates test parameters, and removes a legacy SQL script.
File Description
v2.0/​routes/​data.js Registers integer path-parameter validation.
v2.0/​helpers/​sites/​sites.js Adds validation, list filters, and pagination fixes.
v2.0/​helpers/​sites/​sitequeryfaster.sql Paginates distinct sites before collecting results.
v2.0/​helpers/​download/​download.js Updates download limits and chronology ranking.
v2.0/​helpers/​datasets/​datasets.js Improves validation, filtering, and error handling.
v2.0/​helpers/​datasets/​datasetqueryfaster.sql Selects collection-unit pages before aggregation.
v1.5/​helpers/​search/​updatedExplorerSearch.sql Removes the legacy search SQL script.
test/​v2.0-data-taxa-test.js Updates taxon query inputs.
test/​v2.0-data-taxa-{taxonid}-test.js Changes the taxon ID.
test/​v2.0-data-taxa-{taxonid}-occurrences-test.js Changes the taxon ID and pagination.
test/​v2.0-data-summary-rawbymonth-test.js Updates the summary interval.
test/​v2.0-data-summary-dstypemonth-test.js Updates the summary interval.
test/​v2.0-data-summary-dsdbmonth-test.js Updates the summary interval.
test/​v2.0-data-speleothems-{collectionunitid}-test.js Changes the collection-unit ID.
test/​v2.0-data-spatial-lakes-test.js Updates lake spatial-query inputs.
test/​v2.0-data-spatial-icesheet-test.js Updates ice-sheet spatial-query inputs.
test/​v2.0-data-spatial-faunal-test.js Updates faunal spatial-query inputs.
test/​v2.0-data-sites-test.js Updates site filters and pagination.
test/​v2.0-data-sites-{siteid}-test.js Changes the site ID.
test/​v2.0-data-sites-{siteid}-geopoliticalunits-test.js Changes the site ID and pagination.
test/​v2.0-data-sites-{siteid}-datasets-test.js Changes the site ID.
test/​v2.0-data-sites-{siteid}-datasets_elc-test.js Changes the site ID.
test/​v2.0-data-sites-{siteid}-contacts-test.js Changes the site ID.
test/​v2.0-data-sites-{siteid}-chronologies-test.js Changes the site ID.
test/​v2.0-data-publications-test.js Updates publication filters and pagination.
test/​v2.0-data-publications-{publicationid}-test.js Changes the publication ID.
test/​v2.0-data-pollen-test.js Updates pollen filters and pagination.
test/​v2.0-data-pollen-{id}-test.js Changes the pollen request ID.
test/​v2.0-data-occurrences-test.js Updates occurrence filters and pagination.
test/​v2.0-data-occurrences-{occurrenceid}-test.js Changes the occurrence ID and pagination.
test/​v2.0-data-geopoliticalunits-test.js Updates rank and hierarchy options.
test/​v2.0-data-geopoliticalunits-{gpid}-test.js Changes the geopolitical-unit ID.
test/​v2.0-data-geopoliticalunits-{gpid}-sites-test.js Changes the geopolitical-unit ID and pagination.
test/​v2.0-data-geopoliticalunits-{gpid}-datasets-test.js Changes the geopolitical-unit ID and pagination.
test/​v2.0-data-frozen-{datasetid}-test.js Changes the dataset ID.
test/​v2.0-data-downloads-{datasetid}-test.js Changes the dataset ID.
test/​v2.0-data-dbtables-test.js Updates table-query inputs.
test/​v2.0-data-dbtables-{table}-test.js Updates the table name and query options.
test/​v2.0-data-datasets-test.js Updates dataset filters and pagination.
test/​v2.0-data-datasets-db-test.js Updates the database filter and pagination.
test/​v2.0-data-datasets-{datasetid}-test.js Changes the dataset ID.
test/​v2.0-data-datasets-{datasetid}-taxa-test.js Changes the dataset ID.
test/​v2.0-data-datasets-{datasetid}-sites-test.js Changes the dataset ID.
test/​v2.0-data-datasets-{datasetid}-publications-test.js Changes the dataset ID.
test/​v2.0-data-datasets-{datasetid}-projects-test.js Changes the dataset ID.
test/​v2.0-data-datasets-{datasetid}-lithology-test.js Changes the dataset ID.
test/​v2.0-data-datasets-{datasetid}-doi-test.js Changes the dataset ID.
test/​v2.0-data-datasets-{datasetid}-contacts-test.js Changes the dataset ID.
test/​v2.0-data-datasets-{datasetid}-chronologies-test.js Changes the dataset ID.
test/​v2.0-data-datasets-{datasetid}-assays-test.js Changes the dataset ID.
test/​v2.0-data-datasets_elc-test.js Updates EarthLife Consortium query inputs.
test/​v2.0-data-datasets_elc-{datasetid}-test.js Changes the dataset ID.
test/​v2.0-data-contacts-test.js Updates contact filters and pagination.
test/​v2.0-data-contacts-{contactid}-test.js Changes the contact ID.
test/​v2.0-data-contacts-{contactid}-sites-test.js Changes the contact ID.
test/​v2.0-data-contacts-{contactid}-publications-test.js Changes the contact ID.
test/​v2.0-data-chronologies-{chronid}-test.js Changes the chronology ID.
test/​v2.0-data-aggregatedatasets-{aggdatasetid}-test.js Changes the aggregate dataset ID.
test/​v2.0-data-aedna-taxa-{taxonid}-sequences-test.js Changes the taxon ID.
test/​v2.0-data-aedna-sequences-{datasetid}-test.js Changes the dataset ID.
test/​v2.0-apps-taphonomysystems-test.js Changes the dataset-type ID.
test/​v2.0-apps-depenvt-test.js Updates depositional-environment query IDs.
test/​v2.0-apps-constdb-datasetuploads-test.js Changes the constituent-database ID.
test/​v2.0-apps-constdb-datasets-test.js Changes the constituent-database ID.
test/​v1.5-data-sites-{siteid}-test.js Changes the legacy site ID.
test/​v1.5-data-occurrence-{occurrenceid}-test.js Changes the legacy occurrence ID.
test/​v1.5-data-geopoliticalunits-test.js Updates legacy hierarchy-query options.
test/​v1.5-data-geopoliticalunits-{gpid}-test.js Changes the legacy geopolitical-unit ID.
test/​v1.5-data-downloads-{datasetid}-test.js Changes the legacy download dataset ID.
test/​v1.5-data-datasets-{datasetid}-test.js Changes the legacy dataset ID.
test/​v1.5-data-contacts-{contactid}-test.js Changes the legacy contact ID.
src/​neotomaapi.js Adds validation helpers and revises list parsing.
openapi/​paths/​v20/​data.yml References list parameters for sites and datasets.
openapi/​components/​parameters/​parameters.yml Defines list parameters and documents limits.
openapi.yaml Updates the bundled API specification.
app.js Returns JSON 404 responses for unsupported non-GET requests.

Review effort: Balanced

Comment on lines +316 to +318
example:
- pollen
- diatom
Comment thread src/neotomaapi.js
*/
function allIntegers(ids) {
return ids === null || typeof ids === 'undefined' ||
(Array.isArray(ids) && ids.every(Number.isInteger));
Comment on lines +383 to +385
if (!allIntegers(outobj.siteid) || !allIntegers(outobj.datasetid)) {
return badRequest(res, 'siteid and datasetid must be integers or ' +
'comma separated lists of integers.');
@sedv8808
sedv8808 merged commit 0932913 into production Nov 2, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants