Repository navigation
Develop - #242
Merged
Merged
Develop#242
Conversation
Develop fasterqueries
Fix/doicost
There was a problem hiding this comment.
🟡 Changes recommended
The dataset-type schemas exclude supported values, while ID validation accepts out-of-range integers and silently clears malformed mixed db filters, allowing queries for unrelated datasets.
3 open issues
What changed in this PR
This PR improves Neotoma API request handling and pagination, expands dataset-type filtering, and updates the API documentation and test inputs.
Changes:
- Adds integer validation, standardized HTTP 400 responses, and JSON errors for unsupported non-GET routes.
- Revises site and dataset pagination, download limits, chronology selection, and asynchronous error handling.
- Documents dataset-type lists and pagination behavior, updates test parameters, and removes a legacy SQL script.
| File | Description |
|---|---|
v2.0/routes/data.js |
Registers integer path-parameter validation. |
v2.0/helpers/sites/sites.js |
Adds validation, list filters, and pagination fixes. |
v2.0/helpers/sites/sitequeryfaster.sql |
Paginates distinct sites before collecting results. |
v2.0/helpers/download/download.js |
Updates download limits and chronology ranking. |
v2.0/helpers/datasets/datasets.js |
Improves validation, filtering, and error handling. |
v2.0/helpers/datasets/datasetqueryfaster.sql |
Selects collection-unit pages before aggregation. |
v1.5/helpers/search/updatedExplorerSearch.sql |
Removes the legacy search SQL script. |
test/v2.0-data-taxa-test.js |
Updates taxon query inputs. |
test/v2.0-data-taxa-{taxonid}-test.js |
Changes the taxon ID. |
test/v2.0-data-taxa-{taxonid}-occurrences-test.js |
Changes the taxon ID and pagination. |
test/v2.0-data-summary-rawbymonth-test.js |
Updates the summary interval. |
test/v2.0-data-summary-dstypemonth-test.js |
Updates the summary interval. |
test/v2.0-data-summary-dsdbmonth-test.js |
Updates the summary interval. |
test/v2.0-data-speleothems-{collectionunitid}-test.js |
Changes the collection-unit ID. |
test/v2.0-data-spatial-lakes-test.js |
Updates lake spatial-query inputs. |
test/v2.0-data-spatial-icesheet-test.js |
Updates ice-sheet spatial-query inputs. |
test/v2.0-data-spatial-faunal-test.js |
Updates faunal spatial-query inputs. |
test/v2.0-data-sites-test.js |
Updates site filters and pagination. |
test/v2.0-data-sites-{siteid}-test.js |
Changes the site ID. |
test/v2.0-data-sites-{siteid}-geopoliticalunits-test.js |
Changes the site ID and pagination. |
test/v2.0-data-sites-{siteid}-datasets-test.js |
Changes the site ID. |
test/v2.0-data-sites-{siteid}-datasets_elc-test.js |
Changes the site ID. |
test/v2.0-data-sites-{siteid}-contacts-test.js |
Changes the site ID. |
test/v2.0-data-sites-{siteid}-chronologies-test.js |
Changes the site ID. |
test/v2.0-data-publications-test.js |
Updates publication filters and pagination. |
test/v2.0-data-publications-{publicationid}-test.js |
Changes the publication ID. |
test/v2.0-data-pollen-test.js |
Updates pollen filters and pagination. |
test/v2.0-data-pollen-{id}-test.js |
Changes the pollen request ID. |
test/v2.0-data-occurrences-test.js |
Updates occurrence filters and pagination. |
test/v2.0-data-occurrences-{occurrenceid}-test.js |
Changes the occurrence ID and pagination. |
test/v2.0-data-geopoliticalunits-test.js |
Updates rank and hierarchy options. |
test/v2.0-data-geopoliticalunits-{gpid}-test.js |
Changes the geopolitical-unit ID. |
test/v2.0-data-geopoliticalunits-{gpid}-sites-test.js |
Changes the geopolitical-unit ID and pagination. |
test/v2.0-data-geopoliticalunits-{gpid}-datasets-test.js |
Changes the geopolitical-unit ID and pagination. |
test/v2.0-data-frozen-{datasetid}-test.js |
Changes the dataset ID. |
test/v2.0-data-downloads-{datasetid}-test.js |
Changes the dataset ID. |
test/v2.0-data-dbtables-test.js |
Updates table-query inputs. |
test/v2.0-data-dbtables-{table}-test.js |
Updates the table name and query options. |
test/v2.0-data-datasets-test.js |
Updates dataset filters and pagination. |
test/v2.0-data-datasets-db-test.js |
Updates the database filter and pagination. |
test/v2.0-data-datasets-{datasetid}-test.js |
Changes the dataset ID. |
test/v2.0-data-datasets-{datasetid}-taxa-test.js |
Changes the dataset ID. |
test/v2.0-data-datasets-{datasetid}-sites-test.js |
Changes the dataset ID. |
test/v2.0-data-datasets-{datasetid}-publications-test.js |
Changes the dataset ID. |
test/v2.0-data-datasets-{datasetid}-projects-test.js |
Changes the dataset ID. |
test/v2.0-data-datasets-{datasetid}-lithology-test.js |
Changes the dataset ID. |
test/v2.0-data-datasets-{datasetid}-doi-test.js |
Changes the dataset ID. |
test/v2.0-data-datasets-{datasetid}-contacts-test.js |
Changes the dataset ID. |
test/v2.0-data-datasets-{datasetid}-chronologies-test.js |
Changes the dataset ID. |
test/v2.0-data-datasets-{datasetid}-assays-test.js |
Changes the dataset ID. |
test/v2.0-data-datasets_elc-test.js |
Updates EarthLife Consortium query inputs. |
test/v2.0-data-datasets_elc-{datasetid}-test.js |
Changes the dataset ID. |
test/v2.0-data-contacts-test.js |
Updates contact filters and pagination. |
test/v2.0-data-contacts-{contactid}-test.js |
Changes the contact ID. |
test/v2.0-data-contacts-{contactid}-sites-test.js |
Changes the contact ID. |
test/v2.0-data-contacts-{contactid}-publications-test.js |
Changes the contact ID. |
test/v2.0-data-chronologies-{chronid}-test.js |
Changes the chronology ID. |
test/v2.0-data-aggregatedatasets-{aggdatasetid}-test.js |
Changes the aggregate dataset ID. |
test/v2.0-data-aedna-taxa-{taxonid}-sequences-test.js |
Changes the taxon ID. |
test/v2.0-data-aedna-sequences-{datasetid}-test.js |
Changes the dataset ID. |
test/v2.0-apps-taphonomysystems-test.js |
Changes the dataset-type ID. |
test/v2.0-apps-depenvt-test.js |
Updates depositional-environment query IDs. |
test/v2.0-apps-constdb-datasetuploads-test.js |
Changes the constituent-database ID. |
test/v2.0-apps-constdb-datasets-test.js |
Changes the constituent-database ID. |
test/v1.5-data-sites-{siteid}-test.js |
Changes the legacy site ID. |
test/v1.5-data-occurrence-{occurrenceid}-test.js |
Changes the legacy occurrence ID. |
test/v1.5-data-geopoliticalunits-test.js |
Updates legacy hierarchy-query options. |
test/v1.5-data-geopoliticalunits-{gpid}-test.js |
Changes the legacy geopolitical-unit ID. |
test/v1.5-data-downloads-{datasetid}-test.js |
Changes the legacy download dataset ID. |
test/v1.5-data-datasets-{datasetid}-test.js |
Changes the legacy dataset ID. |
test/v1.5-data-contacts-{contactid}-test.js |
Changes the legacy contact ID. |
src/neotomaapi.js |
Adds validation helpers and revises list parsing. |
openapi/paths/v20/data.yml |
References list parameters for sites and datasets. |
openapi/components/parameters/parameters.yml |
Defines list parameters and documents limits. |
openapi.yaml |
Updates the bundled API specification. |
app.js |
Returns JSON 404 responses for unsupported non-GET requests. |
Review effort: Balanced
Comment on lines
+316
to
+318
| example: | ||
| - pollen | ||
| - diatom |
| */ | ||
| function allIntegers(ids) { | ||
| return ids === null || typeof ids === 'undefined' || | ||
| (Array.isArray(ids) && ids.every(Number.isInteger)); |
Comment on lines
+383
to
+385
| if (!allIntegers(outobj.siteid) || !allIntegers(outobj.datasetid)) { | ||
| return badRequest(res, 'siteid and datasetid must be integers or ' + | ||
| 'comma separated lists of integers.'); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

This pull request improves input validation, enhances API documentation for query parameters, and updates test data to ensure more robust and predictable API behavior. The most significant changes are grouped below.
API Input Validation and Error Handling:
commaSepfunction insrc/neotomaapi.jsto better handle trailing or doubled commas and ensure only valid integer lists are parsed, preventing database errors from malformed input.allIntegers,badRequest, andintegerParamutility functions to validate integer list parameters and return standardized HTTP 400 errors for invalid input, preventing internal server errors and improving API reliability. [1] [2]app.jsto return a 404 JSON error for non-GET requests to/v2.0routes without handlers, instead of redirecting to HTML documentation, which avoids JSON parsing failures in API clients.API Documentation and OpenAPI Specification:
datasettypeQueryandlimitParamparameters in OpenAPI specs (openapi.yaml,parameters.yml), detailing how single and multiple dataset types are handled and how limits work for various endpoints. [1] [2] [3] [4]datasettypeListQueryOpenAPI parameter for endpoints that accept multiple dataset types, and updated relevant routes in OpenAPI and YAML path files to use it. [1] [2] [3] [4] [5] [6]Test Data Updates: