Resource Watch API Reference

This section of the documentation contains technical reference information for each of the endpoints exposed by the RW API.

Back to the documentation homepage

Authentication

JWT tokens

Most RW API endpoints are public, but some actions require user authentication. Each endpoint's documentation will specify whether authentication is required and under what conditions.

The RW API uses JSON Web Tokens to authenticate requests. Check out the Quickstart Guide for instructions on how to create an account and generate a token.

Authentication is performed using JWT tokens. For all authenticated requests, you must provide your JWT using the header Authorization: Bearer <your token>. Please note that your JWT tokens act like passwords, so you should never share or expose them publicly. Doing so may allow other users to impersonate your RW API user account, and may grant them permissions over your datasets, applications, etc.

If you lose your JWT, or if it becomes invalidated (which happens whenever the name, email, or applications associated with your account changes), you can always log in via the browser and generate a new token at https://api.resourcewatch.org/auth/generate-token.

API Keys

Besides JWT tokens, the RW API also uses API keys to identify the client application issuing the request. This is done for moderation and analytical purposes, ensuring it's available to everyone for fair use. Unlike the JWT tokens described above, API keys are not used to grant permissions over data. While it's not critical to keep them private, you should have a different API key for each application you create that uses the RW API.

An API key is generated automatically once you register a client application, which you can learn more about in the corresponding section of the reference docs.

Dataset

What is a dataset?

If you are new to the RW API, or want to learn more about the concept of a dataset, we strongly encourage you to read the dataset concept documentation first. It gives you a brief and clear description of what a dataset is, and what it can do for you.

Once you've read that section, you can come back here to learn more details about using the RW API's datasets feature. The RW API is home to many datasets uploaded by WRI, its partner organizations, or by API users like you. These datasets contain a lot of information about a wide range of topics that you may want to learn about or build on top of. To find out more about finding and accessing the datasets already available on the RW API, check out the documentation on getting datasets. A nice, visual way to explore existing datasets is by using the Resource Watch website.

You can also create you own datasets on the RW API, if you'd like to share your data with the world, or if you are looking to use the RW API and its features to gain insights into your data.

Getting all datasets

This endpoint will allow you to get the list of the datasets available in the API, and it's a great place for new users to start exploring the RW API. By default, this endpoint will give you a paginated list of 10 datasets. In the sections below, we'll explore how you can customize this endpoint call to match your needs.

For a detailed description of each field, check out the Dataset reference section.

Getting a list of datasets

curl -X GET https://api.resourcewatch.org/v1/dataset \
-H "x-api-key: <your-api-key>"

Response:

{
  "data": [
    {
      "id": "00f2be42-1ee8-4069-a55a-16a988f2b7a0",
      "type": "dataset",
      "attributes": {
        "name": "Glad points",
        "slug": "Glad-points-1490086842129",
        "type": null,
        "subtitle": null,
        "application": [
          "data4sdgs"
        ],
        "dataPath": null,
        "attributesPath": null,
        "connectorType": "document",
        "provider": "csv",
        "userId": "58333dcfd9f39b189ca44c75",
        "connectorUrl": "https://gfw2-data.s3.amazonaws.com/alerts-tsv/glad_headers.csv",
        "sources": [],
        "tableName": "data",
        "status": "pending",
        "published": true,
        "overwrite": false,
        "env": "production",
        "geoInfo": false,
        "legend": {
          "date": [],
          "region": [],
          "country": []
        },
        "clonedHost": {},
        "errorMessage": null,
        "updatedAt": "2017-01-13T10:45:46.368Z",
        "widgetRelevantProps": [],
        "layerRelevantProps": []
      }
    }
  ],
  "links": {
    "self": "https://api.resourcewatch.org/v1/dataset?page[number]=1&page[size]=10",
    "first": "https://api.resourcewatch.org/v1/dataset?page[number]=1&page[size]=10",
    "last": "https://api.resourcewatch.org/v1/dataset?page[number]=99&page[size]=10",
    "prev": "https://api.resourcewatch.org/v1/dataset?page[number]=1&page[size]=10",
    "next": "https://api.resourcewatch.org/v1/dataset?page[number]=2&page[size]=10"
  },
  "meta": {
    "total-pages": 99,
    "total-items": 990,
    "size": 10
  }
}

Pagination

Example request to load page 2 using 25 results per page

curl -X GET https://api.resourcewatch.org/v1/dataset?page[number]=2&page[size]=25 \
-H "x-api-key: <your-api-key>"

The Dataset service adheres to the conventions defined in the Pagination guidelines for the RW API, so we recommend reading that section for more details on how paginate your datasets list.

Search

Search for datasets

curl -X GET https://api.resourcewatch.org/v1/dataset?search=wind \
-H "x-api-key: <your-api-key>"

The dataset service offers a simple yet powerful search mechanism that will help you look for datasets. The query argument search allows you to specify a search string that will match:

• The dataset name.
• The dataset's metadata name.
• The dataset's metadata description.
• The relational graph node list.

Filters

Filtering datasets

curl -X GET https://api.resourcewatch.org/v1/dataset?name=birds&provider=cartodb \
-H "x-api-key: <your-api-key>"

Matching vocabulary tags

curl -X GET https://api.resourcewatch.org/v1/dataset?vocabulary[legacy]=umd \
-H "x-api-key: <your-api-key>"

The dataset list provides a wide range of parameters that you can use to tailor your dataset listing. Most of these parameters reflect fields you'll find in a dataset itself (which you can learn more about in the Dataset reference section), while others are convenience filters for things like user role or favourites.

Filtering datasets adheres to the conventions defined in the Filter guidelines for the RW API , so we strongly recommend reading that section before proceeding. In addition to these conventions, you can use the following fields as filters supported by the dataset list endpoint:

Sorting

Basics of sorting

Sorting datasets

curl -X GET https://api.resourcewatch.org/v1/dataset?sort=name \
-H "x-api-key: <your-api-key>"

Sorting datasets by multiple criteria

curl -X GET https://api.resourcewatch.org/v1/dataset?sort=name,description \
-H "x-api-key: <your-api-key>"

Sort by name descending, description ascending

curl -X GET https://api.resourcewatch.org/v1/dataset?sort=-name,+description \
-H "x-api-key: <your-api-key>"

Sorting datasets by the role of the user who owns the dataset

curl -X GET https://api.resourcewatch.org/v1/dataset?sort=user.role \
-H "x-api-key: <your-api-key>"

The Dataset service currently supports sorting using the sort query parameter. Sorting dataset adheres to the conventions defined in the Sorting guidelines for the RW API, so we strongly recommend reading that section before proceeding. Additionally, you can check out the Dataset reference section for a detailed description of the fields you can use when sorting. In addition to all dataset model fields, you can sort the returned results by the name (using user.name) or role (using user.role) of the user owner of the dataset. Keep in mind that sorting by user data is restricted to ADMIN users.

Please also keep in mind that, due to the limitations of the underlying endpoint used to find user ids by name or role, the performance of the request while using this sort might be degraded.

Special sorting criteria

Sorting datasets with special criteria

curl -X GET https://api.resourcewatch.org/v1/dataset?sort=-most-favorited \
-H "x-api-key: <your-api-key>"

curl -X GET https://api.resourcewatch.org/v1/dataset?sort=relevance&status=saved&search=agriculture \
-H "x-api-key: <your-api-key>"

Special search criteria must be used as sole sorting criteria, as it's not possible to combine any of them with any other search criteria. The following criteria can be used for special sorting in datasets:

• most-viewed delegates sorting to the graph component, sorting by the datasets that have been queried more frequently. Supports ascending/descending order.
• most-favorited: delegates sorting to the graph component, sorting by the datasets that have been more favorited. Supports ascending/descending order.
• relevance: delegates sorting to the metadata component, sorting by the datasets which metadata better match the search criteria. Can only be used in conjunction with a search parameter. Does not support ascending order.

You can specify the sorting order by prepending the criteria with either - for descending order or + for ascending order. By default, ascending order is assumed.

Include entities associated with the datasets

Loads metadata and widgets associated with each dataset:

curl -X GET https://api.resourcewatch.org/v1/dataset?includes=metadata,widget \
-H "x-api-key: <your-api-key>"

{
  "data": [
    {
      "id": "06c44f9a-aae7-401e-874c-de13b7764959",
      "type": "dataset",
      "attributes": {
        "name": "Historical Precipitation -- U.S. (Puget Sound Lowlands)",
        "slug": "Historical-Precipitation-US-Puget-Sound-Lowlands-1490086842133",
        ...
        "metadata": [
          {
            "id": "57a2251d5cd6980a00e054d9",
            "type": "metadata",
            "attributes": {
              ...
            }
          }
        ],
        "widget": [
          {
            "id": "73f00267-fe34-42aa-a611-13b102f38d75",
            "type": "widget",
            "attributes": {
              ...
            }
          }
        ]
      }
    }
  ],
  "links": {
    "self": "https://api.resourcewatch.org/v1/dataset?includes=widget%2Cmetadata&page[number]=1&page[size]=10",
    "first": "https://api.resourcewatch.org/v1/dataset?includes=widget%2Cmetadata&page[number]=1&page[size]=10",
    "last": "https://api.resourcewatch.org/v1/dataset?includes=widget%2Cmetadata&page[number]=223&page[size]=10",
    "prev": "https://api.resourcewatch.org/v1/dataset?includes=widget%2Cmetadata&page[number]=1&page[size]=10",
    "next": "https://api.resourcewatch.org/v1/dataset?includes=widget%2Cmetadata&page[number]=2&page[size]=10"
  },
  "meta": {
    "total-pages": 223,
    "total-items": 2228,
    "size": 10
  }
}

Loading layers for the given dataset

curl -X GET https://api.resourcewatch.org/dataset?includes=layer \
-H "x-api-key: <your-api-key>"

Loading the information about the user who authored the dataset

curl -X GET https://api.resourcewatch.org/v1/dataset?includes=user \
-H "x-api-key: <your-api-key>"

When fetching datasets, you can request additional entities to be loaded. The following entities are available:

• widget - loads all widgets associated with each dataset.
• layer - loads all layers associated with each dataset.
• vocabulary - loads all vocabulary entities associated with each dataset.
• metadata - loads all metadata associated with each dataset.
• user - loads the name, email address and role of the author of the dataset. If you do not issue this request as an ADMIN user, or if no user data is available, the user object will be empty. Please keep in mind that, due to the limitations of the underlying endpoint used to find users by ids, the performance of the request while including user information might be degraded.

If you specified an env filter and would like to also filter the included entities, you can provide the filterIncludesByEnv=true query parameter. This is only supported for included widget and layer.

Note: If you include related entities (e.g. layers) with query filters, the filters will not cascade to the related entities.

Getting a dataset by id or slug

Getting a dataset by id:

curl -X GET "https://api.resourcewatch.org/v1/dataset/51943691-eebc-4cb4-bdfb-057ad4fc2145" \
-H "x-api-key: <your-api-key>"

Getting a dataset by slug:

curl -X GET "https://api.resourcewatch.org/v1/dataset/Timber-Production-RDC" \
-H "x-api-key: <your-api-key>"

Response (equal for both cases):

{
    "data": {
        "id": "51943691-eebc-4cb4-bdfb-057ad4fc2145",
        "type": "dataset",
        "attributes": {
            "name": "Timber Production RDC",
            "slug": "Timber-Production-RDC",
            "type": null,
            "subtitle": null,
            "application": ["forest-atlas"],
            "dataPath": null,
            "attributesPath": null,
            "connectorType": "document",
            "provider": "csv",
            "userId": "58750a56dfc643722bdd02ab",
            "connectorUrl": "https://wri-forest-atlas.s3.amazonaws.com/COD/temp/annual%20timber%20production%20DRC%20%28test%29%20-%20Sheet1.csv",
            "sources": [],
            "tableName": "index_51943691eebc4cb4bdfb057ad4fc2145",
            "status": "saved",
            "overwrite": false,
            "legend": {
                "date": ["year"],
                "region": [],
                "country": [],
                "long": "",
                "lat": ""
            },
            "clonedHost": {},
            "errorMessage": null,
            "createdAt": "2017-01-25T21:48:27.535Z",
            "updatedAt": "2017-01-25T21:48:28.675Z"
        }
    }
}

If you know the id or the slug of a dataset, then you can access it directly. Both id and slug are case-sensitive.

Using this endpoint, you can also include entities associated with the dataset, in the same way you do when loading multiple datasets.

Getting a dataset by its including its relationships:

curl -X GET https://api.resourcewatch.org/v1/dataset/06c44f9a-aae7-401e-874c-de13b7764959?includes=metadata,vocabulary,widget,layer \
-H "x-api-key: <your-api-key>"

Getting multiple datasets by ids

This endpoint allows you to load multiple datasets by id in a single request.

Errors for getting multiple datasets by ids

Getting multiple datasets by their ids:

curl -X POST https://api.resourcewatch.org/v1/dataset/find-by-ids \
-H "Content-Type: application/json"  \
-H "x-api-key: <your-api-key>" -d \
'{
    "ids": [
        "0706f039-b929-453e-b154-7392123ae99e",
        "0c630feb-8146-4fcc-a9be-be5adcb731c8",
    ]
}'

Response

    {
  "data": [
    {
      "id": "0706f039-b929-453e-b154-7392123ae99e",
      "type": "dataset",
      "attributes": {
        "name": "Social Vulnerability Index (2006-2010) -- U.S. (New Hampshire)",
        "slug": "Social-Vulnerability-Index-2006-2010-US-New-Hampshire-1490086842541",
        "type": "tabular",
        "subtitle": null,
        ...
      }
    },
    {
      "id": "0c630feb-8146-4fcc-a9be-be5adcb731c8",
      "type": "dataset",
      "attributes": {
        "name": "USGS Land Cover - Impervious Surface (2001) -- U.S. (Alaska)",
        "slug": "USGS-Land-Cover-Impervious-Surface-2001-US-Alaska-1490086842439",
        "type": "raster",
        "subtitle": null,
        ...
      }
    }
  ]
}

Creating a dataset

So you are ready to create your first dataset on the RW API? Great, welcome aboard! These instruction will help you create your dataset on the RW API, so you can start sharing your data with the world and taking full advantage of all the RW API features on your projects.

Before creating a dataset, there are a few things you must know and do:

• In order to be able to create a dataset, you need to be authenticated.
• Depending on your user account's role, you may have permission to create a dataset but not delete it afterwards - read more about this in the RW API role-based access control guidelines.
• All data uploaded to the RW API will be publicly visible and available to other users.

The first thing to consider when creating a dataset is where your data currently is - this will determine the Dataset provider you will need to use and, with it, a set of things you need to take into account - we'll cover each provider in detail shortly. When building your request to the RW API to create your dataset, you will need to take into account both the general details that follow, plus the details for the provideryou are using. Be sure to review both sections when creating your datasets, to avoid any pitfalls.

Creating a dataset is done using a POST request and passing the relevant data as body files. The supported body fields are as defined on the dataset reference section, but the minimum field list you must specify for all datasets is:

• name
• application
• connectorType
• provider

Additionally, depending on the data provider you will be using, other fields may be required - we'll cover those in detail in each provider's specific documentation.

A successful dataset creation request will return a 200 HTTP code, and the dataset details as stored on the RW API. PAy special attention to the id or the slug, as those will allow you to access your dataset later.

There's one aspect of the dataset creation process that you need to keep in mind: it is an asynchronous process.This means that a successful call to the create dataset endpoint will start the process, but the dataset may not be immediately available to be used. The status field will tell you if the dataset creation process is in progress ( status set to pending), if something went wrong (failed, in which case the errorMessage field will have a short description of what went wrong) or if the dataset is available to be used (when status is saved). The amount of time it takes for a newly created dataset to go from pending to saved depends on the provider and amount of data. Just keep in mind that you need to wait for the status to be set to saved before starting to use your datasets. You should check your dataset manually to see when the status is updated.

Errors for creating a dataset

Carto datasets

Example of creating a dataset based on CartoDB data with the minimum fields required

curl -X POST https://api.resourcewatch.org/v1/dataset \
-H "Authorization: Bearer <your-token>" \
-H "x-api-key: <your-api-key>" \
-H "Content-Type: application/json"  -d \
'{
  "dataset": {
    "name":"World Database on Protected Areas -- Global",
    "connectorType":"rest",
    "provider":"cartodb",
    "connectorUrl":"https://wri-01.carto.com/tables/wdpa_protected_areas/table",
    "application":[
      "gfw", 
      "forest-atlas"
    ]
  }
}'

To create a dataset connected to a Carto data source, besides the common required fields, you must provide the following required data:

The RW API will use the information above to directly query the Carto account specified on the connectorUrl field whenever this dataset is accessed on the RW API. This has a few implications that you should be aware of:

• The Carto URL provided will be publicly visible to all RW API users.
• The Carto account will see increased traffic.
• Any changes made to the data hosted on Carto will be automatically reflected on the data served by the RW API for this dataset.
• If you restructure or delete your Carto table, the corresponding RW API dataset will be in an invalid state, and you should delete it.

The tableName value of a carto-based dataset will automatically be filled with the name of the carto table corresponding to the dataset.

When creating a Carto based-dataset, the RW API will try to validate the connectorUrl by trying to connect to the corresponding Carto table - the result of this will determine if the dataset's status will be set to saved or error.

GFW datasets

Example of creating a dataset based on GFW data with the minimum fields required

curl -X POST https://api.resourcewatch.org/v1/dataset \
-H "Authorization: Bearer <your-token>" \
-H "x-api-key: <your-api-key>" \
-H "Content-Type: application/json"  -d \
'{
  "dataset": {
    "name":"RADD Deforestation Alerts",
    "connectorType":"rest",
    "provider":"gfw",
    "connectorUrl":"https://data-api.globalforestwatch.org/dataset/wur_radd_alerts/latest",
    "application":[
      "gfw", 
      "rw"
    ]
  }
}'

To create a dataset connected to a GFW data source, besides the common required fields, you must provide the following required data:

The RW API will use the information above to directly query the GFW dataset specified on the connectorUrl field whenever this dataset is accessed on the RW API. This has a few implications that you should be aware of:

• The GFW URL provided will be publicly visible to all RW API users.
• The GFW REST API will see increased traffic.
• Any changes made to the data hosted on GFW will be automatically reflected on the data served by the RW API for this dataset.
• If you restructure or delete your GFW table or raster file, the corresponding RW API dataset will be in an invalid state, and you should delete it.

The tableName value of a GFW-based dataset will automatically be filled with the name of the GFW table or raster file corresponding to the dataset.

The latest path parameter in connectionUrl requests the most recent public version of the requested dataset in GFW. Alternatively, you may request a dataset by the version number if known.

When creating a GFW-based dataset, the RW API will try to validate the connectorUrl by trying to connect to the corresponding GFW table or raster file - the result of this will determine if the dataset's status will be set to saved or error.

ArcGIS feature layer

Example of creating a dataset based on ArcGIS feature layer data with the minimum fields required

curl -X POST https://api.resourcewatch.org/v1/dataset \
-H "Authorization: Bearer <your-token>" \
-H "x-api-key: <your-api-key>" \
-H "Content-Type: application/json"  -d \
'{
  "dataset": {
    "connectorType":"rest",
    "provider":"featureservice",
    "connectorUrl":"https://services.arcgis.com/uDTUpUPbk8X8mXwl/arcgis/rest/services/Public_Schools_in_Onondaga_County/FeatureServer/0?f=json",
    "application":[
     "prep"
    ],
    "name":"Uncontrolled Public-Use Airports -- U.S."
  }
}'

To create a dataset using ArcGIS feature layer as data source, besides the common required fields, you must provide the following required data:

The RW API will use the information above to directly query the ArcGIS feature layer server specified on the connectorUrl field whenever this dataset is accessed on the RW API. This has a few implications that you should be aware of:

• The ArcGIS feature layer URL provided will be publicly visible to all RW API users.
• The ArcGIS feature layer server will see increased traffic.
• Any changes made to the data hosted on ArcGIS feature layer will be automatically reflected on the data served by the RW API for this dataset.
• If you restructure or delete your ArcGIS feature layer dataset, the corresponding RW API dataset will be in an invalid state, and you should delete it.

The tableName value of a ArcGIS-based dataset will automatically be filled with the name of the ArcGIS feature layer table corresponding to the dataset.

When creating a ArcGIS-based dataset, the RW API will try to validate the connectorUrl by trying to connect to the corresponding ArcGIS Feature Layer - the result of this will determine if the dataset's status will be set to saved or error.

Google Earth Engine

Example of creating a dataset based on Google Earth Engine data with the minimum fields required

curl -X POST https://api.resourcewatch.org/v1/dataset \
-H "Authorization: Bearer <your-token>" \
-H "x-api-key: <your-api-key>" \
-H "Content-Type: application/json"  -d \
'{
  "dataset": {
    "connectorType":"rest",
    "provider":"gee",
    "tableName": "JRC/GSW1_0/GlobalSurfaceWater"
    "application":[
     "rw"
    ],
    "name":"Water occurrence"
  }
}'

To create a dataset using Google Earth Engine (GEE) as data source, besides the common required fields, you must provide the following required data:

The RW API will use the information above to directly query the GEE dataset specified on the tableName field whenever this dataset is accessed on the RW API. This has a few implications that you should be aware of:

• The GEE resource name provided will be publicly visible to all RW API users.
• Any changes made to the data hosted on GEE will be automatically reflected on the data served by the RW API for this dataset.
• If you restructure or delete your GEE dataset, the corresponding RW API dataset will be in an invalid state, and you should delete it.

When creating a GEE-based dataset, the RW API will try to validate it by connecting to the corresponding dataset GEE - the result of this will determine if the dataset's status will be set to saved or error.

WMS

Example of creating a dataset based on WMS data with the minimum fields required

curl -X POST 'https://api.resourcewatch.org/v1/dataset' -d \
-H 'Authorization: Bearer <your-token>'  \
-H "x-api-key: <your-api-key>" \
-H 'Content-Type: application/json' -d \
'{
   "dataset": {
     "application": [
       "rw"
     ],
     "name": "Seasonal variability",
     "connectorType": "wms",
     "provider":"wms",
     "connectorUrl":"https://gis-gfw.wri.org/arcgis/rest/services/prep/nex_gddp_indicators/MapServer/6?f=pjson"
   }
 }
'

To create a dataset using WMS as data source, you should provide the following data:

The RW API will use the information above to directly query the WMS dataset specified on the connectorUrl field whenever this dataset is accessed on the RW API. This has a few implications that you should be aware of:

• The WMS server URL provided will be publicly visible to all RW API users.
• Any changes made to the data hosted on the WMS server data will be automatically reflected on the data served by the RW API for this dataset.
• If you restructure or delete your WMS server data, the corresponding RW API dataset will be in an invalid state, and you should delete it.

When creating a WMS-based dataset, no validation is done - the dataset is automatically created in a saved state.

Document based datasets: JSON, CSV, TSV or XML

Creating a CSV dataset with data provided by externally hosted files:

curl -X POST https://api.resourcewatch.org/v1/dataset \
-H "Authorization: Bearer <your-token>" \
-H "x-api-key: <your-api-key>" \
-H "Content-Type: application/json"  -d \
'{
  "dataset": {
    "connectorType":"document",
    "provider":"csv",
    "sources": [
      "https://gfw2-data.s3.amazonaws.com/alerts-tsv/glad_headers_1.csv",
      "https://gfw2-data.s3.amazonaws.com/alerts-tsv/glad_headers_2.csv",
      "https://gfw2-data.s3.amazonaws.com/alerts-tsv/glad_headers_3.csv"
    ],
    "application":[
     "gfw"
    ],
    "name":"Glad points"
  }
}'

Creating a JSON dataset with data provided on the request body:

curl -X POST https://api.resourcewatch.org/v1/dataset \
-H "Authorization: Bearer <your-token>" \
-H "x-api-key: <your-api-key>" \
-H "Content-Type: application/json"  -d \
'{
  "dataset": {
    "connectorType":"document",
    "provider":"json",
    "application":[
     "your", "apps"
    ],
    "data": {"myData":[
            {"name":"nameOne", "id":"random1"},
            {"name":"nameTow", "id":"random2"}
          ]},
    "name":"Example JSON Dataset"
  }
}'

This dataset hosts data from files in JSON, CSV, TSV or XML format.

Here's a breakdown of the fields you need to specify when creating a document type dataset, besides the common required fields:

When creating a document based dataset, you have multiple ways of providing your data to the API, of which you should use only one:

• The sources field expects a list of URL containing files in the format matching your provider value. This is the recommended way of uploading data.
• The connectorUrl field is similar to sources, but it only accepts a single value. Deprecation notice: The previously used connectorUrl field should be considered deprecated when creating document-based datasets.
• If creating a dataset based on JSON data, you can upload the data on the actual request you issue to the API, inside the data field.

The data passed in sources or connectorUrl must be available on a publicly accessible URLs, specified in the sources array field or the connectorUrl single value field. The URLs must be an accessible CSV, TSV or XML file, non-compressed - zip, tar, tar.gz, etc are not supported. sources allows you to specify multiple URLs for a single dataset, provided all files have the same format and data structure. This is particularly useful when creating very large datasets, as it will allow the creation process to be parallelized. No warranties are provided about the order in which the files or their parts are imported.

Notice: If you want to create a dataset from a file you have, but that it's not available on a public URL, check out our docs for uploading a dataset.

Unlike with other dataset types, when the dataset is created, the data is copied from the provided source into the API's internal Elasticsearch instance, which is the source used for subsequent queries or other operations. This has a few implications that you should be aware of:

• The URLs provided in sources or connectorUrl will be publicly visible to all RW API users.
• On dataset creation, those URLs will be visited as the API loads and copies your data.
• Queries to the dataset will use the API's internal copy of your data. If you wish to update that data, see the documentation below on how to update your data.
• If the URLs provided become unavailable after the creation process is over, the dataset will continue to work normally
  • your data will still be available online.

Notice: When creating a document-based dataset, if any of the fields has a numerical name (for example, column: 3), a string named col_ will be appended to the beginning of the name of the column. This way, an uploaded column named 3 will become col_3.

Tip: If you want to periodically and automatically update your document based dataset with new data, check out the dataset automatic synchronization functionality.

When creating a document-based dataset, the RW API will start a complex process that copies your data into the API's internal database, and perform certain indexing actions. This process is called a Task, and is given a taskId that is stored on the dataset's field with the same name. Depending on the size of your dataset, this may take from a few seconds to a few hours to complete. The result of this import process will determine if the dataset's status will be set to saved or error. You can follow this process by using the taskId value with the Tasks API.

Using the legend fields to define field types

By default, when creating a document based dataset, the data is ingested by the API and the field types are automatically determined by the underlying Elasticsearch dynamic mapping API . However, in some scenarios, it may be desirable to specify some or all of these mappings manually, to match each field type to its Elasticsearch equivalent.

When defining manual mappings, you don't need to map every single field. When processing the data, if a field is found for which there isn't a manual mapping, Elasticsearch will fallback to its dynamic mapping algorithm to try and guess that field's type. This API only supports a single explicit mapping per field, meaning you cannot declare a given field as both text and keyword for example.

Field mapping can only be defined on dataset creation. Should you want to change these mappings, you can only do so by creating a new dataset with the new mapping structure.

The legend field allows explicitly identifying the following mappings:

For more details on the characteristics of each of the basic data types, refer to the Elasticsearch documentation .

Uploading a dataset file

Upload raw data for using in a dataset

curl -X POST https://api.resourcewatch.org/v1/dataset/upload \
-H "Authorization: Bearer <your-token>" \
-H "x-api-key: <your-api-key>" \
-F provider=csv,
-F dataset=@<your-file>

Example response

{
  "connectorUrl": "rw.dataset.raw/some-file.csv",
  "fields": [
    "Country (region)",
    "Positive affect",
    "Negative affect",
    "Social support",
    "Freedom",
    "Corruption",
    "Generosity"
  ]
}

Using the returned connectorUrl to create a new dataset

curl -X POST https://api.resourcewatch.org/v1/dataset \
-H "Authorization: Bearer <your-token>" \
-H "x-api-key: <your-api-key>" \
-H "Content-Type: application/json"
'{
  "dataset": {
    "connectorType":"document",
    "provider":"csv",
    "connectorUrl":"rw.dataset.raw/some-file.csv",
    "application":[
     "your", "apps"
    ],
    "name":"Example RAW Data Dataset"
  }
}'

The upload endpoint allows you to create datasets on the API from local files that aren't available online. If your file is up to 4MB in size, you can upload it to the API by using the upload endpoint. This endpoint accepts a file in the "dataset" field of your POST request, and a provider that matches your file type and extension. The supported formats/extensions are: csv, json, tsv, xml, tif, tiff and geo.tiff. The request uploads the file to the API, and returns a specially crafted connectorUrl value, and a list of fields found in your file. With this data, you can create a document type dataset by passing it to the connectorUrl value of a new document type dataset.

Errors for upload a dataset file

Updating a dataset

There are multiple options to update a dataset, depending on what modification you are trying to achieve, and the underlying data provider of the dataset itself. This section covers the different endpoints that allow you to modify a dataset, their details, and helps you pick the option that best fits your scenario. We recommend reviewing all of them before proceeding, so you can find the endpoint that best matches your needs.

Updating the fields of a dataset

Example request for updating the name of a dataset

curl -X PATCH https://api.resourcewatch.org/v1/dataset/<dataset-id-or-slug> \
-H "Authorization: Bearer <your-token>" \
-H "x-api-key: <your-api-key>" \
-H "Content-Type: application/json" -d \
'{
    "name": "Another name for the dataset"
}'

In order to modify the fields of a dataset, you can use the patch dataset endpoint. This endpoint allows you to modify most of your dataset's details, like the name or publishing status, but can also be used to modify internal fields of a dataset, like connectorType. When making these changes, be mindful that some of these fields are critical to the correct behavior of the dataset, and providing incorrect values may break your dataset. There are other endpoints documented in this page that allow you to perform certain update-like operations (ie. update data on a document-type dataset), so refer to those first before using this endpoint. Also important to keep in mind is the fact that this endpoint does not perform validation on things like connectorUrl /sources URLs.

All the fields in the dataset reference can be modified, except the following:

• slug
• userId
• createdAt
• updatedAt

Additionally, certain fields have special behavior associated with them:

• status can only be modified by users with ADMIN role or by other microservices.
• taskId and errorMessage can only be modified by other microservices.
• published can only be modified by users with ADMIN role.
• env changes will not only affect the dataset, but also related widgets and layers. Refer to the PATCH /widget/change-environment/<dataset id>/<env> and the the PATCH /layer/change-environment/<dataset id>/<env> endpoints respectively for more details.

When passing new values for Object type fields, the new value will fully overwrite the previous one. It's up to you, as an API user, to build any merging logic into your application.

To perform this operation, the following conditions must be met:

• the user must be logged in and belong to the same application as the dataset
• the user must comply with the RW API role-based access control guidelines.

Use this endpoint when:

• Modifying the fields of a dataset, like name, and not the data itself.
• Modifying the source of data for any dataset that's not document based (connectorType values other than document).
• Making advanced changes to a dataset.

Errors for updating the fields of a dataset

Concatenate and append data to a document based dataset

Concatenate data using external data source:

curl -X POST https://api.resourcewatch.org/v1/dataset/:dataset_id/concat \
-H "Authorization: Bearer <your-token>" \
-H "x-api-key: <your-api-key>" \
-H "Content-Type: application/json"  -d \
'{
    "provider": "json",
    "sources": ["<csv1Url>", "<csv2Url>", "<csv3Url>"],
    "dataPath": "data... etc"
}'

Append data using external data source:

curl -X POST https://api.resourcewatch.org/v1/dataset/:dataset_id/append \
-H "Authorization: Bearer <your-token>" \
-H "x-api-key: <your-api-key>" \
-H "Content-Type: application/json"  -d \
'{
    "provider": "json",
    "sources": ["<csvUrl>"],
    "dataPath": "data... etc"
}'

Concatenate data using JSON array in post body:

curl -X POST https://api.resourcewatch.org/v1/dataset/:dataset_id/concat \
-H "Authorization: Bearer <your-token>" \
-H "x-api-key: <your-api-key>" \
-H "Content-Type: application/json"  -d \
'{
    "provider": "json",
    "data": [{},{}]
}'

Using these endpoints, you can add more data to an already existing dataset. They are exclusively available for document based datasets - you'll get a 404 error if you use them on a dataset of a different type. You can either provide the URL for the file containing the data you wish to add, or simply provide that data in the body of your request, as a JSON object.

These process are asynchronous and not instantaneous. Immediately when triggered, these requests will cause the dataset's status to be set to pending, meaning you will not be able to issue new overwrite, concatenate or append requests. Once the request has been fully processed, the status will be automatically set to saved. Depending on factors like API load or the size of the data being uploaded, this may take from a few minutes to a few hours to occur. The API does not issue any notification when the asynchronous operation is finished.

In order to perform these operation, the following conditions must be met:

• the dataset's overwrite property must be set to true.
• the dataset's status property must be set to saved.
• the user must comply with the RW API role-based access control guidelines.
• the user must have at least one overlapping application with the ones associated with the dataset.

While they ultimately achieve a very similar end result, concatenate and append rely on different internal processes, each with its own characteristics.

• The concatenate process relies on a slower approach to ensure the operation is atomic. Until the operation is completed, you will see the dataset data as it was before the concatenate operation was triggered. If the operation fails, the old version of the data is kept accessible as it was before the concatenation process was started. There's a known issue where concatenating new data to already existing large datasets may result in failure.
• The append operation relies on a faster process that does not offer atomicity. The new data is simply added to the current dataset, and any queries done while this process is taking place may produce results based on the preexisting data mixed with parts of the newly added values. Should the operation fail halfway, your dataset may contain all the preexisting records as well as part (but not all) of the newly added ones.

Here's a more detailed description of the request's body fields:

Tip: If you want to periodically and automatically concatenate data to your document based dataset, check out the dataset automatic synchronization functionality.

Use this endpoint when:

• Adding new data to an existing document based dataset, without modifying/removing the existing one.

Errors for concatenating and appending data to a document based dataset

Overwrite data for a document based dataset

Overwrite data using external data source:

curl -X POST https://api.resourcewatch.org/v1/dataset/:dataset_id/data-overwrite \
-H "Authorization: Bearer <your-token>" \
-H "x-api-key: <your-api-key>" \
-H "Content-Type: application/json"  -d \
'{
   "sources": ["<url of the data source>"],
   "provider": "csv"
}'

Overwrite data using JSON array in post body:

curl -X POST https://api.resourcewatch.org/v1/dataset/:dataset_id/data-overwrite \
-H "Authorization: Bearer <your-token>" \
-H "x-api-key: <your-api-key>" \
-H "Content-Type: application/json"  -d \
'{
   "data": [{},{}],
   "provider": "csv"
}'

Using this endpoint, you can add or completely replace the data of an already existing dataset. It's exclusively available for document based datasets - you'll get a 404 error if you use it on a dataset of a different type. All previously existing data will be permanently deleted. You can either provide the URL(s) for the file(s) containing the data you wish to add, or simply provide that data in the body of your request, as a JSON object. There are no requirements regarding similarity of the data structure between existing and new data - your overwrite data can have a completely different data schema.

This process is asynchronous and not instantaneous. Immediately when triggered, this request will cause the dataset's status to be set to pending, meaning you will not be able to issue new overwrite or concat requests, and will not yet be able to access the new data yet. Once the request has been fully processed, the status will be automatically set to saved and the new data will be accessible. Depending on factors like API load or the size of the data being uploaded, this may take from a few minutes to a few hours to occur. The API does not issue any notification when the asynchronous operation is finished.

In order to perform this operation, the following conditions must be met:

• the dataset's overwrite property must be set to true.
• the dataset's status property must be set to saved.
• the user must comply with the RW API role-based access control guidelines.
• the user must have at least one overlapping application with the ones associated with the dataset.

Here's a more detailed description of the request's body fields:

Tip: If you want to periodically and automatically overwrite the data on your document based dataset, check out the dataset automatic synchronization functionality.

Errors for overwriting data for a document based dataset

Use this endpoint when:

• Fully replacing the data of an existing document based dataset, while keeping other values like id or name.
• Modifying the structure of the data for a document based dataset.

Cloning a dataset

Example request for cloning a dataset

curl -X POST https://api.resourcewatch.org/v1/dataset/5306fd54-df71-4e20-8b34-2ff464ab28be/clone \
-H "Authorization: Bearer <your-token>" \
-H "x-api-key: <your-api-key>"
-H "Content-Type: application/json" -d \
'{
  "dataset": {
    "datasetUrl": "/query/5306fd54-df71-4e20-8b34-2ff464ab28be?sql=select%20%2A%20from%20data%20limit%2010",
    "application": [
      "your",
      "apps"
    ],
    "legend": {...},
    "applicationConfig" : {...}
  }
}'

This endpoint allows you to create a new, json based dataset, from the result of a RW API endpoint call. The basic usage example would be to create a new dataset based on a custom query to an existing dataset - this is illustrated in the example curl call in this section. Other use cases could be converting the result of an analysis endpoint into a dataset, or capturing the result of a query to a REST based dataset (cartodb, arcgis, etc) to an internal json representation of the same data, that is kept in the API database.

In order to perform this operation, the following conditions must be met:

• the user must belong to the applications specified in the request body.
• the user must comply with the RW API role-based access control guidelines.

The request requires two fields to be present:

• datasetUrl: the URL from where to load the data to populate the new dataset. Must be RW API, relative path, like /query/5306fd54-df71-4e20-8b34-2ff464ab28be?sql=select%20%2A%20from%20data%20limit%2010.
• application: the array of applications to which the new dataset will belong.

Additionally, you can optionally specify these fields:

• legend: field structure and type of the new dataset. Refer to dataset reference for more details.
• applicationConfig: application-specific configuration. Refer to dataset reference for more details. If not provided, it will be copied from the original dataset.
• published: if the user has ADMIN role, they can set this value to true.

For the fields in the following list, values will be copied from the original dataset to the new one, unless specified:

• name: copied from the original dataset, with a timestamp appended, to ensure uniqueness.
• subtitle
• dataPath: will always be set to data, matching the expected response structure of RW API responses like /query that encapsulate the "real" data in a data json object.
• connectorType: will be set to document
• provider: will be set to json
• attributesPath
• tableName
• dataLastUpdated
• overwrite
• published: if the user has role ADMIN and does not explicitly set published to true on the request body, this value is inherited. Otherwise, it's set to false.

Datasets created through a cloning operation will have a specific clonedHost object, with additional data. Refer to dataset reference for more details on the content of this field.

The dataset cloning requests accepts an optional full boolean query parameter that, when set to true, will clone of vocabulary-related data and metadata from the original dataset to the new one.

Errors for cloning a dataset

Deleting a dataset

Example request for deleting a dataset

curl -X DELETE https://api.resourcewatch.org/v1/dataset/<dataset-id> \
-H "Authorization: Bearer <your-token>" \
-H "x-api-key: <your-api-key>"
-H "Content-Type: application/json"

Use this endpoint if you wish to delete a dataset. Deleting a dataset of type document (connectorType with value document) will cause the API's internal copy of said data to be deleted. Dataset types that proxy data from an external source (ie. carto, gee, etc) will be deleted without modifying said external source.

When deleting a dataset that is associated with multiple application values, the user issuing the request must be associated with all of them in order for the dataset to be deleted. If that's not the case, the application values associated with the user will be removed from the dataset's application list, and no further action will be taken - the dataset itself and its associated resources will continue to exist. The dataset is only actually deleted if the user has access to all of the application to which the dataset belongs.

Besides deleting the dataset itself, this endpoint also deletes graph vocabularies, layers, widgets and metadata related to the dataset itself. These delete operations are issued in this order, and prior to deleting the dataset itself, but are not atomic - if one of them fails (for example, if attempting to delete a protected resource), the following ones are canceled, but the already deleted elements are not restored.

In order to delete a dataset, the following conditions must be met:

• the dataset's protected property must be set to false.
• the user must be logged in and belong to the same application as the dataset
• the user must comply with the RW API role-based access control guidelines.

Errors for deleting a dataset

Deleting datasets by user id

Example request for deleting datasets by user id

curl -X DELETE https://api.resourcewatch.org/v1/dataset/by-user/<user-id> \
-H "Authorization: Bearer <your-token>" \
-H "x-api-key: <your-api-key>"
-H "Content-Type: application/json"

Example response:

{
    "deletedDatasets": [
      {
        "id": "aef2be42-1ee8-4069-a55a-16a988f2b7a0",
        "type": "dataset",
        "attributes": {
          "name": "Glad points",
          "slug": "Glad-points-1490086842129",
          "type": null,
          "subtitle": null,
          "application": [
            "data4sdgs"
          ],
          "dataPath": null,
          "attributesPath": null,
          "connectorType": "document",
          "provider": "csv",
          "userId": "58333dcfd9f39b189ca44c75",
          "connectorUrl": "https://gfw2-data.s3.amazonaws.com/alerts-tsv/glad_headers.csv",
          "sources": [],
          "tableName": "data",
          "status": "pending",
          "published": true,
          "protected": false,
          "overwrite": false,
          "env": "production",
          "geoInfo": false,
          "legend": {
            "date": [],
            "region": [],
            "country": []
          },
          "clonedHost": {},
          "errorMessage": null,
          "updatedAt": "2017-01-13T10:45:46.368Z",
          "widgetRelevantProps": [],
          "layerRelevantProps": []
        }
      }
    ],
    "protectedDatasets": [
      {
        "id": "00f2be42-1ee8-4069-a55a-16a988f2b762",
        "type": "dataset",
        "attributes": {
          "name": "Fires data",
          "slug": "fires-data-1490086842163",
          "type": null,
          "subtitle": null,
          "application": [
            "data4sdgs"
          ],
          "dataPath": null,
          "attributesPath": null,
          "connectorType": "document",
          "provider": "csv",
          "userId": "58333dcfd9f39b189ca44c75",
          "connectorUrl": "https://gfw2-data.s3.amazonaws.com/alerts-tsv/fires.csv",
          "sources": [],
          "tableName": "data",
          "status": "pending",
          "published": true,
          "protected": true,
          "overwrite": false,
          "env": "production",
          "geoInfo": false,
          "legend": {
            "date": [],
            "region": [],
            "country": []
          },
          "clonedHost": {},
          "errorMessage": null,
          "updatedAt": "2017-01-13T10:45:46.368Z",
          "widgetRelevantProps": [],
          "layerRelevantProps": []
        }
      }
    ]
}

Use this endpoint if you wish to delete all datasets associated with a user. Deleting datasets of type document (connectorType with value document) will cause the API's internal copy of said data to be deleted. Dataset types that proxy data from an external source (ie. carto, gee, etc) will be deleted without modifying said external source.

Unlike what happens when deleting a single dataset by id, deleting all datasets for a user does not validate if the datasets belong to different applications and prevents their actual deletion based on that - datasets will be deleted even if they are part of applications that the user is not associated with.

Besides deleting the datasets themselves, this endpoint also deletes graph vocabularies, layers, widgets and metadata related with the datasets themselves. For each dataset, these delete operations are issued in this order, and prior to deleting the dataset itself, but are not atomic - if one of them fails, the following ones are canceled, but the already deleted elements are not restored. Any other datasets associated with the given user may or may not be deleted, as the operation occurs internally in an asynchronous fashion.

Any microservice or user with ADMIN role can use this endpoint. Regular users can use this endpoint to delete the datasets they own.

The response includes two lists of datasets: - deletedDatasets: an unpaginated list of all datasets that were deleted as part of this operation. - protectedDatasets: an unpaginated list of all datasets associated with the provided user, that have protected status set to true, and thus were not deleted.

Errors for deleting datasets by user id

Dataset automatic synchronization

curl -X POST https://api.resourcewatch.org/v1/dataset \
-H "Authorization: Bearer <your-token>" \
-H "x-api-key: <your-api-key>" \
-H "Content-Type: application/json"
'{
    "connectorType":"document",
    "provider":"csv",
    "connectorUrl":"<csvUrl>",
    "application":[
     "your", "apps"
    ],
    "name":"Example SYNC Dataset",
    "overwrite": true,
    "sync": {
        "action":"concat",
        "cronPattern":"0 * * * * *",
        "url":"<updateCsvUrl>"
    }
}'

Certain datasets contain data that evolves frequently over time, and you, as the creator of this dataset, what to ensure that it's up-to-date. As we've seen before, if your data is hosted on one of the supported 3rd party data providers, like Carto or Arcgis, your data will be proxied, so users of the RW API will always see the latest version of your data. However, if you uploaded your data from a file, your data is copied to the RW API database at the time of the dataset's creation, so keeping it up-to-date requires a different approach. It's with scenario in mind that the RW API offers the automatic synchronization mechanism.

The automatic synchronization mechanism is available for document based datasets only, and you can configure it when creating or updating a dataset. When enabled, it will schedule an automatic, periodic task, that will update your document based dataset's data, based on data from an URL you provide. This means that, once configured, you just have to make sure the automatic synchronization mechanism can find the newest version of the data at the specified URL, and the RW API will take care of the actual data update process for you.

To configure automatic synchronization on dataset creation or update, you need to pass a sync object on your calls to the respective endpoints, next to the other fields. See the included example for a clearer idea of how creating a dataset with automatic synchronization looks like.

There are 3 fields, all required, that you need to specify inside the sync object:

• action: choose either concat or overwrite. See more details above about concatenating and overwriting dataset data. Append operations, as documented in the section above, is not supported.
• cronPattern: cron expression representing when and how frequently the synchronization process should take place. The host executing these tasks use UTC timezone.
• url: publicly available URL from where the automatic synchronization mechanism will load the data to replace/concatenate to your dataset.

Internally, the automatic synchronization mechanism will call either the dataset concatenation or dataset overwrite endpoint. If that internal request fails - for example, if overwrite is set to false - the sync process will fail silently. If you have the ADMIN role, you can use the /v1/task endpoint to see scheduled tasks and their error output, but otherwise this failure will be invisible to end users.

On the other hand, if the request goes through, but the concatenation/overwrite process fails - for example, if the URL provided is not available - then your dataset status will be set to error and the errorMessage field will give you a simple description of what happened.

To see the details (including the date) of the last operation performed on the dataset, use the link in the taskId field.

Flush dataset cache

Flush dataset's cache

curl -X POST https://api.resourcewatch.org/v1/dataset/0c630feb-8146-4fcc-a9be-be5adcb731c8/flush \
-H "Authorization: Bearer <your-token>" \
-H "x-api-key: <your-api-key>"

Response:

OK

Flushes the cache for the specified dataset. Take into account that only the dataset itself, query results and fields will be flushed. Any other relation, like metadata, layers or widgets linked to the specified dataset will not be affected by this action.

In order to flush a dataset's cache, the following conditions must be met:

• the user must be logged in.
• the user must comply with the RW API role-based access control guidelines.

Errors for flushing a dataset's cache

Recover

Recover dataset

curl -X POST https://api.resourcewatch.org/v1/0c630feb-8146-4fcc-a9be-be5adcb731c8/recover \
-H "Authorization: Bearer <your-token>" \
-H "x-api-key: <your-api-key>"

Response:

OK

Resets a dataset's status to saved and clears its errors. Keep in mind that this does NOT modify the dataset in any other way - if the underling dataset's data was inconsistent for any reason, this endpoint will not change it, and it's up to you to fix it using a data-overwrite or other endpoints.

In order to recover a dataset, the user must be logged in and have the ADMIN role.

Errors for recovering a dataset

Dataset reference

This section gives you a complete view at the properties that are maintained as part of dataset. When interacting with a dataset (on get, on create, etc) you will find most of these properties available to you, although they may be organized in a slightly different structure (ie: on get, everything but the id is nested inside an attributes object).

You can find more details in the source code.

Fields

What are fields?

Fields are the properties of a dataset your will find when querying its data, use when filtering data through queries, or visualise when rendering a layer or a widget.

Fields are part of a dataset, and are determined by the structure of data provided by the dataset creator. Each field also includes information about its data type, derived automatically by the dataset provider (check out the specifics of each provider in the sections below). If you are uploading your own datasets to the RW API, you don't have to do anything else for your dataset to be available through the fields endpoints - there's no additional data you have to provide. Keep in mind the endpoints described in the sections below are read-only - if you want to change your dataset's fields, you need to upload the dataset's data again.

Fields are targeted at users consuming datasets available on the RW API, particularly if you want to build a flexible and scalable application. The fields endpoints aim at giving you a consistent and uniform view of the structure of data hosted in different dataset providers, that would otherwise be served in a heterogeneous structure.

How to get the dataset fields

Querying the fields of a dataset

curl -X GET https://api.resourcewatch.org/v1/fields/<dataset-id> \
-H "x-api-key: <your-api-key>"

Response example

{
    "tableName": "public.cait_2_0_country_ghg_emissions_toplow2011",
    "fields": {
        "cartodb_id": {
            "type": "number"
        },
        "the_geom": {
            "type": "geometry"
        },
        "the_geom_webmercator": {
            "type": "geometry"
        },
        "x": {
            "type": "string"
        },
        "y": {
            "type": "number"
        },
        "z": {
            "type": "string"
        }
    }
}

Once the dataset has been created and its status is set to saved, you will be able to use this endpoint to get details about a its fields. The resulting response contains two root fields

• tableName: a string containing the table name, as stored in the dataset object. The value and function of this field will depend on the dataset's provider, and you can refer to the dataset documentation for more details about this.
• fields: a key-value pair of field names and their respective details.

While the aim of this endpoint is to provide an homogeneous view of data stored in different systems, you will still encounter slight variations in data, depending on the underlying type of the dataset you're querying. The example response on the right illustrates the typical basic response structure: tableName with a string value, and a fields map that matches each field's name to a type. You'll find this structure throughout all responses, but you may also find additional details for each field, and the data types may vary in name. In the next sections we'll cover some of the specifics for each dataset type.

Carto

Example response for a Carto dataset

{
    "tableName": "gadm28_adm1",
    "fields": {
        "cartodb_id": {
            "type": "number",
            "pgtype": "int4"
        },
        "the_geom": {
            "type": "geometry",
            "wkbtype": "Unknown",
            "dims": 2,
            "srid": 4326
        },
        "the_geom_webmercator": {
            "type": "geometry",
            "wkbtype": "Unknown",
            "dims": 2,
            "srid": 3857
        },
        "objectid": {
            "type": "number",
            "pgtype": "int8"
        },
        "iso": {
            "type": "string",
            "pgtype": "text"
        },
        "centroid": {
            "type": "string",
            "pgtype": "text"
        },
        "area_ha": {
            "type": "number",
            "pgtype": "float8"
        },
        "topojson": {
            "type": "string",
            "pgtype": "text"
        },
        "the_geom_simple": {
            "type": "geometry",
            "wkbtype": "Unknown",
            "dims": 2,
            "srid": 4326
        },
        "geojson": {
            "type": "string",
            "pgtype": "text"
        }
    }
}

Carto datasets provide a type for each field, as well as additional details, depending on the field type. Most fields will have a pgtype field, which will identify a sub-type of sorts - for example, type number is matched with different pgtype values in this example. This is related with the underling Carto implementation, that uses PostgreSQL, PostGIS and its own software stack. We recommend relying on the type as much as possible.

Other, more complex, field types, like geometry types, include additional details for their respective types. As before, this is related with the underlying Carto implementation tools, and not actively maintained or supported by the RW API.

For more information about the different types that can be found in Carto datasets please refer to PostgreSQL's documentation on Data Types or PostGIS's documentation.

GFW

Example fields response for a GFW dataset

{
    "fields": [
        {
            "field_name": "umd_glad_landsat_alerts__date_conf",
            "field_alias": null,
            "field_description": null,
            "field_values": null
        },
        {
            "field_name": "umd_glad_landsat_alerts__date",
            "field_alias": null,
            "field_description": null,
            "field_values": null
        },
        {
            "field_name": "umd_glad_landsat_alerts__confidence",
            "field_alias": null,
            "field_description": null,
            "field_values": [
                "nominal",
                "high",
                "highest",
                "not_detected"
            ]
        },
        {
            "field_name": "gfw_integrated_alerts__date_conf",
            "field_alias": null,
            "field_description": null,
            "field_values": null
        },
        {
            "field_name": "gfw_integrated_alerts__date",
            "field_alias": null,
            "field_description": null,
            "field_values": null
        },
        {
            "field_name": "gfw_integrated_alerts__confidence",
            "field_alias": null,
            "field_description": null,
            "field_values": [
                "nominal",
                "high",
                "highest",
                "not_detected"
            ]
        },
        {
            "field_name": "umd_glad_sentinel2_alerts__date_conf",
            "field_alias": null,
            "field_description": null,
            "field_values": null
        },
        {
            "field_name": "umd_glad_sentinel2_alerts__date",
            "field_alias": null,
            "field_description": null,
            "field_values": null
        },
        {
            "field_name": "umd_glad_sentinel2_alerts__confidence",
            "field_alias": null,
            "field_description": null,
            "field_values": [
                "nominal",
                "high",
                "highest",
                "not_detected"
            ]
        },
        {
            "field_name": "wur_radd_alerts__date_conf",
            "field_alias": null,
            "field_description": null,
            "field_values": null
        },
        {
            "field_name": "wur_radd_alerts__date",
            "field_alias": null,
            "field_description": null,
            "field_values": null
        },
        {
            "field_name": "wur_radd_alerts__confidence",
            "field_alias": null,
            "field_description": null,
            "field_values": [
                "nominal",
                "high",
                "highest",
                "not_detected"
            ]
        }
    ]
}

The fields endpoint for GFW dataset returns an array of objects with each object containing field attributes including name, description and alias. Vector dataset fields include additional attributes indicating whether field is feature info and filter. Fields for raster dataset map to individual raster files associated with a dataset. For example, RADD Deforestation Alerts dataset has confidence field mapping to the primary raster dataset with possible values defined in field_values attribute and additional fields such as is__landmark_indigenous_and_community_lands, is__gfw_oil_palm, etc that correspond to contextual raster datasets used to conduct analysis on the alerts data.

ArcGIS feature layer

Example response for a ArcGIS dataset

{
    "tableName": "conservationMapServer3",
    "fields": {
        "objectid": {
            "type": "esriFieldTypeOID"
        },
        "tcl_name": {
            "type": "esriFieldTypeString"
        },
        "tcl_id": {
            "type": "esriFieldTypeSmallInteger"
        },
        "area_ha": {
            "type": "esriFieldTypeInteger"
        },
        "tx2_tcl": {
            "type": "esriFieldTypeSmallInteger"
        },
        "gfwid": {
            "type": "esriFieldTypeString"
        },
        "globalid": {
            "type": "esriFieldTypeGlobalID"
        },
        "shape": {
            "type": "esriFieldTypeGeometry"
        },
        "shape_Length": {
            "type": "esriFieldTypeDouble"
        },
        "shape_Area": {
            "type": "esriFieldTypeDouble"
        }
    }
}

ArcGIS uses its own type naming convention, as you see reflected in the included example. The RW API passes that information as-is to you. As different ArcGIS datasets rely on different ArcGIS server instances and versions, you may encounter variations in the types you'll find, depending on how ArcGIS evolves these types over type.

For more information about the different types that can be found in ArcGIS datasets please refer to ArcGIS's field data types documentation.

Google Earth Engine

Example response for a Google Earth Engine dataset

{
    "data": {
        "fields": {
            "bands": [
                {
                    "dataType": {
                        "precision": "INT",
                        "range": {
                            "max": 2147483647,
                            "min": -2147483648
                        }
                    },
                    "grid": {
                        "affineTransform": {
                            "scaleX": 0.00833333376795053,
                            "scaleY": -0.00833333376795053,
                            "translateX": -180.0000000000001,
                            "translateY": 90.00000782310963
                        },
                        "crsCode": "EPSG:4326",
                        "dimensions": {
                            "height": 18000,
                            "width": 43200
                        }
                    },
                    "id": "b1",
                    "pyramidingPolicy": "MEAN"
                }
            ],
            "geometry": {
                "coordinates": [
                    [
                        [
                            "-Infinity",
                            "-Infinity"
                        ],
                        [
                            "Infinity",
                            "-Infinity"
                        ],
                        [
                            "Infinity",
                            "Infinity"
                        ],
                        [
                            "-Infinity",
                            "Infinity"
                        ],
                        [
                            "-Infinity",
                            "-Infinity"
                        ]
                    ]
                ],
                "type": "Polygon"
            },
            "id": "users/resourcewatch/cli_030_global_aridity",
            "name": "projects/earthengine-legacy/assets/users/resourcewatch/cli_030_global_aridity",
            "properties": {
                "title": "Image:cli_030 _global_aridity"
            },
            "sizeBytes": "413389345",
            "title": "Image:cli_030 _global_aridity",
            "type": "IMAGE",
            "updateTime": "2017-10-06T18:58:39.646890Z"
        },
        "tableName": "users/resourcewatch/cli_030_global_aridity"
    }
}

Google Earth Engine (GEE) is perhaps the biggest outlier when it comes to fields data. Because it relies on raster data, rather than tabular data, the data structure you'll find in a GEE dataset is very different, as you can see in the example on the side.

The response consists of loading the details for a single image, with the most meaningful information being the properties (of an image) and bands (of said image). Those will be the fields you'll be able to retrieve when using the query endpoints.

NEX-GDDP

Example response for a NEX-GDDP dataset

{
    "fields": {
        "tasavg": {
            "type": "number",
            "uom": "10^0"
        },
        "tasavg_q25": {
            "type": "number",
            "uom": "10^0"
        },
        "tasavg_q75": {
            "type": "number",
            "uom": "10^0"
        },
        "year": {
            "type": "date"
        }
    },
    "tableName": "tasavg/rcp45_30_y"
}

NEX-GDDP datasets have an additional uom (unit of measure) field on some field types, but otherwise has no other info besides the basic details.

BigQuery

Example response for a BigQuery dataset

{
    "tableName": "[bigquery-public-data:ghcn_m.ghcnm_tmin]",
    "fields": [
        {
            "name": "id",
            "type": "INTEGER",
            "mode": "REQUIRED",
            "description": "11 digit identifier, digits 1-3=Country Code, digits 4-8 represent the WMO id if the station is a WMO station. It is a WMO station if digits 9-11=\"000\"."
        },
        {
            "name": "year",
            "type": "INTEGER",
            "mode": "NULLABLE",
            "description": "4 digit year of the station record."
        },
        {
            "name": "element",
            "type": "STRING",
            "mode": "NULLABLE",
            "description": "element type, monthly mean temperature=\"TAVG\" monthly maximum temperature=\"TMAX\" monthly minimum temperature=\"TMIN\""
        },
        {
            "name": "value1",
            "type": "INTEGER",
            "mode": "NULLABLE",
            "description": "monthly value (MISSING=-9999).  Temperature values are in hundredths of a degree Celsius, but are expressed as whole integers (e.g. divide by 100.0 to get whole degrees Celsius)."
        },
        {
            "name": "qcflag3",
            "type": "STRING",
            "mode": "NULLABLE",
            "description": "quality control flag, seven possibilities within quality controlled unadjusted (qcu) dataset, and 2 possibilities within the quality controlled adjusted (qca) dataset."
        },
        {
            "name": "qcflag12",
            "type": "STRING",
            "mode": "NULLABLE",
            "description": "quality control flag, seven possibilities within quality controlled unadjusted (qcu) dataset, and 2 possibilities within the quality controlled adjusted (qca) dataset."
        },
        {
            "name": "dsflag12",
            "type": "STRING",
            "mode": "NULLABLE",
            "description": "data source flag for monthly value, 21 possibilities"
        }
    ]
}

BigQuery has a slight variation on the core structure returned by the fields endpoint for other dataset types. Instead of fields having a key-value map, it has an array of objects, each describing a field. Each of these objects has a name and type property, identifying a field and its type, as well as a mode and description, respectively indicating if the field is nullable or required, and providing a human-friendly description of the field's content or purpose.

Loca

Example response for a Loca dataset

{
    "fields": {
        "xs": {
            "type": "number",
            "uom": "10^0"
        },
        "xs_q25": {
            "type": "number",
            "uom": "10^0"
        },
        "xs_q75": {
            "type": "number",
            "uom": "10^0"
        },
        "year": {
            "type": "date"
        }
    },
    "tableName": "loca_xs/rcp85_30_y"
}

Loca datasets have an additional uom (unit of measure) field on some field types, but otherwise has no other info besides the basic details.

Document-based dataset types (csv, tsv, json and xml)

Example response for a document-based dataset

{
    "tableName": "index_1c9a1e4f455b4c03ac88dd2242a2e4b1_1602065490373",
    "fields": {
        "cartodb_id": {
            "type": "long"
        },
        "commodity": {
            "type": "text",
            "fields": {
                "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                }
            }
        },
        "data_id": {
            "type": "text",
            "fields": {
                "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                }
            }
        },
        "gcm_type": {
            "type": "text",
            "fields": {
                "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                }
            }
        },
        "impactparameter": {
            "type": "text",
            "fields": {
                "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                }
            }
        },
        "iso": {
            "type": "text",
            "fields": {
                "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                }
            }
        },
        "region": {
            "type": "text",
            "fields": {
                "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                }
            }
        },
        "scenario": {
            "type": "text",
            "fields": {
                "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                }
            }
        },
        "value": {
            "type": "float"
        },
        "year": {
            "type": "long"
        }
    }
}

Document based datasets from the 4 different types of sources (json, xml, csv and tsv) all share a common internal storage engine, based on Elasticsearch. So, when querying for fields of a document-based dataset, you'll find information matching the field mechanism used by Elasticsearch. This is particularly evident on text fields, where the keyword information is present. During most uses cases, we don't anticipate this information being needed, as relying on the type value should cover the majority of your needs.

Dataset types not supported by the fields endpoint

The following dataset types have not been integrated with the fields endpoint, so you will get a 404 response when querying for their fields:

• wms
• rasdaman
• vector

Errors for getting a dataset fields

Query

In order to retrieve data from datasets, you can send queries to the API using a syntax very similar to SQL. Using these endpoints, you can also download the results of a particular query. If you are new to the RW API, or want to learn more about the concept of a querying datasets, we strongly encourage you to read the query concept documentation first. It gives you a brief and clear description of what a query is, and what it is useful for.

Please note that some SQL features might not be supported. Check here for a reference of the SQL features' support for each dataset provider.

Querying datasets

Structure of the endpoint for executing a query:

curl -i -X GET 'https://api.resourcewatch.org/v1/query/<dataset.id>?sql=SELECT * FROM <dataset.tableName>' \
-H "x-api-key: <your-api-key>"

In order to query a dataset, you'll need two pieces of information:

• The id of the dataset you're trying to query.
• The SQL query that represents the data you are trying to retrieve.

The dataset documentation covers different ways that you can use to browse the existing dataset catalog or upload your own, all of which will give you the details of a dataset, including the dataset id you'll need to query it.

The SQL query will have to be custom built by you to fit your needs, but a good starting point for newcomers would be something like SELECT * FROM <dataset.tableName> limit 10.

Notice: the limit parameter restricts our results to 10 rows, and is not required. However, for our learning purposes, this is useful, as it keeps the API responses small and fast.

Most of the SQL query is up to you to define, based on your needs and the support provided for the dataset type you are using. The FROM clause, however, does use a special value - the dataset's tableName value, which you can also get from the dataset documentation described above.

Example endpoint for executing a query:

curl -i -X GET 'https://api.resourcewatch.org/v1/query/098b33df-6871-4e53-a5ff-b56a7d989f9a?sql=SELECT cartodb_id, iso, name_0, name_1, type_1 FROM gadm28_adm1 limit 10' \
-H "x-api-key: <your-api-key>"

With both pieces of information at hand, you can now send your query to the API and get the response. The example cURL to the side shows how that would look like.

Query response body

Example response:

{
  "data": [
    {
      "cartodb_id": 1830,
      "iso": "MEX",
      "name_0": "Mexico",
      "name_1": "Ciudad de México",
      "type_1": "Distrito Federal"
    },
    {
      "cartodb_id": 1109,
      "iso": "HTI",
      "name_0": "Haiti",
      "name_1": "L'Artibonite",
      "type_1": "Département"
    },
    ...
  ],
  "meta": {
    "cloneUrl": {
      "http_method": "POST",
      "url": "/dataset/098b33df-6871-4e53-a5ff-b56a7d989f9a/clone",
      "body": {
        "dataset": {
          "datasetUrl": "/query/098b33df-6871-4e53-a5ff-b56a7d989f9a?sql=SELECT%20*%20FROM%20gadm28_adm1%20limit%2010",
          "application": [
            "your",
            "apps"
          ]
        }
      }
    }
  }
}

The following table describes the response body fields:

Query endpoint parameters

Example of requesting the query results as CSV data:

curl -i -X GET 'https://api.resourcewatch.org/v1/query/9be3bf63-97fc-4bb0-b913-775ccae3cf9e?sql=SELECT alert__date from gadm28_adm1 limit 2&format=csv' \
-H "x-api-key: <your-api-key>"

Example response:

"alert__date",
"_id"
"2019-04-12",
"AW6O0fqMLu2ttL7ZDM4P"
"2015-08-22",
"AW6O0fqMLu2ttL7ZDM4T"

Example of requesting to freeze the query results:

curl -i -X GET 'https://api.resourcewatch.org/v1/query/9be3bf63-97fc-4bb0-b913-775ccae3cf9e?sql=SELECT alert__date from gadm28_adm1 limit 2&freeze=true' \
-H "Authorization: Bearer <your-token>" \
-H "x-api-key: <your-api-key>"

Example response:

{
  "url": "https://storage.googleapis.com/query-freeze/1589458072773.json"
}

The following parameters can be provided as query parameters, in order to customize the output of the response returned:

Filter query results by geostore

Example query providing a geostore id as query parameter to filter the results:

curl -i -X GET 'https://api.resourcewatch.org/v1/query/1d7085f7-11c7-4eaf-a29a-5a4de57d010e?sql=SELECT * FROM dis_001_significant_earthquakes LIMIT 5&geostore=972c24e1da2c2baacc7572ee9501abdc' \
-H "x-api-key: <your-api-key>"

Some dataset providers support receiving a geostore query parameter. When providing this parameter, you can request geo-referenced data that fits within the bounding box of the geostore with id provided. You can obtain the id of the geostore using the RW API Geostore API. If the data is not geo-referenced, or if the dataset provider does not support the geostore query parameter, it will be ignored.

The following providers support this parameter:

• CartoDB (carto)
• Global Forest Watch (gfw)
• ArcGIS (featureservice)
• Google Earth Engine (gee)
• BigQuery (bigquery)
• Rasdaman (rasdaman)
• NEX-GDDP (nexgddp)
• Loca (loca)

Alternative ways for querying datasets

While the GET request described above is the recommended way of querying datasets, there are other ways to query the RW API datasets that may be more suited for specific use cases.

Using the dataset slug instead of the id

Example query not using the dataset id in the request path, and using the dataset slug in the FROM clause:

curl -i -X GET 'https://api.resourcewatch.org/v1/query/9be3bf63-97fc-4bb0-b913-775ccae3cf9e?sql=SELECT alert__date from gadm28_adm1 limit 2' \
-H "x-api-key: <your-api-key>"
curl -i -X GET 'https://api.resourcewatch.org/v1/query/Glad-Alerts-Daily-Geostore-User-Areas_3?sql=SELECT alert__date from gadm28_adm1 limit 2' \
-H "x-api-key: <your-api-key>"

When referencing a dataset's id in a query, you have the option to use the dataset's slug instead, obtaining the same result. This is also applicable to the alternative query methods described in the sections below.

POST requests

The same query executed as GET, and as a POST request providing the SQL as request body param:

curl -i -X GET 'https://api.resourcewatch.org/v1/query/9be3bf63-97fc-4bb0-b913-775ccae3cf9e?sql=SELECT alert__date from gadm28_adm1 limit 2' \
-H "x-api-key: <your-api-key>"
curl -i -X POST 'https://api.resourcewatch.org/v1/query/9be3bf63-97fc-4bb0-b913-775ccae3cf9e' \
-H "x-api-key: <your-api-key>" \
-H 'Content-Type: application/json' \
-d '{
    "sql": "SELECT alert__date from gadm28_adm1 limit 2"
}'

Using the GET request is the recommended approach, as it allows HTTP caching of your result - subsequent requests for the same query will see a great performance increase, even if they are made by a different application or client.

Alternatively, you can also query a dataset using a POST request. POST requests are not cached, so you will not benefit from these speed improvements. However, GET requests can sometimes hit URL length restrictions, should your query string be too long. Using a POST request is the recommended solution for these cases. See the example on the side to see how you can query a dataset with a POST request.

Dataset id as the FROM clause

Three different but equivalent syntaxes for the same query:

curl -i -X GET 'https://api.resourcewatch.org/v1/query/098b33df-6871-4e53-a5ff-b56a7d989f9a?sql=SELECT cartodb_id, iso, name_0, name_1, type_1 FROM gadm28_adm1 limit 10' \
-H "x-api-key: <your-api-key>"

curl -i -X GET 'https://api.resourcewatch.org/v1/query?sql=SELECT cartodb_id, iso, name_0, name_1, type_1 FROM 098b33df-6871-4e53-a5ff-b56a7d989f9a limit 10' \
-H "x-api-key: <your-api-key>"

curl -i -X POST 'https://api.resourcewatch.org/v1/query' \
-H "x-api-key: <your-api-key>" \
-H 'Content-Type: application/json' \
-d '{
    "sql": "SELECT cartodb_id, iso, name_0, name_1, type_1 FROM 098b33df-6871-4e53-a5ff-b56a7d989f9a limit 10"
}'

The examples we've seen so far expect the URL to have the /query/<dataset id or slug>?sql=SELECT * FROM <dataset.tableName> format. However, you can also use the equivalent /query?sql=SELECT * FROM <dataset id> syntax. You can also use this alternative syntax with POST requests.

Redundant FROM clause (document based datasets only)

Example query providing a document-based dataset id in the request path or as the FROM clause:

curl -i -X GET 'https://api.resourcewatch.org/v1/query?sql=SELECT alert__date FROM 9be3bf63-97fc-4bb0-b913-775ccae3cf9e limit 10' \
-H "x-api-key: <your-api-key>"
curl -i -X GET 'https://api.resourcewatch.org/v1/query/9be3bf63-97fc-4bb0-b913-775ccae3cf9e?sql=SELECT alert__date FROM data limit 10' \
-H "x-api-key: <your-api-key>"

When querying a document based dataset using either GET or POST /query/<dataset id or slug> request, the FROM clause is required but ignored, meaning you don't have to provide the dataset's tableName as you normally would. The example on the side illustrates this.

Downloading data from a dataset

Structure of the endpoint for downloading the results of a query:

curl -i -X GET 'https://api.resourcewatch.org/v1/download/<dataset.id>?sql=SELECT * FROM <dataset.tableName>' \
-H "x-api-key: <your-api-key>"

Example endpoint for downloading the results of a query:

curl -i -X GET 'https://api.resourcewatch.org/v1/download/098b33df-6871-4e53-a5ff-b56a7d989f9a?sql=SELECT cartodb_id, iso, name_0, name_1, type_1 FROM gadm28_adm1 limit 10' \
-H "x-api-key: <your-api-key>"

The download endpoint allows you to download the results of the execution of a query over a dataset. This endpoint is greatly based on the query datasets endpoint, so we strongly suggest you read that section of the documentation.

Note: Some dataset providers do not support downloading query results. You can download query results for the following dataset providers:

• Google Earth Engine
• Document-based datasets
• Carto
• GFW
• BigQuery
• ArcGIS FeatureService

Like when querying datasets, in order to download the results of the execution of query, you'll need two pieces of information:

• The id of the dataset you're trying to download the query execution results.
• The SQL query that represents the data you are trying to download.

The dataset documentation covers different ways that you can use to browse the existing dataset catalog or upload your own, all of which will give you the details of a dataset, including the dataset id you'll need to query it.

The SQL query will have to be custom built by you to fit your needs, but a good starting point for newcomers would be something like SELECT * FROM <dataset.tableName> limit 10.

Notice: the limit parameter restricts our results to 10 rows, and is not required. However, for our learning purposes, this is useful, as it keeps the API responses small and fast.

As with the query endpoint, the FROM clause should reference the dataset's tableName value, which you can also get from the dataset documentation described above. And also, don't forget that you can check the support provided for the dataset type you are using if you are having trouble writing your SQL query.

Download response body

Example of downloading query results (by default, CSV data is assumed):

curl -i -X GET 'https://api.resourcewatch.org/v1/download/9be3bf63-97fc-4bb0-b913-775ccae3cf9e?sql=SELECT alert__date, alert__count from gadm28_adm1 limit 2' \
-H "x-api-key: <your-api-key>"

Example CSV response:

"alert__date",
"alert__count",
"_id"
"2019-04-12",
5,
"AW6O0fqMLu2ttL7ZDM4P"
"2015-08-22",
6,
"AW6O0fqMLu2ttL7ZDM4T"

Example of downloading query results requesting format as JSON:

curl -i -X GET 'https://api.resourcewatch.org/v1/download/9be3bf63-97fc-4bb0-b913-775ccae3cf9e?sql=SELECT alert__date, alert__count from gadm28_adm1 limit 2&format=json' \
-H "x-api-key: <your-api-key>"

Example JSON response:

{
  "data": [
    {
      "alert__date": "2019-04-12",
      "alert__count": 5,
      "_id": "AW6O0fqMLu2ttL7ZDM4P"
    },
    {
      "alert__date": "2015-08-22",
      "alert__count": 6,
      "_id": "AW6O0fqMLu2ttL7ZDM4T"
    }
  ]
}

The response body of executing the download endpoint will contain the data to be downloaded. You can use the format query parameter to customize the format of the data returned. By default, format=csv will be assumed, so you will receive the corresponding query results the actual CSV data in the response body. If you provide format=json, the returned result will be a JSON object with a data index containing the results of the execution of the query provided.

Download execution errors

Calling the download endpoint might sometimes result in an error being returned. The following table describes the possible errors that can occur when downloading query execution results:

Download endpoint parameters

Example of requesting to freeze the download results:

curl -i -X GET 'https://api.resourcewatch.org/v1/download/9be3bf63-97fc-4bb0-b913-775ccae3cf9e?sql=SELECT alert__date from gadm28_adm1 limit 2&freeze=true' \
-H "x-api-key: <your-api-key>" \
-H "Authorization: Bearer <your-token>" \
-H "x-api-key: <your-api-key>"

Example response:

{
  "url": "https://storage.googleapis.com/query-freeze/1589458072773.json"
}

You can use the following query parameters to customize the output of the download query execution results endpoint:

Filter download results by geostore

Example download request providing a geostore id as query parameter to filter the results:

curl -i -X GET 'https://api.resourcewatch.org/v1/download/1d7085f7-11c7-4eaf-a29a-5a4de57d010e?sql=SELECT * FROM dis_001_significant_earthquakes LIMIT 5&geostore=972c24e1da2c2baacc7572ee9501abdc' \
-H "x-api-key: <your-api-key>"

Some dataset providers support receiving a geostore query parameter. When providing this parameter, you can request geo-referenced data that fits within the bounding box of the geostore with id provided. You can obtain the id of the geostore using the RW API Geostore API. If the data is not geo-referenced, or if the dataset provider does not support the geostore query parameter, it will be ignored.

The following providers support this parameter:

• CartoDB (carto)
• Global Forest Watch (gfw)
• ArcGIS (featureservice)
• Google Earth Engine (gee)
• BigQuery (bigquery)
• Rasdaman (rasdaman)
• NEX-GDDP (nexgddp)
• Loca (loca)

Alternative ways for downloading query execution results

As in the case of querying datasets, there are some alternative ways that you can use for downloading query execution results. While the GET request described above is the recommended way of downloading query results, there are other ways to download query results that may be more suited for specific use cases.

Using the dataset slug instead of the id

Example request for downloading the query execution results not using the dataset id in the request path, and using the dataset slug in the FROM clause:

curl -i -X GET 'https://api.resourcewatch.org/v1/download/9be3bf63-97fc-4bb0-b913-775ccae3cf9e?sql=SELECT alert__date from gadm28_adm1 limit 2' \
-H "x-api-key: <your-api-key>"
curl -i -X GET 'https://api.resourcewatch.org/v1/download/Glad-Alerts-Daily-Geostore-User-Areas_3?sql=SELECT alert__date from gadm28_adm1 limit 2' \
-H "x-api-key: <your-api-key>"

When referencing a dataset's id in th SQL query, you have the option to use the dataset's slug instead, obtaining the same result. This is also applicable to the alternative download methods described in the sections below.

POST requests

The same download request executed as GET, and as a POST request providing the SQL as request body param:

curl -i -X GET 'https://api.resourcewatch.org/v1/download/9be3bf63-97fc-4bb0-b913-775ccae3cf9e?sql=SELECT alert__date from gadm28_adm1 limit 2' \
-H "x-api-key: <your-api-key>"
curl -i -X POST 'https://api.resourcewatch.org/v1/download/9be3bf63-97fc-4bb0-b913-775ccae3cf9e' \
-H "x-api-key: <your-api-key>" \
-H 'Content-Type: application/json' \
-d '{
    "sql": "SELECT alert__date from gadm28_adm1 limit 2"
}'

Using the GET request is the recommended approach, as it allows HTTP caching of your result - subsequent requests for the same download endpoint call will see a great performance increase, even if they are made by a different application or client.

Alternatively, you can also download the query results using a POST request. POST requests are not cached, so you will not benefit from these speed improvements. However, GET requests can sometimes hit URL length restrictions, should your SQL query string be too long. Using a POST request is the recommended solution for these cases. See the example on the side to see how you can download the query execution results with a POST request.

Dataset id as the FROM clause

Three different but equivalent syntaxes for the same call to the download endpoint:

curl -i -X GET 'https://api.resourcewatch.org/v1/download/098b33df-6871-4e53-a5ff-b56a7d989f9a?sql=SELECT cartodb_id, iso, name_0, name_1, type_1 FROM gadm28_adm1 limit 10' \
-H "x-api-key: <your-api-key>"

curl -i -X GET 'https://api.resourcewatch.org/v1/download?sql=SELECT cartodb_id, iso, name_0, name_1, type_1 FROM 098b33df-6871-4e53-a5ff-b56a7d989f9a limit 10' \
-H "x-api-key: <your-api-key>"

curl -i -X POST 'https://api.resourcewatch.org/v1/download' \
-H "x-api-key: <your-api-key>" \
-H 'Content-Type: application/json' \
-d '{
    "sql": "SELECT cartodb_id, iso, name_0, name_1, type_1 FROM 098b33df-6871-4e53-a5ff-b56a7d989f9a limit 10"
}'

The examples we've seen so far expect the URL to have the /download/<dataset id or slug>?sql=SELECT * FROM <dataset.tableName> format. However, you can also use the equivalent /download?sql=SELECT * FROM <dataset id> syntax. This alternative syntax is also available for POST requests.

Redundant FROM clause (document-based datasets only)

Example download providing a document-based dataset id in the request path or as the FROM clause:

curl -i -X GET 'https://api.resourcewatch.org/v1/download?sql=SELECT alert__date FROM 9be3bf63-97fc-4bb0-b913-775ccae3cf9e limit 10' \
-H "x-api-key: <your-api-key>"
curl -i -X GET 'https://api.resourcewatch.org/v1/download/9be3bf63-97fc-4bb0-b913-775ccae3cf9e?sql=SELECT alert__date FROM data limit 10' \
-H "x-api-key: <your-api-key>"

When downloading the query results for a document based dataset using either GET or POST /download/<dataset id or slug> request, the FROM clause is required but ignored, meaning you don't have to provide the dataset's tableName as you normally would. The example on the side illustrates this.

Deleting data from a dataset

Example requests to delete data from a dataset:

curl -i -X GET 'https://api.resourcewatch.org/v1/query/:dataset_id?sql=DELETE FROM index_bf86b945c4ec41d2b5b7af00f3f61423' \
-H "x-api-key: <your-api-key>"
curl -i -X GET 'https://api.resourcewatch.org/v1/query/:dataset_id?sql=DELETE FROM index_bf86b945c4ec41d2b5b7af00f3f61423 WHERE x = "y"' \
-H "x-api-key: <your-api-key>"

Write queries such as INSERT or UPDATE are not supported in the RW API. You can use dataset endpoints to append or overwrite a given dataset's data, but you cannot use SQL to write data into the datasets.

Most providers do not support DELETE queries either. However, in the case of document-based datasets (i.e. where the connectorType is document), you can delete the dataset's data via SQL DELETE query. Executing a DELETE query requires authentication, and additionally, one of the following conditions must be met:

• have role MANAGER and own the dataset;
• have role ADMIN and belong to all the apps to which the dataset is associated.

If the query is successfully executed, the request will return an HTTP response with status code 204 No Content. Please note that executing a delete query is an asynchronous process - as in the case of appending or overwriting a dataset's data, the dataset will have its status updated to pending, and updated once again to saved once the deletion process is completed.

Delete query execution errors

Supported SQL syntax reference

This section details the support for SQL syntax for the different dataset providers RW API supports. Keep in mind that your HTTP request parameters (like sql) should always be escaped according to the official HTTP specification. You can use this online tool as an example of proper URL parameter encoding.

CartoDB datasets

This section describes the SQL support for querying datasets with provider cartodb.

CartoDB geo-spatial query support

CartoDB datasets can be queried using PostGIS functions. This means if your dataset contains geo-referenced data, you can execute PostGIS functions on the data to extract the information you need. The table below displays some examples of supported PostGIS functions:

GFW Datasets

This section describes the SQL support for querying datasets with provider gfw.

Note: This table was generated automatically with the help of this repository. If you are maintaining the docs, please do not edit manually these tables.

ArcGIS Feature Service datasets

This section describes the SQL support for querying datasets with connector type rest and provider featureservice.