Page Number With Total Records Based Pagination

In this pagination mechanism, the huge datasets are split into multiple pages.

The Rest API returns you a response with partial datasets in a single request and then again consecutive requests needs to be triggered in order to get the complete dataset.

Along with the dataset, the api provides other details that are required in order to fetch the next page, such as:

• Total number of records in the complete dataset
• Current page number

Apart from above details, user should be aware of the below details:

• Limitation on maximum number of records that api return per page
• The query param name for the next page

Once we know above details, user can configure pagination in the below manner. Consider we are getting below response from API:

{
  "data": {
    "total": {
      "numberItems": 3
    },
    "page": 1,
    "items": [
      {
        "col_A": "val_1",
        "col_B": "val_2",
        "col_C": "val_3"
      }
    ]
  }
}

In the above response:

• the total number of records in the complete dataset is represented by data.total.numberItems
• the current page number is represented by data.page
• assume that maximum number of records that api return per page is 1
• the query param name for next page is page

Then, the pagination details can be captured in the checkpoint file in the below manner:

    pagination = {
      type = "PageNumberBasedPagination"
      total-number-of-records-attribute = "$.data.total.numberItems"
      current-page-number-attribute = "$.data.page"
      max-records-per-page = 1
    }

The total-number-of-records-attribute and current-page-number-attribute are represented as JsonPath.

The complete example of checkpoint file including pagination is:

name = "checkpoint_page_number_with_total_records_based_pagination"

data = {
  data-request = {
    url = "http://localhost:8080/page_number_with_total_records_based_pagination"
  }

  data-response = {
    body = {
      type = "Text"
      text-format = {
        type = "JSONTextFormat"
        data-column-name = "data.items"
      }
    }

    pagination = {
      type = "PageNumberWithTotalRecordsBasedPagination"
      total-number-of-records-attribute = "$.data.total.numberItems"
      current-page-number-attribute = "$.data.page"
      max-records-per-page = 1
    }

    persistence = {
      type = "LocalFileSystem"
      file-format = {
        type = "ParquetFileFormat"
      }
      file-path = "/tmp/pagination"
    }
  }
}