Quantcast
Channel: Developer topics
Viewing all articles
Browse latest Browse all 17897

Too many requests in Web.Contents. Can I share a request?

$
0
0

Hi,

 

I am querying a web api using Web.Contents in a loop to get paged data. The Api has a throttled request limit of 300 requests per 10 minutes.

 

 

{
    "results": {
        "timesheets": {
            "123456": {
                "id": "123456",
                "job_code": "0001",
                "user": "9999"
            }
        }
    },
    "more": "true",
    "supplemental_data": {
        "job_codes": {
            "0001": {
                "name": "Job 1"
            }
        },
        "users": {
            "9999": {
                "name": "Trevor"
            }
        }
    }
}

I use List.Generate, combined with Web.Contents to get all of the data for a specific date range, using the "more" value to determine if more pages are available.

 

To model, The "timesheets" records become my facts, and the supplemental data becomes my dimensions. The main query creates my fact table, then I reference that query and transform to extract my users and jobcodes dimensions.

 

By referencing the main query, I wind up making the paged Api calls 3x (one for each table). Ideally, I'd like to have the Api called only once, and populate my 3 tables without having to go back to the source. Is there any way to "cache" the Api requests, and reuse them as the source for multiple queries?

 


Viewing all articles
Browse latest Browse all 17897

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>