Hello!
The Setup:
- Each month I need to grab gov open data from a Socrata REST API and update an Azure SQL instance.
- There may be 2M rows! Or there may be 1000 rows. I need to hit several different datasets (may make a function for each).
- I would like to use Azure Functions that run once a month, grab data from socrata REST API, and insert it into Azure SQL
- I assume I can't hold 2M rows in memory for an Azure function
- That's okay though, because Socrata can paginate (up to 50k rows apparently)
- Not only is memory an issue with azure functions, but if I loop through paginated scorata results, the azure function may time out before it finishes? (just guessing)
- The final destionation of the Azure SQL data is a PowerBI app, where the app owns the data (ie dashboard shows up in a standalone node application).
What is the best solution here to get the data from Socrata into an Azure SQL? Can Azure Functions handle this situation alone?
What are the options here? It would be great to hear some different solutions, from quick to implement, to more involved solutions.