Auto-refresh a Google Sheet from Databricks every night

Every night on a cron schedule (default 2:00 AM), run a saved SQL query against my Databricks SQL warehouse and mirror the results into a Google Sheet, so my team always has a fresh, no-login copy of the numbers. This is a deterministic data sync with no judgement involved, so build it as a code workflow.

Use the Databricks Execute SQL Statement operation, passing my configured warehouse and SQL statement. Capture the returned statement_id, then poll the Databricks Get SQL Statement operation until status.state is SUCCEEDED. While the state is PENDING or RUNNING, wait and poll again; if it comes back FAILED, CANCELED, or CLOSED, stop the run and surface the error instead of writing partial data to the sheet.

Read the column names from the result schema (manifest) to build the header row. When the result set is large and split across chunks, page through it with the Databricks Get SQL Statement Result Chunk operation, following the chunk index and next-chunk link until every chunk has been fetched, and concatenate the rows in order so no data is dropped.

By default (snapshot mode), write the results into a specific tab using the Google Sheets Update Values operation so the target range is overwritten with the latest snapshot on every run. Write the column headers as the first row, then all data rows beneath them, into the configured spreadsheet, tab, and A1 range.

Also offer an append-only history mode as a configurable option: instead of overwriting, use the Google Sheets Append Values operation to add the latest rows after the last row of the tab so the team keeps a running history rather than only the newest snapshot. In append mode, only write the header row when the target tab is still empty.

Make the SQL query, the Databricks warehouse, the destination spreadsheet, the tab name, the A1 range, and the write mode (snapshot overwrite vs append history) all configurable inputs, so I can point the same workflow at different reports without editing it. The schedule should also be configurable so I can run it nightly, hourly, or only on business days.