Keep your Pinecone index in sync with your Notion wiki

Every night at 2am on a cron schedule, keep my Pinecone index in sync with my Notion knowledge base so that every assistant, search box, and RAG app I build on top of that index is working from current documentation.

Start by listing the pages to index. Use the Notion Search by Title operation with an empty query, which returns everything shared with the integration, and keep only results whose object type is page. Page through the results using has_more and next_cursor until there are no more. From each search result capture the page id, the page title, the page url, and last_edited_time, because those become the metadata later. Notion returns nothing for pages that have not been explicitly shared with the connection, so if the list comes back empty, report that clearly rather than treating it as a successful run over zero pages.

For each page, pull the body with Notion Retrieve Block Children, paginating the same way. Extract the plain text out of the rich_text of the readable block types (paragraph, the heading levels, bulleted and numbered list items, to do, toggle, quote, callout, and code) and join them in document order into a single string per page. Skip blocks with no text content such as images and dividers. If a block has children, fetch one level down so nested toggles and lists are not silently dropped. Pages that end up with no text at all should be skipped and counted separately.

Split each page into chunks of roughly 1000 to 1500 characters with a small overlap of about 100 characters. Break on paragraph boundaries where possible so sentences are not cut mid thought, and keep short pages as a single chunk. Number the chunks starting at 0 in document order.

Load the chunks into Pinecone. Pinecone data plane calls go to the per index host rather than api.pinecone.io, so resolve the host once at the start of the run with Describe Index and reuse it for the rest of the workflow. Write the chunks with Upsert Records into a dedicated namespace such as notion-kb. Upsert Records embeds the text server side, so no separate embedding provider is needed, but it requires an index created with an integrated embedding model, and the records endpoints send application/x-ndjson rather than plain JSON. The index dimension must match the output dimension of the embedding model the index was created with. Batch the records rather than sending one request per chunk.

Build each record id from the Notion page id, a # separator, and the chunk number, giving ids like 8a7f...c21#0 and 8a7f...c21#1. This is the load bearing detail of the whole workflow: because the ids are deterministic, a re-run overwrites the previous version of every chunk instead of appending a second copy, which is what keeps a nightly schedule from duplicating the entire knowledge base every night. The # separator also makes the page id a listable id prefix in Pinecone. Store the page title, the page url, and last_edited_time as metadata on every record, along with the page id and chunk number, so results can be filtered by recency or traced back to a specific page later.

Handle pages that get shorter. If a page used to produce eight chunks and now produces three, chunks 3 through 7 from the previous run would otherwise linger as stale content. After upserting a page, use List Vector IDs with the page id plus # as the prefix, and if any ids sit at or above the current chunk count, remove them with Delete Vectors. Optionally, the same idea extends to pages deleted or unshared in Notion, whose records stay in the index until something clears them out.

Finish by calling Describe Index Stats to read the new total record count, then post a short message to Slack with the Send a Message operation reporting how many pages were processed, how many chunks were written, and the new total record count in the index. Keep it to a couple of lines, for example: Notion to Pinecone sync complete. 42 pages processed, 318 sections indexed, index now holds 1,204 records.

A few practical constraints. Notion allows roughly three requests per second, so pace the page and block calls and respect the Retry-After header on a 429 instead of failing the run. Both Notion and Pinecone paginate, so never assume a single response is the complete set. If one page fails to fetch, log it, continue with the rest, and mention the failure count in the Slack message rather than aborting the whole sync, since a partial refresh is far more useful than none.