AI Knowledge Base Pipeline: Embedding Financial Data into Pinecone
A scheduled n8n pipeline pulls source data, chunks and embeds it with OpenAI, and keeps a Pinecone vector index current - the retrieval layer behind an AI agent that can actually answer questions on real data.
Reference data lived scattered across spreadsheets and web sources, with no way for an AI agent to search or reason over it beyond what fit in a single prompt.
A scheduled n8n workflow reads source rows from Google Sheets, pulls fresh content via Firecrawl, chunks the text, generates embeddings through the OpenAI API, and upserts everything into a Pinecone index - keeping the retrieval layer current automatically so any AI agent built on top of it has an accurate, searchable knowledge base.
Stack
How It Works
Scheduled pull
A trigger reads the source list of items to process from Google Sheets on a schedule.
Fetch and extract
Firecrawl retrieves and extracts clean content from each source.
Chunk and embed
Content is split into chunks and passed to the OpenAI API to generate vector embeddings.
Upsert to Pinecone
Embeddings are pushed into the Pinecone index, keeping the AI agent's knowledge base current without manual re-indexing.
Results
automatic re-indexing
embedding stack
orchestration
