Making influencer content easier to discover
AI Topic Search is a project I built at Cloudbreakr to turn influencer captions and video transcripts into useful, searchable topics. It brings together content preparation, topic tagging, theme summaries, and semantic retrieval across Instagram and YouTube.
The work covers the full journey from source data to search results: extracting records from MySQL, preparing exports, processing text, storing results in MongoDB, generating embeddings, and retrieving relevant posts.
Starting with reliable inputs
Before generating tags, I needed to make the source content consistent enough to process. I built extraction tools with retries and type-safe cursors, alongside structured exports for reviewing influencer and post datasets.
The preparation stage cleans captions and transcripts, fixes Unicode issues, removes control characters and emoji, and de-duplicates posts. Input validation and meaningful-text checks help prevent empty or unusable content from becoming unnecessary processing requests.
Two ways to process content
For larger workloads, I implemented chunked, resumable OpenAI Batch API jobs. JSONL sharding keeps the work manageable, while preflight validation and duplicate-request detection catch problems before submission. Diagnostics make it easier to understand which requests need attention.
For work that needs a more immediate response, I built an asynchronous Python tagger using asyncio. It manages rate limits through adaptive concurrency and backoff, and records timing and token usage per row and entity.
Both modes feed the same broader goal: generating English topic tags and summarizing themes in a form that can be stored, reviewed, and searched.
Making long-running jobs recoverable
I developed a MongoDB ingestion CLI with duplicate-safe upserts, per-file progress, and resume checkpoints. SQL side-lookups bring in supporting engagement data such as likes, comments, and follower counts.
Atomic checkpointing and configurable processing limits help make interrupted runs recoverable. These tools were designed with Windows and managed MongoDB environments in mind, where a reliable restart matters as much as a successful first run.
Searching by meaning
The embedding stage batches requests, retries failures, writes results in bulk, and exposes progress and estimated completion time. It also supports controlled overwrites when embeddings need to be regenerated.
I implemented two retrieval paths:
- Atlas Vector Search: server-side retrieval using MongoDB's
$vectorSearchpipeline with projected similarity scores. - Cosine similarity fallback: batched MongoDB reads and top-K calculations in Python for self-hosted environments without Atlas Vector Search.
I also built a lightweight two-tier classifier using lexical Jaccard similarity and embedding thresholds, with cached embeddings and support for local or hosted embedding backends.
Building tools people can operate
Interactive CLIs, progress bars, clear configuration, and analyst-ready Excel exports are part of the project, alongside validators and timing counters. They make the pipeline easier to inspect and operate without treating every run as a debugging exercise.
This project brought together the areas I enjoy most: backend engineering, data quality, and practical search. It reinforced how much useful retrieval depends on the less visible work of cleaning inputs, handling interruptions, and making each stage understandable.
