Building a Developer AI Assistant with CrewAI and Dagster

‌

‌

A coding assistant that thinks, plans, and executes alongside you isn't just sci-fi anymore—it's achievable with the right architecture.

I recently built one to manage my development backlog and automate repetitive data tasks. Here's exactly how it works.

🏗️ The Architecture

✅ CrewAI: Orchestrates specialized LLM agents (scrapers, classifiers, summarizers)
✅ Dagster: Manages pipeline execution, scheduling, retries, and observability
✅ Neo4j: Handles entity relationship graphs
✅ Postgres: Stores raw data and job state
✅ React: Displays agent outputs and system status

The Execution Flow

1. Dagster triggers the scheduled ingest job at 4am
2. CrewAI spins up specialized agents (ScraperAgent, ClassifierAgent, SummarizerAgent)
3. Agents process independently, passing structured JSON through Dagster's I/O manager
4. Neo4j receives entity relationships while Postgres stores the raw data
5. The React frontend presents the completed results

🧠 The Secret Sauce

The architecture works because of radical decoupling. Each CrewAI agent handles one narrow task exceptionally well while Dagster manages overall coordination and failure recovery. This separation creates both resilience and visibility into what's happening at each step.

Takeaway: You don’t need one superagent. You need many simple agents orchestrated well.