Available for new work

Building in public — Docent, an autonomous support agent for pydantic/pydantic.

Emanuele Czofei

AI Product Engineer

Building production LLM systems · RAG · Agents · Python + TypeScript

I find the failure your eval is hiding — a TNR of 0.00, a miscalibrated threshold — then rebuild the judge until the numbers are real.

Basque Country, Spain · Open to remote (EU / US via EOR)

What I build

Three things I do at production depth.

01

RAG systems with real eval harnesses

Retrieval pipelines built alongside the evaluation infrastructure that proves they work — not just demos that look right.

02

Full-stack AI products

Python backends and Next.js / TypeScript frontends, owned end to end — from retrieval logic to the interface users actually touch.

03

LLM evaluation infrastructure

Retrieval metrics, LLM-as-judge, and meta-evaluation — the tooling that tells you whether the judge itself can be trusted.

Projects

The work, with the receipts.

Building now

Docent — Autonomous OSS Support-Engineer Agent

Agent System · pydantic/pydantic
View repo

An autonomous support agent for the pydantic/pydantic repository. It triages incoming issues, researches them against a hybrid RAG corpus of ~1,500 resolved issues, and drafts grounded, citation-backed replies — or escalates to a human when the quality gate says it shouldn't answer.

PythonFastAPILangGraphCeleryPostgres/pgvectorRedisMCPLangfuseNext.js
The architecture

Five-phase pipeline: INTAKE → RESEARCH → DRAFT → GATE → READY/ESCALATE. MCP implemented as both client and authored server. Celery Beat scheduled ingestion, prompt + semantic caching, per-request cost metering, circuit breaker with degraded mode, prompt-injection defense.

Spanish Safety RAG Assistant

View repo

A RAG assistant over the Spanish INSHT / NTP occupational-safety corpus. It answers regulatory queries with retrieved evidence and grounded generation — every claim traceable to a source.

PythonLangChainChromaDBVoyageAI voyage-3rerank-2.5Claude Haiku APINext.jsTypeScript
The differentiator

The judge initially showed TNR 0.00 due to threshold miscalibration and missing reference answers. I diagnosed the failure, rebuilt the judge with reference-aware prompting and explicit fault enumeration, and validated the fix.

eval · results
FaithfulnessTPR100%TNR100%
CompletenessTPR100%TNR87.5%
Hit@8→ 0.98
MRR→ 0.955
Precision@8→ 0.69
Golden set50 human-labeled items

Triage — Automated Code-Diagnosis Agent

Full-Stack Agent · Personal Project
View repo

A full-stack agent (RepoDoctor) that clones any repo, auto-detects its stack (Python / Node + package manager), installs dependencies, and runs its own test / build / lint suites to diagnose and explain what's failing. Built as a LangGraph state machine — clone → detect → install → diagnose → report — with every claim grounded in real command output. Each repo runs in an isolated sandbox with async subprocess execution and per-command timeouts; the Next.js frontend streams progress live via server-sent events.

PythonLangGraphOpenAI APINext.jsReactTypeScriptDocker Compose

SlideSpeak — Async Conversion Service

Technical Challenge · SlideSpeak
View repo

Fast-tracked past intermediate interview rounds. Built a PowerPoint-to-PDF async conversion service: upload triggers a Celery worker via Redis, LibreOffice converts server-side, the result lands in S3, the frontend polls for completion.

ReactTailwind CSSFastAPICeleryRedisAmazon S3LibreOffice

Skills

The stack I reach for.

LLM Systems
RAG pipelineshybrid retrieval (BM25 + dense)cross-encoder rerankingLLM-as-judgejudge meta-evaluationeval harnessesMCP (client + authored server)prompt & semantic cachingprompt engineering
Backend
PythonFastAPILangChainLangGraphPostgreSQL / pgvectorCeleryRedisChromaDBVoyageAIDocker
Frontend
Next.jsTypeScriptReactTailwind CSSSSE streaming
Infra
GitHub Actions (eval gates)RailwayVercelAWS S3

Experience

Where I've shipped.

Independent AI Engineer

2024 — Present
Remote · Gipuzkoa, Spain

Design, build and ship production LLM systems end to end — retrieval architecture, agent orchestration, evaluation harnesses, deployed interfaces. Selected work below.

Technical Challenge — SlideSpeak

2025
AI presentation startup

Built an async PowerPoint-to-PDF conversion service under time constraint. Fast-tracked past intermediate interview rounds.

Fullstack AI Engineer Intern — Gaddr

Sept — Dec 2025
Stockholm, Sweden · Remote

Built authentication flows in Next.js / TypeScript and integrated early API features into the product.

Contact

Building something in AI? Let's talk.