Case study
An internal knowledge application for drafting security questionnaire and RFP answers from approved company documents, with citations and human review.
Problem
Security and sales teams repeatedly answer the same questions, while approved wording is scattered across policies, previous questionnaires and product documents. A plausible answer without a source creates review and compliance risk.
My work
I implemented document ingestion, background processing, hybrid retrieval, cited answer generation, organisation-scoped APIs, review states, audit records, the TypeScript interface and automated tests.
Implementation
Documents are validated and stored, then split and processed by a Python worker. Qdrant supplies semantic candidates; PostgreSQL full-text search recovers exact policy language. FastAPI returns source excerpts with each draft. Approval and access control remain application state, not model output.
Hybrid retrieval
Qdrant vector search is combined with PostgreSQL full-text ranking so exact policy language is not lost to semantic similarity alone. Thresholds and source limits make the behaviour tunable, but they also require dataset-specific evaluation.
Application controls
Keycloak authenticates users; PostgreSQL records their organisation and role. Queries remain organisation-scoped, while approval states and audit records are ordinary application data. This adds workflow steps, but a model response cannot bypass access control or approve itself.
Failure handling
If the retrieved documents do not support an answer, Evidella returns an explicit missing-source result instead of completing the draft. A human can supply the document and run retrieval again; no answer is approved automatically.
Verification