Self-hosting a voice-agent evaluation system gives your team control over the application environment, database, credentials, and deployment process. It does not automatically make every data flow private or make a deployment compliant with a particular framework.
The useful question is more specific: which components do you operate, where does conversation data travel, and which security and reliability responsibilities move to your team?
VaaniEval's self-hosted architecture
VaaniEval is a full-stack application rather than a library or command-line tool. The repository contains a React review workspace, a FastAPI service, a database-backed queue, a worker, database migrations, and adapters for supported voice and evaluator providers.
A typical evaluation flow has six steps:
- A reviewer connects a supported voice provider.
- The API creates an import job.
- The worker fetches and normalizes conversation transcripts, metadata, and available media references.
- An evaluation job sends conversation context to the configured evaluator provider.
- Metric scores and rationales are stored with the conversation.
- The product workspace presents conversations, evidence, scores, and aggregate trends.
SQLite is available for local use, while the documented production topology uses PostgreSQL shared by the API and worker. The queue is database-backed, so workers need consistent database access and jobs must remain safe to retry.
What you control
Running VaaniEval in your own environment lets your team choose the application host, network boundaries, database, secrets, allowed origins, provider accounts, evaluator endpoint, backup process, and operational access rules.
The current backend encrypts connected provider credentials before storage and scopes API reads and writes to an authenticated workspace. Production configuration also supports secure cookie settings and explicit allowed origins.
These are useful building blocks, not a complete security program. Your deployment still needs appropriately managed secrets, database encryption and backups, access controls, monitoring, patching, retention rules, and an incident-response process.
Data that can leave your environment
Self-hosting does not mean conversation data never reaches an external service. The worker communicates with configured voice providers to import calls, and evaluation jobs send conversation context to the evaluator provider selected by the operator.
Before using production data, document each external destination and verify the provider configuration, contract, retention behavior, and permissions that apply to your use case. If a workflow requires stricter boundaries, confirm that the selected providers and deployment topology can support them rather than assuming the application alone provides that guarantee.
Operational tradeoffs
Self-hosting replaces some vendor dependency with internal operational work:
- Maintenance: your team owns upgrades, database migrations, dependency patches, and compatibility testing.
- Reliability: the API, worker, and database need health checks, backups, recovery procedures, and enough capacity for import and evaluation jobs.
- Cost: infrastructure spend may be predictable, but engineering and on-call time can exceed the hosting bill.
- Security: direct control reduces one layer of access, while also making your team responsible for configuration errors, key rotation, auditability, and incident response.
- Scaling: larger call volumes increase database growth, queue depth, provider API usage, evaluator cost, and the need for worker concurrency controls.
For a small evaluation program, a single API, worker, and PostgreSQL database may be enough. As volume grows, measure job latency, retries, database load, media access patterns, and evaluator rate limits before adding workers. More concurrency without those controls can amplify provider throttling and duplicate operational load.
A practical deployment review
Before putting production conversations through a self-hosted evaluation system, verify:
- The API and worker use the same production database.
- Development secrets have been replaced and provider credentials can be rotated.
- Secure cookies, allowed origins, ingress, and administrator access match the intended environment.
- Database backups and migration recovery have been tested.
- Voice-provider and evaluator data flows are documented.
- Retention and deletion behavior covers transcripts, media references, evaluation results, logs, and backups.
- Queue retries, failed jobs, and provider rate limits are observable.
When self-hosting is a good fit
Self-hosting is most useful when a team needs direct control over deployment and data architecture and has the engineering capacity to operate the stack. A managed path may be more practical when speed and low operational overhead matter more than infrastructure control.
VaaniEval is open source under the GNU Affero General Public License v3.0 (AGPL-3.0-only). You may use, modify, and deploy it under that license; if you modify it and offer it to users over a network, you must offer those users the corresponding source for your version.
Explore the open-source VaaniEval architecture