SPERO
AI Reliability Engineering
Ship AI features you can actually measure.
Your LLM feature works in the demo. Then someone changes a prompt, the model provider ships an update, and nobody knows what broke until a customer does. I build the evals, regression tests and release gates that make AI behave like the rest of your software.
15+ years in QA and test automation 路 Master's in Computer Engineering 路 4-8 hours overlapping with any US time zone
Most AI features ship on vibes.
Prompt changes are blind.
Someone tweaks a system prompt to fix one case. Three other cases quietly break. No test catches it because there is no test.
Model updates change behavior.
Your provider ships a new version, or you switch to a cheaper model to cut costs. Output quality shifts and you find out from support tickets.
"Looks good to me" is the eval.
An engineer reads twenty outputs, they seem fine, it ships. That's not a quality process. It's a coin flip with extra steps.
What I build
AI Reliability Audit
A two-week review of your LLM features. I map failure modes, test your current prompts against real edge cases and hand you a prioritized risk report with a concrete fix plan.
Eval Suites & LLM Regression Testing
Golden datasets, automated evals (deterministic checks, calibrated LLM-as-judge, human review loops) and a regression suite that runs on every prompt or model change.
Release Gates in CI
Evals wired into your pipeline. A prompt change that drops quality below your threshold doesn't merge. Same for model swaps.
Agent & RAG Testing
Tool-call correctness, retrieval quality, hallucination checks, cost and latency budgets. Tested per step, not just on the final answer.
Test Automation for the rest of the product
E2E, API and integration testing with Playwright and modern tooling. Because your AI feature sits inside an app that also has to work.
Already have an eval tool? Good. I implement it.
Braintrust, Langfuse and promptfoo are great tools. A tool doesn't decide what to test, build the golden dataset, calibrate the judge against human reviewers or wire the gate into your release process. That's the work I do.
I'm tool-agnostic. If you already use one, I build on top of it. If you don't, I help you pick the one that fits your stack, and everything stays in your account.
Building an eval or observability tool? I'm open to implementation partnerships.
How an engagement works
- 01
Reliability review 路 30 min, free
I look at one AI feature with you and tell you honestly whether I can help. If I can't, I say so.
- 02
Audit 路 2 weeks, fixed price
Failure mode map, baseline eval scores, risk report, fix plan. You own everything I produce, even if you stop there.
- 03
Build 路 3 to 6 weeks
I implement the eval suite and CI gates inside your repo, working with your engineers, not around them.
After the audit
The audit shows which of these comes next. Same core every time: evals, tests and release gates.
Model Migration Assurance
Pays for itselfFixed project 路 2 to 3 weeks
Moving to a cheaper model or another provider? I run your golden dataset on both, compare quality, cost and latency per use case and tell you where it's safe to switch and where it isn't.
AI Red Teaming + Evidence Pack
Fixed project 路 paid quarterly retest
Prompt injection, jailbreaks, PII leaks, agent tool misuse. You get a test evidence report ready for security questionnaires, SOC 2, HIPAA and enterprise procurement. QA evidence, not a pentest.
Fractional AI Quality Lead
Monthly 路 defined deliverables
Your part-time head of AI quality until you hire one. I review prompt and model changes, maintain datasets, set metrics and train your team.
Why teams work with me
I come from QA, not from AI hype.
15+ years breaking software before customers do. I treat LLMs as one more system that needs tests, metrics and gates.
A senior engineer, directly.
You talk to the person who writes the code. No account managers, no juniors learning on your repo.
Your time zone.
Based in Brazil, 4-8 hours overlapping with any US time zone. Daily async updates, calls when you need them.
You own the work.
Everything lives in your repo, in your CI, documented. No lock-in, no proprietary platform.
15+
years in QA and test automation
2 weeks
from kickoff to prioritized risk report
0
API keys needed to run my public eval harness in CI
100%
of the work delivered in your repo
Don't take my word for it. Read the code.
Every repository below is public. Clone it, run it and judge the engineering yourself before you book a call.
Eval harness 路 TypeScript
llm-eval-guardrails
A test harness for LLM, RAG and agentic apps. Semantic, schema and groundedness assertions, confidence metrics, snapshot regression detection and a CI gate that blocks the deploy when quality drops.
- Runs fully offline by default, no keys or services
- Plugs into real models and vector stores through env vars
AI for testing 路 TypeScript
ai-test-generator
A CLI that crawls a page with Playwright and uses a two-step prompt pipeline to generate typed, executable Playwright tests, including negative paths and edge cases.
- Prompts isolated for audit and iteration
- GitHub Actions workflow that generates and runs the suite
Public analysis 路 Java
realworld-api-automation
One API suite written in three Java stacks (TestNG with HttpClient, Spring RestClient, REST Assured) against a public reference app.
- Ten documented contract gaps in the target API
- Side by side comparison of the three approaches
E2E comparison 路 TypeScript
realworld-ui-automation
The same end to end suite in Cypress, Playwright and Selenium against cypress-realworld-app, so the trade-offs are visible in real code.
- Identical scenarios across three frameworks
- Open for anyone to clone and run
Models and tools I test with
Recommendations
What people I worked with say
Douglas was our first software engineer in test hire at Olly Olly. He was instrumental in getting our testing discipline off the ground. Leveraging his years of experience in DevOps and Test Automation, Douglas helped guide the creation of our testing framework, which was challenging because there were no proven approaches for our stack. Without Douglas, our testing discipline would be severely behind. If you have a unique test automation problem to solve, Douglas has the skills and experience to analyze your situation and build a thorough, scalable solution.
Benjamin De Point
Software Engineering Executive
Managed Douglas directly 路 Mar 2026
Proximo hired Douglas when we were looking for a QA expert to help jump start a quality testing automation project for one of our platforms. I and the rest of the team were impressed with Douglas and brought him on. He was definitely the right choice, as he had us up and running in less than two months. He worked with existing team members and always kept in mind what we were trying to accomplish. He's also very pleasant and energetic, and our people liked working with him. I'd wholeheartedly recommend Douglas for any quality assurance work.
David Ricciardi
CEO of Proximo
Hired Douglas 路 Oct 2021
I had the pleasure of working with Douglas on our team. He consistently demonstrated exceptional attention to detail, a strong commitment to quality and an evangelist for agile best practices that significantly enhanced our product reliability. He is a true asset to any team, bringing not only his technical skills around automation, but also a collaborative spirit that makes working together a pleasure.
Devin Lam
Lead Product Manager
Worked on the same team 路 Apr 2024
I wholeheartedly recommend Douglas for the position of Quality Engineer. During our time working together, he not only excelled in executing quality assurance protocols but also actively contributed ideas for process improvements, displaying a proactive approach to problem-solving. His ability to analyze complex systems and identify potential areas for enhancement greatly contributed to our team's success. I am confident that Douglas will bring the same dedication, expertise, and collaborative spirit to any role in Quality Engineering.
Tapesh Nagarwal
Senior Specialist, Quality Engineer | Founder, TapQuality AI LLC
Managed Douglas directly 路 Apr 2024
I worked with Douglas for a few months during his time at Fabric. He has a lot of knowledge about quality systems and how to bring quality first thinking into your applications. He is great at coming up with new processes that are documented in detail about how you can improve quality and sets up process to measure and check the quality of engineering regularly. Douglas is also willing to take on new things and learn more about new tools and is a good team player!
Pranay Karnany
Manager at Fabric
Managed Douglas directly 路 Oct 2021
Douglas has a natural drive to solve problems. He is really focused and has a broad technical background in software engineering, ranging from dynamic programming and testing to data analysis. Besides that, he always shares his knowledge with colleagues, creating a learning environment wherever he is. If you have a tough problem that needs to be solved, it's better to call Douglas.
Gabriel Gomes
PhD | Applied Machine Learning | Signal Processing | Electrical Engineering
Worked with Douglas on different teams 路 Oct 2021
Douglas is an excellent collaborator. Always thinking ahead and proposing new ideas, he is a great team member. Our time working together was really fun.
Leonardo Kochake Gomes
DevOps Automation Engineer | SAFe庐 4 DevOps Practitioner
Worked on the same team 路 Oct 2021
An outstanding professional who spares no effort to deliver a final result of the highest quality.
Fernando Flor锚ncio da Mata
Owner at NIPN Neg贸cios Digitais
Reported to Douglas directly 路 Oct 2021
Start with the audit
AI Reliability Audit
Two weeks. Fixed price. A clear answer to "how reliable is our AI feature, and what do we fix first?"
From $4,500
- Failure mode map of up to 2 AI features
- Baseline eval scores on a starter golden dataset
- Prioritized risk report
- Implementation plan with effort estimates
- 60-min walkthrough with your team
FAQ
Stop guessing whether your AI works.
30 minutes, one AI feature, an honest assessment.
Book a reliability review