Calidad Infotech
    Insights

    What a Paid Software Vendor Pilot Needs

    A pitch shows how a vendor sells; a pilot shows how it builds. Here's how to scope a paid pilot, set success criteria up front, and make the scale-or-stop call.

    · 8 min read · Calidad Infotech

    A paid pilot with a software vendor should include one bounded outcome, a small team that mirrors the one you'd scale, success criteria written and signed before work starts, a fixed time box, and a decision date at the end. Judge it on working software shipped through your own review and security gates, not on slides or effort reports.

    The rest of this post covers how to set up each part, what to check while the work is running, and how to make the call when it ends.

    Why run a paid pilot instead of choosing a vendor from a pitch?#

    A pitch shows you how a vendor sells. A pilot shows you how it builds.

    That gap has grown. AI tools make a working demo cheap to produce, and a demo that runs proves very little. Veracode's 2025 GenAI Code Security Report tested more than 100 large language models on 80 coding tasks. The models produced functional code, but that code had security vulnerabilities in 45% of cases. At the team level, the 2024 DORA report found that AI adoption raises individual productivity but hurts software delivery stability and throughput. It says fundamentals like small batch sizes and solid testing still matter.

    So you need to see how a vendor gets code from an engineer's editor into production. A pitch can't show you that. Paying for the pilot also matters:

    • You get the real team. A vendor that's paid for the work has a reason to staff it with the people who would actually continue.
    • You can use the output. Paid work should come with clear ownership of the code and artifacts, so the pilot isn't wasted if you stop.
    • Both sides take it seriously. A budget line means the criteria, the schedule and the decision all get attention.

    At Calidad, every engagement starts this way: a paid pilot with one pod and one outcome, success criteria agreed up front, and the result judged on working software.

    How do you pick the right outcome for a pilot?#

    Pick a real, bounded piece of your roadmap that matters to the business but won't do serious damage if it slips.

    A good pilot outcome meets most of these criteria:

    1. It's real work. It ships to production, or at least to a production-like environment with real data, not to a sandbox nobody will use again.
    2. It touches your actual stack. Your CI pipeline, your auth, your data, your code review rules. Most friction shows up there.
    3. It fits the time box. A team should be able to finish it and still have time to fix what review turns up.
    4. It has a baseline. You can measure where things stand today, so improvement is visible.
    5. It tests the riskiest part of the future engagement. If you plan a cloud migration, move one low-risk workload first so the runbook is proven before anything critical moves.

    Examples that work well: one partner API integration, a feature released behind a flag to internal users, one service moved off a legacy platform, or a grounded AI assistant for a single use case with its own evaluation set.

    Examples to avoid: a greenfield toy app, a vague "explore our codebase" brief, or anything that depends on a decision your leadership hasn't made yet.

    Who should be on the pilot team, from both the vendor and your side?#

    The vendor should staff the pilot with the same people and roles that would continue the work. Your side should commit people with real time and decision rights.

    Vendor side. For product work, you want a complete small team rather than a couple of individual developers. A Calidad Product Engineering Pod, for example, includes an engineering lead, full-stack engineers, QA and automation, DevOps, and product and UX. Ask the vendor two direct questions:

    • Will these exact people stay on the work if we scale?
    • Can we interview them? (On Calidad's dedicated teams, clients interview every engineer.)

    Your side. The pilot fails quietly if your people aren't available. Name:

    • A subject-matter expert who can answer domain questions the same day.
    • A product owner who can accept or reject work.
    • A senior engineer from your team who reviews pull requests and will judge maintainability.
    • A security contact who knows your policies and tooling.
    • An executive sponsor who owns the decision at the end.

    The vendor's engineers should join the same stand-ups and channels as your team. If they can only reach your experts through tickets and a weekly status call, you're testing a hand-off process, not a partnership.

    How do you define success criteria before the pilot starts?#

    Write three to six criteria that cover the outcome, the quality of the work and the way the team operates. Each one needs a measure, a threshold and a named person who judges it. Both sides sign before kickoff.

    Area Example criterion Evidence
    Outcome Feature live behind a flag for internal users by the agreed date Working software in your environment
    Delivery Changes merged through your CI; lead time and change failure rate tracked Pipeline data
    Quality Meaningful tests on new code; no failing checks at release Test reports, PR history
    Security No open critical findings; meets your agreed security level Scanner output, security review
    Maintainability Your engineers can explain and extend the code Handover session, review comments
    Collaboration Blockers raised within the agreed response window Stand-up notes, channel history

    For delivery, the standard reference is DORA's four key metrics: deployment frequency, lead time for changes, change failure rate and time to restore service. For security, the OWASP Application Security Verification Standard gives you a shared vocabulary. Its Level 2 is for applications that hold sensitive data, and OWASP recommends it for most apps.

    Leave out vanity measures like hours logged, story points or lines of code. AI tools can inflate all three without moving your outcome.

    Also write down what failure looks like. A stop condition agreed in advance is much easier to act on than one argued about afterwards.

    The structure of a paid pilot, from choosing the outcome to the decision review
    The structure of a paid pilot, from choosing the outcome to the decision review

    What should you check during the pilot: code quality, security, review and time-zone overlap?#

    Look at the work every week, not just at the end. By the final demo, the habits you needed to see are already hidden.

    Code quality

    • Read a sample of pull requests yourself, or have your senior engineer do it.
    • Check that the tests cover real behavior and edge cases, not just the happy path.
    • Ask which changes were AI-generated and how each one was verified.

    Security

    • Confirm that dependency, secret and static analysis scans run on every change, not once before handover.
    • Use NIST's Secure Software Development Framework (SP 800-218) as a checklist for the vendor's practices. NIST has also published SP 800-218A, which adds practices specific to generative AI models.
    • Watch how findings get handled. Are they fixed, accepted with a reason, or ignored?

    Review

    • Find out who approves each change and how senior they are.
    • Look for release evidence: a record of what was tested, scanned and reviewed for each release. Calidad's engineering standard puts every change through three gates: tests prove it works, security checks prove it's safe, and a senior engineer's review proves it's right. That applies whether a person or an AI wrote it. We describe the method in detail in how to verify AI-generated code before production.

    Time-zone overlap

    • Count the shared working hours you actually got, not the ones promised in the proposal.
    • Track how long a blocking question waits for an answer.
    • Notice whether problems come up early or only at the deadline.

    If you're planning long-term capacity, this is where offshore partnerships usually succeed or fail. Calidad's dedicated teams work in the client's time zone, with a delivery lead accountable for pace and quality. Hold any vendor to the same standard during the pilot.

    Should a pilot be fixed-fee or time-and-materials?#

    A fixed fee for a clearly scoped outcome usually fits a pilot better. It ties the vendor's payment to delivering the result rather than to hours spent. Time-and-materials makes sense when the scope is unknown, as in early discovery.

    Fixed-fee Time-and-materials
    Best for A defined outcome with agreed criteria Discovery, research, unclear scope
    Who carries scope risk Mostly the vendor Mostly you
    What it tests Whether the vendor can estimate and deliver Whether the vendor can work with you
    Watch out for Scope cut quietly to protect margin Effort growing without progress

    Whichever model you choose, put these points in writing: who owns the code and artifacts, what happens if criteria are missed, and what commitment (if any) follows the pilot. Calidad prices work as agreed outcomes in fixed-fee pods, with no long-term contracts. If a pilot doesn't earn the next phase, nothing locks you in.

    How do you decide whether to scale, adjust or stop after the pilot?#

    Hold a decision review against the criteria you signed, and choose one of three outcomes: scale, adjust and re-run, or stop.

    • Scale when the criteria are met, the gates held, and the team you saw is the team you'll get. Plan the next scope before adding people, so new capacity joins work that already ships.
    • Adjust when results were partly met and the cause is specific and fixable. Typical causes are slow environment access, an expert who wasn't available, or a scope that was too wide. Fix the cause, then run a short second phase.
    • Stop when quality or security gates were missed, or your team had to manage the vendor into basic practices. Take the code and the lessons, and move on.
    How to decide between scaling, adjusting or stopping after a pilot
    How to decide between scaling, adjusting or stopping after a pilot

    The same logic applies at larger scale. For a Global Capability Center, Calidad's approach is to seed one senior pod on real work and add headcount only as output is proven. A pilot is that first step at a smaller size.

    Conclusion#

    A useful pilot is small, real and measured: one outcome, one team that would actually continue, criteria signed before work begins, weekly checks on code, security, review and overlap, and a decision you've committed to make. Run it that way and you're judging working software, not a sales story.

    If you'd like to scope a pilot like this for your roadmap, talk to an engineering lead →

    Frequently asked questions

    How long should a paid software development pilot last?

    Set the length by the outcome, not the calendar. The pilot should be long enough to ship several changes through your real pipeline and review process, and short enough that the decision stays fresh. Calidad's pilots aim for results in days, with the scope sized to fit.

    Who owns the code written during a paid pilot?

    Agree on this in writing before work starts, usually in the statement of work. In most paid pilots the client should own the code and artifacts, so the work is still usable if you decide not to continue. Check that the agreement also covers third-party and open-source components.

    Should we run pilots with several vendors at the same time?

    You can, if every vendor gets the same outcome, the same criteria and the same access, so the comparison is fair. The real cost is your own experts' time, since each pilot needs a product owner, a subject-matter expert and a reviewer. Many teams run one pilot at a time to keep that load manageable.

    What if the pilot only partly meets the success criteria?

    Find the cause before deciding. If it's something specific, like delayed access, an expert who wasn't available or a scope that was too wide, fix it and run a short second phase. If the vendor missed quality or security gates, treat that as a reason to stop.

    Can a pilot be the first step toward a Global Capability Center?

    Yes. A pilot pod working on real headquarters work is a low-risk way to test the operating model, quality bar and time-zone working before you commit to hiring. Calidad's GCC approach starts with one senior seed pod and adds pods as output is proven.

    • vendor selection
    • paid pilot
    • software outsourcing
    • engineering quality
    • DORA metrics

    Newsletter

    Get new articles by email

    A short monthly round-up of what we've written on engineering pods, quality and AI. No spam.

    Keep reading

    Start with proof

    Skip the pitch.
    See it work.

    A paid pilot: one pod, one outcome, judged on working software — results in days. No long-term contracts.