Automated Testing for a Non-Profit Platform Launch

The Situation

This is the later half of the non-profit platform project. By late summer the organization’s new platform had grown well past a website: a public site, a staff portal for inventory, tickets and checklists, a board governance portal, online shop and ticketing, a content management system, and a new sign-in system tying them together.

User acceptance testing was mine to run, and it had a problem. Staff testers had an hour at a time and were not technical. Three sign-in incidents in a few weeks had happened in front of the people the organization least wanted to see them. And much of what needed checking was the kind of thing people miss by hand: whether every role sees the right screens, whether a filter actually filters, whether a date saved on one screen shows the same date on another.

Manual UAT was still going to happen. But there was too much surface for staff alone to cover before launch.

Planning Before Code

I wrote the plan before any test code. In late August I drafted 43 user-journey scripts in plain language, one per thing a real person does on the platform: a shop manager files a weekly checklist, a board member reviews meeting materials, a visitor buys a ticket.

Then, in late September, I classified all 750 steps in the manual UAT plans by how each could be tested: automatically, automatically with test data we didn’t have yet, or only by a person. That classification is what turned “write some tests” into a scope with edges. It also showed where the manual sessions should spend their time.

Every script got a short design document before it was written. That sounds slow, but it meant the AI could write the test from a reviewed description of the behavior instead of guessing from the code.

Testing the Real Thing

The development team already had a local test suite that starts its own server and database. I built a separate one that runs against the deployed test site, signed in with real role accounts, because that is where the sign-in problems and the missing content actually showed up.

Testing a deployed site safely meant writing rules first:

• No vendor traffic. Every browser call to the payment processor, email services, the donor database and analytics is aborted and logged. The test site’s integrations were treated as live production.
• No writes without approval. Every POST to the site is blocked unless the running test declares it, and a test that creates records only runs if its flow is named on an approved list.
• Locked to the test site. The run fails before a browser opens if it is pointed anywhere else.
• Careful with sign-ins. The site shares a limit of 20 sign-in attempts per 15 minutes, so the suite signs in once per role and reuses the session.
• No changes to the application. Where the front end made a test fragile, a label not linked to its input, a button with no accessible name, the test works around it and leaves a marked note. A report collects those notes into a handoff list for the front-end developer. There are 326 of them, and most double as accessibility fixes.

What Got Built

Area	What the tests cover
Staff portal	Tickets, inventory, catalog, POS sync, checklists, reports, forms, donations
Board portal	Meetings, paperwork, requisitions, resolutions, documents, committees
Shop	Cart, checkout up to the final submit, coupons, tickets, membership, giving
CMS	Publishing, version restore, media upload
Identity	Sign-in, role portals, account administration
Public site	Forms, search, events calendar, analytics consent

The October 9 report lists 925 tests from 139 spec files across six site areas. Specs never contain raw selectors. Every locator lives in a page object, so a test reads like a sentence about the job: file the weekly checklist for this shop as this manager.

• e2e-deployed/the suite that runs against the deployed test site
  • specs/the tests, one folder per area
    • staff/ · board/ · shop/ · cms/ · identity/ · marketing/
  • pages/page objects: every locator lives here, never in a test
  • data/who signs in, and what each test expects to see
  • files/the library of upload files
  • support/the vendor and write guards, and shared helpers
  • tools/the phased runner and the review report
  • runs/every run's results, kept on the machine and never committed

The first full runs took over an hour. I rebuilt the runner in phases: sign in once, run read-only tests in parallel, then run approved write tests in lanes so they don’t collide. Each phase leaves its own report and a merge step combines them. An earlier run improved from 62 minutes to 16 after that change; the larger October 9 run reported here took 43.4 minutes. Those are different runs with different scope, not a like-for-like speed comparison.

A Report People Can Read

Playwright’s own report is built for developers. It tells you a test passed or failed, and leaves you to open the code to find out what that test was checking. The people who needed these results were project staff and leads who will never open a spec file.

So the runner finishes by writing a second report of its own. For every test it says, in plain language, what was checked and what the test saw, grouped by area and topic. It also points out where a green tick proves less than it looks like it does. Alongside it sits a machine-readable version with every test’s status, steps, errors and screenshots, so an AI agent can go through a run and pull out what needs attention.

With it, a review can start from what the site actually did when a board member opened a meeting, rather than from a count of failures.

Each run distinguishes a failed check from a test that could not run. A skip can mean a known blocker, missing test-site data, an open decision or work that is not built yet. A failure can mean the site behaved differently from the expectation, or that it was unavailable. The report keeps those reasons visible so the team can decide what to investigate and what to run again.

The window below uses the original report’s stylesheet, colors and layout. It contains one run’s summary and a shortened selection of anonymized results. Open a result to inspect its notes and steps, or use the status filters. The full local report also includes screenshots and raw diagnostic evidence.

Deployed-site reviewOpen report example ↗

Files Made for the Job

A lot of the platform takes uploads: CMS media, board documents, checklist photos, catalog imports. Testing those properly needs real files, and a folder of random PDFs pulled off the internet doesn’t do it.

I used AI to make a library of purpose-built upload files: spreadsheets laid out exactly like the site’s import templates, branded PDFs, Word and PowerPoint documents, photos, a short video, a zip, and a saved email. Each one is made for a specific test and looks like something a staff member would really upload. The library also has the files the site should refuse, such as an empty file, the wrong type, or a file named as one type that is really another, so the tests can check that the site turns them away properly.

Every file is generated by a script, checked by a person once, and listed in an inventory, so anyone can see what a test uploads and why.

What It Found

The suite gave the team a repeatable way to see what the deployed site actually did across roles and workflows. Its value was the kind of evidence it produced on each run: what worked, what differed from the expectation, and what could not yet be checked.

Findings needed different kinds of follow-up:

• Site defects, behavior the team could reproduce and investigate, such as a filter that did not hold its selection, inconsistent totals between screens, or a saved date displayed differently from the date entered.
• Questions for the plan, expectations that needed a decision before a test could call the behavior right or wrong. These might concern the intended workflow, an acceptance criterion, or whether a feature belonged in the agreed scope.
• Test-data gaps, missing records, account roles or fixtures that prevented a journey from being exercised. The report made those prerequisites visible so the team could prepare the environment before asking staff to test it.

A problem in the test itself or an unavailable site also needed to stay distinguishable from an application defect. That classification helped route the work to the right place: fix the application, clarify the plan, prepare the data, or repair the test and run it again.

How AI Fit In

I used AI developer tools throughout the work: planning the journeys, writing design documents, implementing tests, generating upload files, running the suite, and reviewing its results. That work happened inside the repository, alongside the application code. The tools could read the implementation, follow the test designs and agent skills, make changes, and inspect the evidence from a run in the same workspace.

That proximity mattered. A question about a screen could be checked against its code; a proposed test could be compared with the journey it was meant to prove; and a failure could be followed from the report back to the test and the application. Planning and execution stayed connected to the work itself.

The skills carried the rules, locator priority, the vendor and write guards, what goes in the handoff report, so each new test followed them without me restating them. My time went into the parts that needed judgment: what a journey should prove, whether a failure was the site or the test, and which findings were worth a developer’s afternoon.

Current Status

The suite was built and handed over as a tool the development team can run and extend. Each run leaves evidence for review and follow-up; the findings log feeds the continuing backlog. The suite supports launch review, while the team remains responsible for resolving findings and deciding whether the site is ready.