I’ve been following QA or the Highway for many years, and this year I finally made it - as both an attendee and a speaker. Last week’s conference was everything I’d hoped for. In my day-to-day work, I sometimes wear the QA, developer, and architect hats for startups without their own testing teams. That means xUnit, bUnit, NSubstitute, and a CI/CD pipeline that runs tests on every branch. But keeping my skills sharp means learning from dedicated testing practitioners - so I went straight to the source.
These are some of the key take-aways from there.
Matt Eland’s Keynote - Ensuring Software Quality in the World of AI Developers
Matt Eland talked about how much AI is proliferating the software development world and why QA still seems like a last line of defense between “code seems fine” and “a misunderstanding in the spec is causing a production fire”. He talked about the importance of QA in a world where AI is treated like a team of junior developers. He mentioned the importance of the human checkpoints, not just the dangerous situation of AI doing all the things - writing the requirements, writing the code, performing the code review, and deploying the changes. Face it - in a world where AI can do all the things, it doesn’t mean that it should do these things unattended or unsupervised. There are opportunities for catching things - like decision making processes and getting into the process earlier rather than waiting until the end. Overall, Matt’s talk really pointed out the importance of a human-powered QA team and what to watch for with the rise of AI.
The real danger isn’t AI writing bad tetss - it’s teams using AI to skip the thinking process entirely.
Decision Records - Understanding Why Those Decisions Were Made!
Photo Credit: Matt Eland
This is my talk that has been making its rounds at many local conferences. I intentionally call it “decision records” and do not limit it to architectural decision records (ADRs). In the talk, though, I do mention ADRs and the architectural significance behind why some things are documented. We talked about the who, what, when, why, how, and where of capturing decisions. There was a question about storing ADRs in Confluence - if that’s where those who need access to the documents all commonly have access, then Confluence might be the place to put your ADRs. You need to make sure that your ADRs are accessible to those who need them. Also - do you really want to make your business team read docs in Git? I do everything I can to keep them out of there if I can avoid it because the confusing user experience of Git and markdown for non-technical users distracts from the ability to read through the decisions.
This was a great talk to give, and my audience was an engaging audience. I’m thankful for the opportunity to share this session with the QA or the Highway community.
Playwright + MCP Server + Claude - a Powerful Trio by Matthew Eakin
Matthew Eakin ran a live demo of Claude with the Playwright MCP server. The server knows Playwright’s docs, so Claude could spin up Page objects and locators fast. But he didn’t hide the downsides - hallucinations and gaslighting are real. My take: make it talk through the plan before it writes a line of code. Do not take the AI’s code for granted - make sure to review it for accuracy and to be sure it is testing relevant things.
Making Migrations Safer and Cheaper with AI-Powered Testing by Federico Toledo
This abstract caught my attention by mentioning COBOL. No, I don’t have COBOL experience. However, the COBOL part told me what kind of migration was in store - moving from older languages into more modern tooling. Federico Toledo reminded us that legacy code is still out there and a lot still needs to be migrated to more modern codebases. He started with a story about a grocery chain that chose the wrong type for their store IDs. It supported 10 stores, but then they had to increase the setup for 11 stores. Oops! They implemented a lot of tests on that project to ensure that it was a correct move. He mentioned that testing made up 90% of that migration project. His COBOL reference in the abstract goes back to a 30-year-old codebase in FinTech. It was a 2-year migration project that had 2 Quality Engineering teams - one for functional end-to-end tests and one to focus on the technical automated tests. He also demoed a tool called Tero from Abstracta to build autonomous agents for software development and QA. It was interesting - and accurate - to see how he made AI appear as an ally for QA people, especially in migration situations.
Lunch
When I’m at conferences, I will sometimes sit with people I don’t know to learn their stories. While I was looking for a table, I saw a lady move her laptop for me, so I took the invitation and joined the table. There were 3 other people of varying backgrounds and experiences. From our newbie to the field to the one who has to bring the messaging of AI and automated tests to a team of manual testers to the one who just recently heard about the conference and wanted to see if it was worth bringing more of her teammates, we talked of our adventures in the field. I had the “old dev” hat on - as I’ve been writing tests since 2011 and coding much longer than that. Yes, I was that grumpy dev who didn’t want to write “extra lines of code that really wouldn’t help me that much”. I was also that grumpy dev who didn’t like that AI agents had a field day with code much like a toddler in a sandbox and shut that down for a bit. However, in both cases, I gave things another chance and turned out to like writing tests and do use my mom voice with the agents to keep them in line. The newest one to the field asked what it’s like to experience these things when I was already in the field, and I mentioned those grumpy moments. But I also mentioned that we have to see AI and testing as the tools and practices that they are, learn about the rewards we can reap by using them, and understanding the trade-offs of it all. It was great to chat with others about our adventures.
Deploying and Testing Ephemeral Environments by Christopher Harbert
The last session I caught was on deploying and testing ephemeral environments. I had heard of them in passing, and I was curious what they were compared to Testcontainers. Testcontainers are good for individual dependencies. Ephemeral environments are used to spin up an isolated, full-stack application for feature branches and pull requests. Christopher Harbert started by talking about the importance of choosing branching strategies before getting into ephemeral environments. He showed us a checklist for setting up ephemeral environments - and how it is key to not only automate provisioning and deprovisioning but that these steps should also be fairly quick. He talked of test data strategies for ephemeral environments, including cloning production data, sampling production data, masking production data, and handcrafted testing data. Ephemeral environments can be incorporated into CI/CD pipelines and manual deployments. While he talked a lot about the Testery platform and what it offers, I gathered enough details to know how to continue my search into whether ephemeral environments make sense. Next up, I will explore Pulumi’s Review Stacks and Time-to-Live Stacks, as they sound similar to what the ephemeral environments need.
The Hallway Track!
While at the event, I also caught up with Andy Knight. I first met him about 1.5 years ago at CodeMash. His energy is fantastic - if you ever get a chance to catch his sessions, do so! Such great energy and very informative! Anyhow, I really wanted to catch his session on Playwright and AI. Unfortunately, I couldn’t have AI present my decision records talk while I skipped to go to Andy’s talk. But I did find out more about his talk. I need to follow up on this conversation, as he mentioned using the Playwright CLI and how it saves tokens compared to using an IDE with the Playwright MCP server. So I have more to explore here!
Conclusion
This was my first time at QA or the Highway. There was a lot of learning. It was interesting to see how other practitioners are using AI in their QA adventures. MCP servers make it easier to get automated tests built. Using the CLI doesn’t use as many tokens as an IDE approach does. Ephemeral environments could make our testing adventures more interesting on some of our client projects. . As for QA or the Highway, I hope to go back there next year, as it’s a great opportunity for me to keep my testing skills sharp.