Why Take-Home Tests Break Down as Engineering Teams Scale

The exercise that used to work

At a five-person engineering team building editorial tooling, a take-home coding exercise is easy to run. The founder or first engineer writes the prompt, candidates submit their solutions over a weekend, and the same person reads every submission before the week is out. There's one reviewer, one rubric (even if it lives only in that person's head), and enough time to give each submission real attention.

This setup tends to survive the first few hires without anyone noticing a problem. Then the team hires its eighth or tenth engineer, and the same exercise, run the same way, starts producing worse outcomes even though nothing about the prompt has changed. What changed is everything around it.

The review bottleneck nobody assigned

At five engineers, one person can absorb the review load because there are only a handful of open roles at any time. At ten, a growing team is often hiring for two or three positions simultaneously: a backend engineer for the CMS, a frontend engineer for the reader-facing app, maybe a platform hire to handle publishing pipelines. Each role attracts a stack of submissions, and the person who used to review everyone now has a full-time job that doesn't include reviewing code.

What typically happens next is quiet and informal. Review gets delegated to whoever has a free afternoon, sometimes different people for different candidates, sometimes the same overworked lead skimming submissions late at night. Nobody explicitly decided to lower the bar. But without one person applying one standard, the standard drifts. A submission that would have been rejected in month three gets a pass in month fourteen because the reviewer that week was more forgiving, or less experienced, or simply tired.

The EEOC's guidance on employment tests and selection procedures is written for a different purpose, but the underlying point applies directly here: a selection method that isn't applied consistently isn't really one method. It's several methods wearing the same name, and that inconsistency is exactly what makes take-home grading fall apart once no single person owns it.

Who the take-home test actually screens for

A take-home exercise doesn't just test coding ability. It tests whether a candidate has several uninterrupted hours to give a company for free, on short notice, without pay, while still doing their current job and whatever else fills their week. That's a real cost, and it isn't distributed evenly.

The Bureau of Labor Statistics' American Time Use Survey tracks how unpaid caregiving and household responsibilities fall disproportionately on specific groups. A hiring process that requires several unpaid hours of focused work outside a job is, in effect, selecting for people with more discretionary time. That isn't a hypothetical concern for a five-person team hiring through personal networks. It becomes a structural bias once the pipeline widens and the company is trying to hire from outside its existing circle, which is usually exactly what happens between the fifth and tenth engineer.

The piece on what changed when the second engineer joined covers a related shift: the informal habits that work when a team is small enough to run on shared context stop working once the team needs to function as a system rather than a collection of individuals who happen to like each other. Hiring is one of the first systems to show the strain.

When the exercise starts rewarding the wrong skill

There's a second, quieter failure mode. Take-home exercises tend to get revised over time, usually to fix whatever went wrong with the last hire. A candidate who wrote messy but functional code gets rejected, so the next version of the prompt adds stricter requirements. A candidate who over-engineered a simple problem gets rejected, so the prompt gets simplified. After a few rounds of this, the exercise has been optimized against specific failure cases rather than designed around what the job actually requires.

The result is a test that rewards people who are good at reading hiring prompts and guessing what evaluators want, a skill that correlates only loosely with the day-to-day work of shipping features inside an existing codebase, debugging a production issue during a traffic spike, or working inside someone else's editorial CMS with its accumulated quirks. Meta-analytic research on personnel selection, the kind published in outlets like the American Psychological Association's Psychological Bulletin, has long pointed out that structured, job-relevant evaluation methods predict performance far more reliably than exercises optimized through informal trial and error.

What replaces it, and what that costs

Teams that outgrow the take-home test tend to land on one of two replacements. The first is a paid trial project: a short, scoped piece of real work, compensated at a fair rate, usually a few days rather than a weekend. This fixes the unpaid-labor problem directly and tends to produce a more honest signal, since the work resembles the job. But it's expensive to run at volume, and it requires legal and payroll infrastructure that a five-person team usually doesn't have. It also assumes candidates can take a few days away from a current job, which reintroduces a milder version of the same time-availability bias.

The second is a structured pairing session: a scheduled block of live collaborative coding against a fixed rubric, scored the same way for every candidate. Google's internal research on structured interviewing is built on this same principle, that consistent scoring criteria applied by multiple trained interviewers outperforms ad hoc judgment. Pairing sessions solve the consistency problem better than either take-home tests or unstructured trials, but they demand interviewer training and calendar coordination, and they can disadvantage candidates who think well independently but underperform under real-time observation.

Neither replacement is free, and neither eliminates trade-offs. They exchange one kind of bias and cost for another.

Diagnosing which problem you actually have

The useful question for a team feeling this friction isn't whether take-home tests are good or bad in the abstract. It's whether the current pain is a scaling problem or a design problem. A scaling problem looks like a review bottleneck: submissions piling up, no single owner, inconsistent turnaround times. That's fixed by adding structure and ownership, not by changing the exercise itself. A design problem looks like an exercise that has drifted away from the actual job through years of reactive patching, or one that quietly filters out anyone without several free unpaid hours.

Most teams have some of both. The honest move is to separate them before redesigning anything, because a new hiring process built to fix a scaling problem won't fix a design flaw, and vice versa. A team that gets this diagnosis wrong tends to end up eighteen months later with a different exercise and the exact same complaints.