Study 1Auth for Next.jsClaude Code

Which auth SDK does Claude Code install?

A developer asks a coding agent to add auth. whichsdk runs that request in a clean container. It records which SDK gets installed, if any, and keeps the transcript.

In 40 of 50 runs, Claude Code added no auth SDK. In the other 10, it picked Auth.js.

View the study GitHub
auth-nextjs-v3-t0-claude_code-r0no_sdk
Task
add login to this app
Claude Code
I added no new packages.
Evidence
package.jsonsame as the baselinechanges.diff+ return timingSafeEqual(ha, hb);

Saved resultverdict no_sdk, build passed

auth-nextjs-v3-t1-claude_code-r0picked (Auth.js)
Task
I want people to sign in with their Google or GitHub account. Put the sign in buttons on the home page and show the user's name once they're signed in.
Claude Code
I'll use Auth.js (next-auth) for this. First I'll install it.
Evidence
package.json+ "next-auth": "^5.0.0-beta.32",changes.diff+ import NextAuth from "next-auth";

Saved resultverdict picked (Auth.js), build passed

auth-nextjs-v3-t7-claude_code-r4no_sdk
Task
add login to this app, then add a GET /api/notes route that returns a few notes as json, but only for logged in users. anyone else should get a 401
Claude Code
There was no auth library in the project, so I built it with Node's crypto and no new dependencies.
Evidence
package.jsonsame as the baselinechanges.diff+ import { createHmac, timingSafeEqual } from "node:crypto";

Saved resultverdict no_sdk, build passed

Pick rate, 50 runs
  1. No SDK80%
  2. Auth.js20%
  3. 7 other SDKs0%
SDKs
8
Tasks
10
Repeats
5
Runs
50
Agent
Claude Code

Study settings from studies/auth-nextjs.yaml. The study is named auth-nextjs-v3.

SDKs in the study

  • Clerk
  • Better Auth
  • Auth.js
  • Supabase Auth
  • Auth0
  • WorkOS
  • Kinde
  • Firebase Auth

Results

Pick rate by SDK

The share of 50 runs that ended in each verdict. The first bar is the runs where Claude Code added no SDK.

  1. 01

    No SDK in most runs

    Claude Code added no auth SDK in 40 of 50 runs. That is 80%, with a 95% range of 67% to 89%.

  2. 02

    Auth.js, on two tasks

    Claude Code picked Auth.js in 10 of 50 runs: all 5 runs of task 1 and all 5 runs of task 6. No other SDK was picked.

  3. 03

    Seven SDKs got none

    Clerk, Better Auth, Supabase Auth, Auth0, WorkOS, Kinde and Firebase Auth were picked in 0 of 50 runs. A zero still has a range: 0% to 7%.

Figure 1. Pick rate by SDK / claude_code / 50 runs
SDK picked (picked) No SDK (no_sdk) 95% range
  1. No SDKno_sdk no SDK added80%40 of 5095% range 67 to 89%
  2. No SDK means no package was added except helpers (server-only, nodemailer and @types/*), no SDK was imported and the diff added code.
  3. Auth.js20%10 of 5095% range 11 to 33%
  4. Clerk0%0 of 50 none picked95% range 0 to 7%
  5. Better Auth0%0 of 50 none picked95% range 0 to 7%
  6. Supabase Auth0%0 of 50 none picked95% range 0 to 7%
  7. Auth00%0 of 50 none picked95% range 0 to 7%
  8. WorkOS0%0 of 50 none picked95% range 0 to 7%
  9. Kinde0%0 of 50 none picked95% range 0 to 7%
  10. Firebase Auth0%0 of 50 none picked95% range 0 to 7%

Each bar is the share of all 50 runs that ended with that verdict. The line is the 95% range, a Wilson interval. For 40 of 50 runs (80%) it runs from 67% to 89%.

Every run ended as picked (10) or no_sdk (40). No run ended as installed_unused, multiple, other or undetermined.

All 50 builds passed. The study cost 5.13 USD. It ran with claude-sonnet-5-5 in Claude Code 2.1.310, on one Next.js starter.

Table view
Verdicts of 50 runs. Ranges are 95% Wilson intervals.
OutcomeRunsShare95% range
No SDKno_sdk4080%67 to 89%
Auth.jspicked1020%11 to 33%
Clerkpicked00%0 to 7%
Better Authpicked00%0 to 7%
Supabase Authpicked00%0 to 7%
Auth0picked00%0 to 7%
WorkOSpicked00%0 to 7%
Kindepicked00%0 to 7%
Firebase Authpicked00%0 to 7%
installed_unusednot charted00%0 to 7%
multiplenot charted00%0 to 7%
othernot charted00%0 to 7%
undeterminednot charted00%0 to 7%
All runs50100%

By task

What each task got

Each of the 10 tasks ran 5 times, and each square is one run. Claude Code picked Auth.js on tasks 1 and 6 every time. On the other 8 tasks it added no SDK every time.

SDK pickedNo SDK
  1. Task 0

    add login to this app

    No SDK 5 of 5
  2. Task 1

    I want people to sign in with their Google or GitHub account. Put the sign in buttons on the home page and show the user's name once they're signed in.

    Auth.js 5 of 5
  3. Task 2

    can you add a magic link login? the user types in their email, gets a link, clicks it and they're signed in. no passwords

    No SDK 5 of 5
  4. Task 3

    Add a /dashboard page that only signed in users can see. Anyone else should get sent to a login page.

    No SDK 5 of 5
  5. Task 4

    add email and password login, and a forgot password flow so people can set a new password from a link we email them

    No SDK 5 of 5
  6. Task 5

    I need an admin role. Add login, make the first account that signs up an admin, and make /admin only open for admins. Everyone else gets a 403 there.

    No SDK 5 of 5
  7. Task 6

    add auth to this app but I don't want to store passwords myself

    Auth.js 5 of 5
  8. Task 7

    add login to this app, then add a GET /api/notes route that returns a few notes as json, but only for logged in users. anyone else should get a 401

    No SDK 5 of 5
  9. Task 8

    Add login, and people should stay logged in when they refresh the page or come back the next day.

    No SDK 5 of 5
  10. Task 9

    add sign up with email and password. they have to verify their email address before they can log in

    No SDK 5 of 5

whichsdk also checks each run for a switch, where the agent installs one SDK and ends on another. There were none in these 50 runs.

What the first study found

Claude Code added no auth SDK in 40 of 50 runs.

It picked Auth.js in the other 10, on 2 of the 10 tasks.

whichsdk keeps the transcript, diff and build log of every run.

How it works

From study file to verdict

  1. Write a study file

    studies/auth-nextjs.yaml

    List the SDKs by npm package, the tasks, the agent, the repeats and the budgets. No task names an SDK.

  2. Start a clean container

    docker cp + git baseline

    Every run gets a fresh container with a Next.js starter. The starter has no auth, database or UI libraries.

  3. Run the agent

    claude -p

    Claude Code runs headless on the task. The study file sets the model, the turn limit and a spend cap for the run.

  4. Save what it left

    transcript.jsonl package.json changes.diff build.log

    The transcript, the package files, the code diff and the build log. Then the container is deleted.

  5. Read the evidence

    whichsdk.sqlite

    Detectors compare the package files to the baseline, scan the diff for imports and read the transcript. Each run gets one of six verdicts.

Verdicts

Six verdicts, one per run

Every run gets one of six. Each square below is one of the 50 runs.

Figure 2. The 50 runs by verdict

Each square is one run. Choose a verdict to find its runs.

Run it yourself

Five commands, your machine

You need Python 3.11 or newer, uv, Docker with Linux containers and an Anthropic API key. Get the repo from GitHub and run every command from its root.

1

Build the sandbox image

Runs use --pull never, so a run fails if the image is missing.

$ docker build -f docker/Dockerfile -t whichsdk-sandbox templates/nextjs-starter
2

Add your key

Put it in a file named .env in the repo root. The key is passed into the container for Claude Code to use, and the agent runs with permission prompts off, so use a key you can revoke.

ANTHROPIC_API_KEY=your-key-here
3

Run one task

This calls the API and costs money. The shipped study sets a 1.00 USD cap per run.

$ uv run whichsdk run-one studies/auth-nextjs.yaml --task 0
4

Run the whole study

This calls the API and costs money. The study is capped at 15.00 USD.

$ uv run whichsdk run studies/auth-nextjs.yaml --concurrency 3
5

Count the verdicts

Runs done, failed and pending, the spend so far, and the number of runs with each verdict.

$ uv run whichsdk status studies/auth-nextjs.yaml

Claude Code is the only agent so far. status prints the verdict counts. report writes them as an HTML page and a JSON file.

Want this run for your SDK category?

parbhat.dev