Full Login Flow (Mail OTP) Load Test

Test Overview

Item	Details
Test Date	December 17, 2025
Target Flow	Complete OAuth 2.0 login with Mail OTP
Purpose	Measure end-to-end login performance including OTP verification
Sharding	Region Shards 32, Code Shards 8, Refresh Token 64

Executive Summary

With 32-shard optimization, DO errors and latency spikes from the previous test (Dec 13) were completely eliminated. Stable operation confirmed at 150 LPS (Logins Per Second).

Target LPS	Completed Flows	Success Rate	P50	P95	P99	DO P99	Status
50 LPS	7,312	100%	587ms	688ms	748ms	2,846ms	⚠️ Cold start
100 LPS	14,624	100%	570ms	696ms	753ms	1,393ms	✅ Good
125 LPS	18,232	100%	621ms	761ms	826ms	1,153ms	✅ Good
150 LPS	21,937	100%	639ms	756ms	853ms	955ms	✅ Excellent

Comparison: Dec 13 vs Dec 17

Metric	Dec 13 (16 shards)	Dec 17 (32 shards)	Improvement
100 LPS DO Errors	443 (0.39%)	0	100% eliminated
150 LPS P95	3,015ms	283ms	91% reduction
150 LPS DO P99	1,648ms	955ms	42% reduction

Test Environment

K6 Cloud Configuration

Parameter	50 LPS	100 LPS	125 LPS	150 LPS
Executor	ramping-arrival-rate	ramping-arrival-rate	ramping-arrival-rate	ramping-arrival-rate
Test Duration	3 min	3 min	3 min	3 min
Preallocated VU	200	400	500	600
Max VU	400	600	800	1000
Seed Users	500	1000	1250	1500
Load Zone	amazon:us:portland	amazon:us:portland	amazon:us:portland	amazon:us:portland

Sharding Configuration

Store	Shards	Generation
SessionStore	32	Gen 2
ChallengeStore	32	Gen 2
AuthCodeStore	32	Gen 2
RevocationStore	32	Gen 4
RefreshTokenRotator	64	Gen 4

Test Methodology

Full Login Flow (5 Steps)

flowchart LR
    A["1. AuthorizeInit
(GET /authorize)"] --> B["2. OTP Generate
(POST /email-code/generate)"]
    B --> C["3. OTP Verify
(POST /email-code/verify)"]
    C --> D["4. AuthorizeCode
(GET /authorize prompt=none)"]
    D --> E["5. Token
(POST /token)"]

Step Descriptions

1. AuthorizeInit: OAuth 2.0 authorization request, display login page
2. EmailCodeGenerate: Generate & store OTP code (ChallengeStore)
3. EmailCodeVerify: Verify OTP code, issue session (SessionStore)
4. AuthorizeCode: Issue authorization code (AuthCodeStore)
5. Token: Issue access token and ID token

Success Criteria

• P95 Latency < 5,000ms
• Success Rate > 95%

Results - Performance Metrics

50 LPS Test

Test Period: 2025-12-16T16:01:30Z - 16:06:30Z UTC

Metric	Value
Worker Total Requests	30,478
Worker P50 Duration	22.6ms
Worker P99 Duration	46.7ms
DO Total Requests	45,688
DO Errors	0
DO Wall Time P50	46ms
DO Wall Time P99	2,846ms
D1 Read Queries	1,143,239
D1 Write Queries	28,478

⚠️ Note: High DO P99 due to cold starts. At 50 LPS with 32 shards, each shard receives only 1.56 req/s.

100 LPS Test

Test Period: 2025-12-16T16:09:00Z - 16:14:00Z UTC

Metric	Value
Worker Total Requests	59,761
Worker P99 Duration	67.4ms
DO Total Requests	89,531
DO Errors	0
DO Wall Time P99	1,393ms
D1 Read Queries	1,169,929
D1 Write Queries	41,785

✅ Good: Zero DO errors (vs 443 on Dec 13)

125 LPS Test

Test Period: 2025-12-16T16:15:50Z - 16:21:00Z UTC

Metric	Value
Worker Total Requests	74,176
Worker P99 Duration	88.4ms
DO Total Requests	111,184
DO Errors	0
DO Wall Time P99	1,153ms
D1 Read Queries	1,209,902
D1 Write Queries	61,789

✅ Good: DO Wall Time P99 continues to improve

150 LPS Test

Test Period: 2025-12-16T16:23:00Z - 16:28:00Z UTC

Metric	Value
Total Requests	89,000
HTTP Failures	0
Peak RPS	598 req/s
P95 Response Time	283ms
Worker P99 Duration	69.2ms
DO Total Requests	133,400
DO Errors	0
DO Wall Time P99	955ms
D1 Read Queries	1,248,445
D1 Write Queries	81,019

✅ Excellent: Best results across all metrics. DO P99 under 1 second.

Step-by-Step Latency (150 LPS)

Step	Count	Avg	P50	P95	P99	Max
AuthorizeInit	21,937	106ms	103ms	129ms	173ms	1,102ms
EmailCodeGenerate	21,937	217ms	210ms	279ms	331ms	1,202ms
EmailCodeVerify	21,937	260ms	252ms	336ms	400ms	2,627ms
AuthorizeCode	21,937	68ms	63ms	88ms	104ms	8,351ms
Full Flow	21,937	652ms	639ms	756ms	853ms	8,871ms

Sharding Analysis

Cold Start vs Load Relationship

LPS	Load per Shard	DO P99	Status
50	1.56 req/s	2,846ms	⚠️ Cold starts frequent
100	3.13 req/s	1,393ms	✅ Stable
125	3.91 req/s	1,153ms	✅ Stable
150	4.69 req/s	955ms	✅ Optimal

Finding: Each shard needs 3+ req/s to remain warm and perform optimally.

32 Shard Optimization Effect

Metric	Dec 13 (16 shards)	Dec 17 (32 shards)
K6 P95	3,015ms	283ms
Worker P99	365ms	69ms
DO P99	1,648ms	955ms
DO Errors	2	0
Success Rate	99.97%	100%

Capacity Recommendations

Load Level	LPS	Monthly Logins	Expected Service Scale
Low	~50	~130M	Small/Medium (use 16 shards)
Normal	~100	~260M	Medium services
Recommended Max	~150	~390M	Large services
Limit	200+	520M+	Not tested

User Scale Conversion

1 Login = 5 HTTP requests:

• 150 LPS × 5 = 750 RPS (HTTP request equivalent)
• 390M monthly logins = ~19.5M monthly active users (weekly login)
• 390M monthly logins = ~13M monthly active users (3 logins/month)

Key Findings

1. 32 Shards Eliminated All Issues

• DO Errors: 443 → 0 (100% fix)
• P95: 3,015ms → 283ms (91% reduction)
• Spikes: Present (30s+) → None

2. Higher Load = Better DO Performance

Counterintuitively, DO P99 improves as load increases:

• 50 LPS: 2,846ms (cold starts)
• 150 LPS: 955ms (warm shards)

3. Worker Remains Efficient

Worker P99 stays under 100ms even at 150 LPS.

4. EmailCodeVerify is the Slowest Step

At 260ms average (40% of total flow time), this step involves:

• OTP verification
• Session creation
• Multiple DO operations

Performance Visualization

DO Wall Time P99 by Load

xychart-beta
    title "DO Wall Time P99 (ms) - Improves with Higher Load"
    x-axis ["50 LPS", "100 LPS", "125 LPS", "150 LPS"]
    y-axis "Latency (ms)" 0 --> 3000
    bar [2846, 1393, 1153, 955]

LPS	DO P99	Status	Note
50	2,846ms	⚠️	Cold start frequent
100	1,393ms	✅	Stable
125	1,153ms	✅	Stable
150	955ms	✅	Optimal

Counter-intuitive finding: DO Wall Time improves as load increases due to reduced cold starts.

Bottleneck Analysis

Layer	50 LPS	100 LPS	125 LPS	150 LPS
Worker P99	47ms	67ms	88ms	69ms
DO P99	2,846ms ⚠️	1,393ms	1,153ms	955ms ✅
K6 P95	688ms	696ms	761ms	756ms
Verdict	Cold start	Good	Good	Optimal

Infrastructure Architecture

flowchart TB
    subgraph Test["Test Environment"]
        k6["k6 Cloud (Portland)"]
    end

    subgraph CF["Cloudflare Edge"]
        subgraph Workers["Workers"]
            OA["op-auth Worker
(66K req)"]
            OT["op-token Worker"]
            OM["op-mgmt Worker
(22K req)"]
        end

        subgraph DO["Durable Objects (shared)"]
            SS["SessionStore
(32 shards)"]
            ACS["AuthCodeStore
(32 shards)"]
            CS["ChallengeStore
(32 shards)"]
        end

        subgraph DB["Database"]
            D1["D1 Database (conformance)"]
        end
    end

    k6 -->|HTTPS| OA
    k6 -->|HTTPS| OT
    k6 -->|HTTPS| OM
    OA --> SS
    OA --> ACS
    OA --> CS
    OT --> SS
    OM --> SS
    SS --> D1
    ACS --> D1
    CS --> D1

Remaining Issues

1. Cold Start at Low Load

At 50 LPS, each shard receives only 1.56 req/s, causing cold starts.

Mitigation:

• Use 16 shards for low load environments
• Extend warmup period

2. DO P99 Variability

P99 ranges from 955ms to 2,846ms depending on load.

Mitigation:

• Choose shard count based on expected load
• Dynamic shard adjustment (future)

Test Execution Details

K6 Cloud Run URLs

Test	URL
50 LPS	https://authrim.grafana.net/a/k6-app/runs/6338621
100 LPS	https://authrim.grafana.net/a/k6-app/runs/6338674
125 LPS	https://authrim.grafana.net/a/k6-app/runs/6338743
150 LPS	https://authrim.grafana.net/a/k6-app/runs/6338820

Conclusion

With 32-shard optimization, the Full Login Flow achieves:

• 150 LPS (390M monthly logins) with stable performance
• 100% success rate at all tested levels
• DO errors eliminated (443 → 0)
• P95 reduced by 91% (3,015ms → 283ms)

Key Improvements:

• DO Errors: 443 → 0 (100% fix)
• P95 Latency: 3,015ms → 283ms (91% reduction)
• DO Wall Time P99: 1,648ms → 955ms (42% reduction)

Cold start at low load (50 LPS) is a known characteristic and can be mitigated by adjusting shard count based on expected traffic.