Customer story · Performance testing
The first season that opened without a waiting room
Camis moved a reservation platform that buckled above 100 concurrent users to one processing 3,500 reservations in 10 minutes, with the waiting room switched off.
01
A year of demand arrives in about an hour
Camis builds the online reservation system and in-park software that campers, park staff, call center agents and accounting teams use across the United States and Canada. When a booking season opens, close to a year of demand arrives in about an hour.
02
They could see the wave coming and could not hold it
Camis knew exactly what was coming and could not meet it. Their clients’ sites saw between 2,200 and 8,300 connections in a one-hour rush, peaking roughly four minutes after the season opened. The platform struggled above 100 concurrent users. Page response times climbed with load and the application went down during peaks.
The seasonality made it worse. The problem only appeared during the opening rush, so it could not be reproduced, diagnosed or debugged in the moment. They had limited people available during the peak, no view of where the bottlenecks were, and no number for the maximum concurrent load the system could hold. After three years of running it they had a clear picture of the demand and no way to test against it.
Connections across the one-hour opening rush
The rest of the year sits near the baseline, which is why the problem could never be reproduced outside the opening hour.
03
The waiting room managed the symptom
They built a waiting room. It stopped the crashes by holding people out of the application during the rush, and it left Camis and their clients facing customers who could not get in. It managed the symptom for three seasons.
What waiting costs, in the published research
A platform answering in 15 seconds during its highest-intent hour of the year is losing customers it already won. Three seconds is the threshold in the research above. That is the number this engagement had to reach.
Google, The Need for Mobile Speed, 2016. 3,700 sites, global. Measures abandonment during page load. Akamai with SOASTA, State of Online Retail Performance, 2017. About 10 billion visits, online retail only.
04
Testing built to find the breaking point
Testing ran through January and February against a season that opened in March. There was no room to be late.
We designed a testing process that mimicked real production usage, including the multiple geographies Camis serves, with the explicit goal of finding the breaking point. Two use cases, campsite booking and swimming reservation, ramped to 1,000 users over 15 minutes against a staging environment of two web servers and a database server.
We monitored the entire application stack during the tests. The findings were specific.
- Response times began climbing between 400 and 750 users and kept climbing with load
- At 1,000 users the database hit deadlocks and response times spiked
- Significant network bandwidth bottlenecks
- Web server CPU and ASP queuing, downstream of database errors and slow queries
- Base page response times far below total page response times, pointing at asset allocation
Response time against concurrent users, from the load tests
Concurrent users, a different unit from the connection counts above.
05
Five database fixes, in priority order
The counters below are what the monitoring returned. The sentence after each one is why it ranked where it did.
- SQL errors per second. Top priority, on the volume of user errors, info errors and killed connections across testing.
- Workfiles created per second. Running far above acceptable rates, caused by missing indexes and inefficient queries. Workfiles spill to physical disk once memory is exhausted.
- Lock timeouts per second. More lock requests timing out than expected, meaning user queries were not completing.
- Latch waits per second. Requests that could not be granted immediately. Resources with waits over 500ms were reviewed.
- Deprecated SQL functions. Superseded in later SQL Server versions and producing unwanted results if left in place.
The first round of fixes alone returned 15 percent more throughput. It resolved excessive disk IO, SQL log writes outrunning log expansion, and load balancing. The rest came through successive rounds of testing and tuning.
On our recommendation the staging environment was resized from two web servers and one database server to three web servers and a database server with more CPU. Camis also upgraded their network pipe on the same testing evidence.
06
The first season without the waiting room
| Before | After | |
|---|---|---|
| Concurrent user ceiling | Under 100 | 1,000-user test target met |
| Page response time | Over 15 seconds | Under 3 seconds, non-payment |
| Payment response time | Not measured | 5 seconds |
| Peak demand handled | 50 percent of target | 3,500 reservations in 10 minutes |
| Waiting room | Required | Eliminated |
Payment transactions clear a third-party gateway, so they sit outside the three-second target and are reported separately. Payment response time was not measured before the engagement, so there is no before figure to compare against.
“This is the first year we were able to handle our expected volumes.”
Development manager, Camis30 minutes with the engineer who would do the work. Bring the system that worries you, and you will leave knowing what it takes.