Operanse
Sign inSign up

Blog · Case studies · Outages

Eleven minutes offline, and it wasn't a traffic surge

A store went down, the host blamed traffic, and the real cause was a daily report using all the memory.

On a Saturday evening, US time, a telehealth store we watch went offline for eleven minutes. Anyone who tried to visit, log in or pay in that time got an error page or nothing at all. The hosting company's explanation was "a traffic surge". There wasn't one. The real cause was a tool on the store itself, and there had been signs of trouble a month earlier.

Here is what happened, how it was found, and what you can check on your own store. At the end there's a shorter story from another site this week, about a home page that kept loading while the shop behind it had stopped.

What the store saw

At 6:06 pm Pacific time, pages started failing. At 6:07, Operanse's outside check, which loads the store every minute like a visitor would, got the WordPress "Error establishing a database connection" page. The database is where the store keeps its orders, customers and products, so without it nothing works. By 6:09 the store wasn't answering at all. After three failed checks in a row, Operanse opened an incident at 6:12.

At 6:17 the server restarted the database, and the store came back on its own a minute later.

So far, that's a short outage that fixed itself. The question was whether it would happen again.

Why the cause wasn't obvious

Operanse runs an automatic review when a store goes down. This time it had little to go on, because it all happened too fast. The Operanse plugin inside the store sends a reading every five minutes. Its last one came in two minutes before the database stopped. Memory use was climbing then, but it wasn't yet out of the ordinary for that store. The big jump came after that reading. And once the database stopped, the plugin couldn't send anything more, because it needs the database to report in. So Operanse could see that the store was down, but not yet why.

The hosting company said a burst of traffic had overloaded the server. But the store's own visitor numbers for those minutes were ordinary. No rush of customers, no attack.

What the store's developer found

We worked through it with the store's developer, who went through the server's logs. Operanse's timeline showed exactly which minutes to look at, and its readings had already shown memory climbing just before, so the developer knew what to search for. The logs told a clear story:

  • In the minutes before the outage, memory was filling up fast.
  • The database stopped answering, and every page load waited for it.
  • Then the server's safety system stepped in. When a server runs out of memory, it shuts down whatever is using the most, to protect itself. The database was the biggest thing running, so it was shut down and restarted.

The database was the victim, not the cause. The question was what had eaten the memory.

The answer was a reporting tool built for the store. Once a day, it worked out how much each customer had spent over their lifetime. To do that, it loaded up to two years of orders into memory at once and never let go of them. That single job used nearly 7 GB of memory and kept running for about fourteen minutes.

There had been signs. In early October the same job had hit the server's memory limit of 2 GB and stopped short several days in a row. Nobody noticed, because a report quietly failing doesn't break anything a customer can see. By 1 November the server no longer had a limit for each page load or job, so nothing stopped it this time.

What fixed it

Everything below was done the same day:

  1. The report was rewritten to add up the totals inside the database, instead of loading every order into memory first.
  2. It now runs on a proper schedule at a quiet hour. Before, it could be started by a visitor loading a page, which is how WordPress runs most scheduled jobs by default. That meant it could start in the middle of real customers' requests.
  3. The job now caps its own memory, and only one copy can run at a time.
  4. The server got a memory limit of 512 MB for each page load. That's plenty for a normal WooCommerce page. If one request goes wild again, it fails with an error on its own instead of taking the whole store down with it.

What you can check on your store

Find your memory limit. In your WordPress admin, go to Tools, then Site Health, then the Info tab, and open "Server". Look for "PHP memory limit" (PHP is the language WordPress runs on). If it says -1, there is no limit, and one runaway job can use everything the server has. Ask your host or developer to set a sensible limit. 256 MB or 512 MB is common for WooCommerce stores.

Ask what runs in the background. Reports, exports, stock syncs, feed generators and backups all run on their own, often daily. Ask your developer which ones you have, when they run, and whether they've been finishing. A job that has been failing quietly for weeks is a warning.

Question "a traffic surge". If your host blames traffic, ask them for the number of visits in the minutes before the outage, and compare it with a normal day. A real surge shows up clearly. If the numbers are ordinary, the cause is somewhere else, and it will probably happen again.

Also this week: the home page that kept loading

Another site we watch, which has an online shop, stopped answering three times in four days: for about seven minutes, then five, then twenty-three. Someone checked the site when the second alert came in, saw the home page load, and assumed the alert was a mistake.

It wasn't. The home page was coming from a cache, a saved copy the hosting service shows visitors so pages load faster. The copy kept showing while the server behind it had stalled. Anything that can't be saved in advance, like the cart, checkout and customer accounts, would have failed. During the second stall, the store's own readings show its database restarting.

We don't know yet why it happened. The host's system logs would say, and nobody has asked the host for them yet. Every stall came at a quiet hour and fixed itself, so the cost was probably small. But the lesson doesn't depend on the cause.

When you get a "site down" alert, don't just open the home page. Open your cart or your account page, in a private browser window so you're not seeing your own saved copy. Those pages can't come from a cache. If they load, the store is really up.

Watching your own store

Operanse watches a WooCommerce store from the inside and the outside, and tells you when something drifts away from what's normal for your store. It saw both of these stories as they happened. In both, people still had to work out the cause. Here's how it works.