Yesterday evening at 20:58, Cheméo disappeared from the internet. It came back for a few minutes at 21:08, went away again, and was fully back at 21:55. With two shorter cuts earlier in the day, the site was unreachable for about an hour in total. In ten years of running Cheméo, this is the first time it has been down for so long.
In June, I wrote that the server never noticed the AI crawlers hammering it, and that I had not needed to log in and fix anything in a long time. Well, I logged in yesterday.
What happened
The server did not crash. Cheméo kept running the whole time, it simply could not talk to anybody, because the network link between the server and the switch in the OVHcloud datacenter in Roubaix had dropped. Basically, the shop was open with the lights on, but someone had taken away the street.
The cuts, in CEST:
- 12:09 to 12:19, for 10 minutes.
- 19:55 to 19:58, for 3 minutes.
- 20:58 to 21:55, with a short return between 21:08 and 21:15.
OVHcloud opened an intervention ticket at 21:06, and a technician was working on the server at 21:10. They rebooted it, ran their hardware checks, and closed the ticket at 21:56 with a short note: "Connectivity restored after reboot."
Your data is fine
No data was lost. The user account database passed an integrity check after the restarts, and the compound database had not been written to since 19 October. One of the restarts was abrupt, which is exactly the case the database is designed for: a transaction is either fully written or not written at all.
247 drops since July
The server keeps a log of its network link, and the log goes back to 1 July, when Cheméo moved onto it. The hardware itself is ten years old. It was running other Céondo services, our code forge among them, and in July we moved them to another server and gave this one to Cheméo.
Since 1 July, the link has dropped 247 times. Most drops lasted a few seconds, about 94 minutes in total, and nobody noticed, me included. After six quiet weeks from the end of July, the drops came back in September, and October is the worst month so far with almost four a day. If a Cheméo page failed to load for no reason over the past weeks, this is very likely why.
In 228 of these 247 drops, the link came back at a tenth of its normal speed. That points at the hardware on the link: the cable, the port on the switch, or the network card of the server. On 14 July, OVHcloud already fixed the same symptom with the same reboot (the dotted line on the chart), and no hardware was replaced, neither in July nor yesterday. So the fault is likely to come back.
We will see what OVHcloud tells me about the hardware. After ten years of service, maybe it has simply reached the end of its life. I do not want to wait for the 248th drop to find out.
A new server in the coming two weeks
In the coming two weeks, a new server will be provisioned and tested, and Cheméo will be transferred to it. I will take the time to test it properly, trading a bad network link for a bad disk would not be much of an improvement.
Until the move, Cheméo stays on the current server, which has been answering normally since 21:55 yesterday. I will be watching its network link a lot more closely than I did over the summer.