GandiMail delays (UPDATE)
By Ryan on Tuesday 6 January 2009, 10:36 - Maintenance - Permalink
Hello,
Following the end of the holiday season, the heavy use of our mail service is causing an abnormal load on the GandiMail platform this morning.
Our engineers are currently working on the problem to unblock the service as soon as possible. Please accept our apologies for the inconvenience.
The Gandi Team
UPDATE: following the degradation of Gandi's mail servers, we have decided to reboot them, which should solve the problem. No mail will be lost, though your mail client may not be able to log into our machines for 5 - 10 minutes.
3:00 PM The reboot is over. The service will progressively return to normal.
Wednesday 10 AM: The overload is unfortunately still an issue this morning. Our engineers are working to solve this as fast as possible.
Wednesday 2 PM - Explanations: Explanations: a bug in our Mail solution has been identified. Basically, when the multiplication of email download requests are creating an important charge on the server, the requests are queued up, and so until the boxed get jammed. Being all impatient to retrieve your messages, which is quite understandable, you are frequently requesting the download of your messages, which worsen the problem. Yesterday's reboot improved the situation, but the requests being again multiplied today, the problem arises once more.
Our engineers are working on solving the issue and our system administrators are fighting to give you all access again. A temporary solution is in place : a limitation of the download requests. This means that if you do not get your emails following a download request, the most efficient thing to do is to wait 5 minutes before trying again. This will insure a speedy recovery of emails for all.
Please accept our apologies for the inconvenience, and rest assured that we are taking all the necessary measures to put an end to this problem once and for all.
UPDATE: Thursday, 11:29 AM (Paris time)
We are going to go into more detail soon on our blog, but here is a quick summary of the situation.
The server load problem was fixed this morning. We are still in the process of delivering messages that date from yesterday for some customers.
In all appearances we have a kernel bug on our hands (on NSF), but we have found a way to work around it at present. We are in the process of identifying it so as to put in a patch.
For those of you that are still experiencing difficulties, we need you to contact us at mail@support.gandi.net, so that we can help you on a one-to-one basis AND to provide us with more information.
We sincerely do apologize for this situation and the performance of this service, which has indeed been operating well below our objectives over the past few days. E-mail is a service that can never be perfect (and nobody in the world is), but the fact of the matter is that we do have an obligation to provide you with a service that is free from reoccurring problems. Whether or not the service in question is free is not the question; we are speaking of a basic service.
We have done everything possible over the past few months (software and hardware) to seriously put an end to the repetitive problems of this service in particular. Yesterday's incident, despite of the bug in question, something that makes all of this even more infuriating. This will not happen again, as 2009 will be a year of shoring up the existing services (something that I will go into detail in an upcoming entry).
Those of you that, despite everything, continue to give us that most precious thing which is your trust, will not regret it. I think that once the anger of the moment has passes, which I share, the majority of you will realize that we are still here: honest, up front, and professional.
UPDATE: Thursday, 7:40 PM (Paris time)
Some users may be experiencing delays in mail delivery of up to three hours, which will subside through the evening.
Additionally, some users reported being unable to access their mailbox between 4pm and 5pm -- this has now been fixed.
We have identified certain operating system bugs and other issues associated with the heavy load on the server platform and are working to apply the necessary updates to these systems in order to avoid any reoccurence of the problem. We will verify these updates from Friday.
We will, of course, keep you informed of any night time maintenance periods should they be necessary.
Following the end of the holiday season, the heavy use of our mail service is causing an abnormal load on the GandiMail platform this morning.
Our engineers are currently working on the problem to unblock the service as soon as possible. Please accept our apologies for the inconvenience.
The Gandi Team
UPDATE: following the degradation of Gandi's mail servers, we have decided to reboot them, which should solve the problem. No mail will be lost, though your mail client may not be able to log into our machines for 5 - 10 minutes.
3:00 PM The reboot is over. The service will progressively return to normal.
Wednesday 10 AM: The overload is unfortunately still an issue this morning. Our engineers are working to solve this as fast as possible.
Wednesday 2 PM - Explanations: Explanations: a bug in our Mail solution has been identified. Basically, when the multiplication of email download requests are creating an important charge on the server, the requests are queued up, and so until the boxed get jammed. Being all impatient to retrieve your messages, which is quite understandable, you are frequently requesting the download of your messages, which worsen the problem. Yesterday's reboot improved the situation, but the requests being again multiplied today, the problem arises once more.
Our engineers are working on solving the issue and our system administrators are fighting to give you all access again. A temporary solution is in place : a limitation of the download requests. This means that if you do not get your emails following a download request, the most efficient thing to do is to wait 5 minutes before trying again. This will insure a speedy recovery of emails for all.
Please accept our apologies for the inconvenience, and rest assured that we are taking all the necessary measures to put an end to this problem once and for all.
UPDATE: Thursday, 11:29 AM (Paris time)
We are going to go into more detail soon on our blog, but here is a quick summary of the situation.
The server load problem was fixed this morning. We are still in the process of delivering messages that date from yesterday for some customers.
In all appearances we have a kernel bug on our hands (on NSF), but we have found a way to work around it at present. We are in the process of identifying it so as to put in a patch.
For those of you that are still experiencing difficulties, we need you to contact us at mail@support.gandi.net, so that we can help you on a one-to-one basis AND to provide us with more information.
We sincerely do apologize for this situation and the performance of this service, which has indeed been operating well below our objectives over the past few days. E-mail is a service that can never be perfect (and nobody in the world is), but the fact of the matter is that we do have an obligation to provide you with a service that is free from reoccurring problems. Whether or not the service in question is free is not the question; we are speaking of a basic service.
We have done everything possible over the past few months (software and hardware) to seriously put an end to the repetitive problems of this service in particular. Yesterday's incident, despite of the bug in question, something that makes all of this even more infuriating. This will not happen again, as 2009 will be a year of shoring up the existing services (something that I will go into detail in an upcoming entry).
Those of you that, despite everything, continue to give us that most precious thing which is your trust, will not regret it. I think that once the anger of the moment has passes, which I share, the majority of you will realize that we are still here: honest, up front, and professional.
UPDATE: Thursday, 7:40 PM (Paris time)
Some users may be experiencing delays in mail delivery of up to three hours, which will subside through the evening.
Additionally, some users reported being unable to access their mailbox between 4pm and 5pm -- this has now been fixed.
We have identified certain operating system bugs and other issues associated with the heavy load on the server platform and are working to apply the necessary updates to these systems in order to avoid any reoccurence of the problem. We will verify these updates from Friday.
We will, of course, keep you informed of any night time maintenance periods should they be necessary.