Gandi IWI Blog

Hello,

Following the end of the holiday season, the heavy use of our mail service is causing an abnormal load on the GandiMail platform this morning.

Our engineers are currently working on the problem to unblock the service as soon as possible. Please accept our apologies for the inconvenience.

The Gandi Team


UPDATE: following the degradation of Gandi's mail servers, we have decided to reboot them, which should solve the problem. No mail will be lost, though your mail client may not be able to log into our machines for 5 - 10 minutes.

3:00 PM The reboot is over. The service will progressively return to normal.

Wednesday 10 AM: The overload is unfortunately still an issue this morning. Our engineers are working to solve this as fast as possible.

Wednesday 2 PM - Explanations: Explanations: a bug in our Mail solution has been identified. Basically, when the multiplication of email download requests are creating an important charge on the server, the requests are queued up, and so until the boxed get jammed. Being all impatient to retrieve your messages, which is quite understandable, you are frequently requesting the download of your messages, which worsen the problem. Yesterday's reboot improved the situation, but the requests being again multiplied today, the problem arises once more.
Our engineers are working on solving the issue and our system administrators are fighting to give you all access again. A temporary solution is in place : a limitation of the download requests. This means that if you do not get your emails following a download request, the most efficient thing to do is to wait 5 minutes before trying again. This will insure a speedy recovery of emails for all.

Please accept our apologies for the inconvenience, and rest assured that we are taking all the necessary measures to put an end to this problem once and for all.

UPDATE: Thursday, 11:29 AM (Paris time)

We are going to go into more detail soon on our blog, but here is a quick summary of the situation.

The server load problem was fixed this morning. We are still in the process of delivering messages that date from yesterday for some customers.

In all appearances we have a kernel bug on our hands (on NSF), but we have found a way to work around it at present. We are in the process of identifying it so as to put in a patch.

For those of you that are still experiencing difficulties, we need you to contact us at mail@support.gandi.net, so that we can help you on a one-to-one basis AND to provide us with more information.

We sincerely do apologize for this situation and the performance of this service, which has indeed been operating well below our objectives over the past few days. E-mail is a service that can never be perfect (and nobody in the world is), but the fact of the matter is that we do have an obligation to provide you with a service that is free from reoccurring problems. Whether or not the service in question is free is not the question; we are speaking of a basic service.

We have done everything possible over the past few months (software and hardware) to seriously put an end to the repetitive problems of this service in particular. Yesterday's incident, despite of the bug in question, something that makes all of this even more infuriating. This will not happen again, as 2009 will be a year of shoring up the existing services (something that I will go into detail in an upcoming entry).

Those of you that, despite everything, continue to give us that most precious thing which is your trust, will not regret it. I think that once the anger of the moment has passes, which I share, the majority of you will realize that we are still here: honest, up front, and professional.

UPDATE: Thursday, 7:40 PM (Paris time)

Some users may be experiencing delays in mail delivery of up to three hours, which will subside through the evening.

Additionally, some users reported being unable to access their mailbox between 4pm and 5pm -- this has now been fixed.

We have identified certain operating system bugs and other issues associated with the heavy load on the server platform and are working to apply the necessary updates to these systems in order to avoid any reoccurence of the problem.  We will verify these updates from Friday.

We will, of course, keep you informed of any night time maintenance periods should they be necessary.

Eurid, registry of .eu domain names is currently proceeding with a maintenance (scheduled from 6PM to 2AM today - Paris Time : GMT +1).

As a result all operations concerning this extension are suspended. Please proceed with your operation after this time.

We apologize for the inconvenience.

Update : maintenance on Eurid is completed.

Hello,

Due to a technical problem with the server that processes customer questions, the usual response time of 6 hours might not be met today. We are doing everything possible to answer your questions as soon as possible.

Please accept our apologies for any inconvenience this may cause.

Sincerely, The Gandi Team

Hello,

We are currently performing domain maintenance on Gandi's webmail service, which should take about ten minuted. We apologize for any inconvenience this may cause.

The technical team.

UPDATE 11:41 CET: Operation completed

Hello,

Following an abnormal charge on our mail servers, some users will experience difficulties to access their mail box. The problem started this morning around 11 AM, and was solved around 1.30 PM. It has arisen again however, and our technical team is currently working on a permanent fix. Please accept our apologies for this interruption.

We will be performing an update on the blog platform today. The service might be made temporarily unavailable.

Please accept our apologies for this potential disruption of service.

The technical team.

Edit : The maintenance is finished

In order to improve the redundancy of some servers, we have scheduled maintenance for the evenings (European time) of May 28th and 29th. During this time, some secondary services may be rendered temporarily unavailable.

This action should go unnoticed, though you may find that the following services may be slower and/or unavailable between 10:00 PM and 2:00 AM (GMT):

• GandiMail: some of you will not have pop3/imap access, or will experience slow mail delivery
• Some DNS servers (though not all at the same time)
• possibly (though for a very brief moment) access to the gandi.net website

We will be making changes on our network between midnight and 1am, in order to upgrade our connection to 2gbps. This interruption will cut/slow down/disturb access to Gandi sites and services for only some of you (23% of you, to be precise).

This modification is part of the tests begun last week to increase the bandwidth per hosting share.

We apologize for the disturbance.

Hello, tonight (02/28 to 02/29), our technical team is going to change some hardware on one of our servers. This maintenance is scheduled from 2 AM to 3 AM tonight, Paris time. The following services will be impacted :

• Reseller's API-XML
• Gandi Mail Operations
• Gandi Hosting Operations

This does not mean that emails will not go through or that servers will be unavailable, but only that certain operations (reboot, disk creation, adding a share, etc...), will need to wait the end of the maintenance to be validated.