Gandi IWI Blog - Maintenance

Explanation of the outage that lasted 4 hours

Nicolas — Thu, 02 Jul 2009 19:12:00 +0200

Dear Customers,

As some of you may have noticed, we experienced a power outage from 11:30 AM to 3:30 PM (CET) today. We apologise for the inconvenience caused.

The company which manages the datacenter, that houses one of our server rooms performed maintenance this morning on some of their electrical equipment. Unfortunately, a human error by the provider undertaking the maintenance led to a total blackout of the datacenter for several minutes. The outage impacted many major companies on the internet (Dailymotion, Skyblog, Pixmania, etc.), as well as a portion of our customers.

Our system automatically transfers data to another server room in the event of a problem and it was working normally until the threshold was reached too quickly. There are several reasons for this: a massive outage on a scale seldom seen, combined with our choice to build-up the other server locations gradually rather than all at once, a server rebooting process that still requires perfecting and the success of our free server operation...in short, a chain of events caused this outage for some of you.

It goes without saying that we will examine what we could have been done at our end to prevent this, for example, a quicker building up of the other server locations (the hosting operation helps this enormously), and the continued optimization of the rebooting script (in the event that the electricity gods or the datacenter's subcontractor are still out to get us).

Additionally, we will ensure we refund you for today, and are at your disposal for any additional requests you may have. We will be adding one day to the expiration date for all of the affected hosting resources throughout the day.

Please accept our sincere apologies for this inconvenience. We are working to ensure that our infrastructure is built-up in a way which assures good performance and excellent management, so that ALL of you can benefit from our technology.

Technical note: In the event of a Gandi AI server, if your server responds but one of your services (web, ftp, etc.) is not working correctly, you can fix this by simply stopping and starting your server from Gandi's website. In the event that you have an expert server, we recommend that you activate your account's emergency console and execute a fsck command on your disks.

click here to comment this news

Maintenance on Gandi Network Infrastructure

Leland Vandervort — Wed, 01 Jul 2009 13:53:00 +0200

We are performing some minor maintenance on the access and distribution layers of the Gandi Network. During this time, users may experience slight interruptions of connectivity for the hosting services, but these should last only at most a minute or so for the network to restabilise.

We apologise for any inconvenience.

Update: This maintenance is now completed. (15:15 CEST)

Late mail delivery for 27% of email mailboxes

Ryan — Tue, 30 Jun 2009 16:11:00 +0200

Following a minor incident on the GandiMail platform, 27% of the email mailboxes have be experiencing late delivery (5h maximum). No emails will be lost.

update 05h15 : the delay on the mail delivery is now resorbed.

Please accept our apologies for any inconvience this may have caused.

Customer Service Disruption [resolved]

Ryan — Mon, 08 Jun 2009 10:41:00 +0200

Hello, Due to a problem with the fiber optic network at our headquarters, the customer service department is unavailable until further notice. Please accept our apologies for any inconvenience this may cause.

UPDATEThe support team once more has access to all of its tools, though has a large backlog to recover. We are doing everything possible to return to an acceptable quality of service.

Incident on the hosting.

aegiap — Sat, 06 Jun 2009 11:15:00 +0200

Due to a network equipment failure, some virtual machines are now unreachable. Our staff is currently working on fixing this issue. We apologise for any inconvenience.

12:15 AM : All maintenance are completed.

Network Infrastructure Upgrades - 4-5-6 June 2009

Leland Vandervort — Thu, 04 Jun 2009 08:18:00 +0200

We will be performing upgrades to our optical network infrastructure during the nights of 4-5 and 5-6 June 2009 to install new optical mulitplexers. During this activity, customers may experience several short periods of interruption to Gandi services lasting approximately five minutes each. The work, however, will be performed in a staged manner to minimise the impact on service. This work will further optimise our network infrastructure as well as providing additional capacity for growth in the coming months and years so that we can endeavour to provide the highest quality service for our customers.

We apologise in advance for any inconvenience that this work may cause.

Update: 6/6/09 07h00: These upgrades have been successfully completed.

Interruption in the reception of English support mails

Ryan — Tue, 26 May 2009 17:04:00 +0200

Dear all,

Due to the recent implementation of a new software application for the management of your support requests, we have encountered a problem whereas mails sent from the support form on the English version of our website were not being sent to our support team over the past few days.

However, because we have a record of mails that were sent via the form, we have been able to recover these "lost" mails and re-inject them into our support processing tool. Our team is working overtime to assure that we get your questions replied to as soon as possible. If your problem is particularly urgent, please contact us again using the online form (which now works), by including the keyword URGENT in the "domain name" or "IP" field.

It should also go without saying that customers whose activity was financially affected by the non-response to one of these tickets will be able to write to us concerning an eventual compensation, which we will examine on a case-by-case basis.

We ask that you forgive us for this unique service interruption, and accept our apologies for any inconvenience this may have caused.

Sincerely,
The team at Gandi

UPDATE 28/05/09 : We have finished replying to each and every mail that was sent to us during that period. We expect to be within the usual response time shortly.

.cc .tv .name domains unavailable from the registries

Leland Vandervort — Fri, 22 May 2009 19:27:00 +0200

Due to the unavailabilty of the registry responsible for the .CC, .TV, and .NAME domains, new domain creations are delayed, and real-time operations (whois, DNS changes, etc.) are temporarily unavailable.

We apologise for any inconvenience.

EDIT : The three registries are now available. We apologized for the inconvenience.

Gandi Mail: maintenance

Leland Vandervort — Tue, 12 May 2009 11:39:00 +0200

As a result of an issue detected on one of the Gandi mail file servers, we are performing urgent maintenance on the platform.

During this maintenance, users may experience some service interruptions.

We apologise for any inconvenience this may cause.

Update: 02:18 pm -- maintenance is still in progress.

Update: 08:47 pm : After the end of the maintenance of the storage unit gandimail service was unreachable due to high load. The situation should be completly corrected in the middle of the night. Some maintenance will take place during the night to prevent the problem from happening again.

Update: 11:13 pm: There is no delay in delivery any more. We should perform some maintenance during the night therefor some brief indisponibility could occur.

Update 12/05 06:41am: All maintenance are completed.

Update 12/05 11:22am: Some of our users keep experiencing connection problems to their inbox, and / or unusually long delays in the reception of their emails. Our technical team is currently working on fixing these problems.

Update 12/05 : Maintenance is now complete.

Billing processing delays

Ryan — Thu, 07 May 2009 09:53:00 +0200

Hello,

Our billing system has been encountering some problems following a last night's maintenance operation by our payment partner. As a result, there has been a delay in the completion of some of your orders. We are currently working on the problem and will get it corrected as soon as possible.

Please accept our apologies for any inconvenience this may have caused.

UPDATE 10;12AM CET: The problem has been corrected. All of the orders will be completed in a few minutes.

Incident with an edge core router.

Leland Vandervort — Tue, 05 May 2009 12:15:00 +0200

We are currently encountering an incident with one of our core edge routers and will need to restart the affected equipment. This will cause some of our transit and peering routes to flap for several minutes whilst the network stablilises. Some users may encouter a short but temporary loss of service while routes reconverge.

We apologise for any inconvenience.

Updated (15:00) : The problem appears to be with a memeber at the PaNAP peering exchange in Paris. For the moment we have shut down our port to the PaNAP until the issue there is resolved, at which time we will bring our peering sessions at PaNAP back up.

Updated (18:00): The problem at the PaNAP has been resolved and our BGP peering sessions are now restored.

Gandi Blog: service disruption

Leland Vandervort — Mon, 04 May 2009 10:54:00 +0200

We have identified an issue with one of the servers responsible for hosting blog data. We are currently working to correct the problem.

We apologise for any inconvenience.

Edit : Gandiblog are now ok

Slight disruptions to access Gandimail (POP3)

Sam — Wed, 29 Apr 2009 14:29:00 +0200

The access to our Gandimail service in POP3 is slightly problematic. We are actively working on stabilizing this access.

The technical team

Slower creation time for GandiAI servers

aegiap — Tue, 21 Apr 2009 12:06:00 +0200

The creation and update of servers using GandiAI is currently taking longer than usual due to an issue with the package server. We are working to resolve the issue.

The technical team.

edit: Following ubuntu gutsy 7.10 end of life and removal from the official repositories, we are setting up a new dedicated mirror. It is currently synching and will be fully available in a few hours.

GandiAI installations will occur at this time with no intervention.

Experts users will benefit from this repository by upgrading /etc/apt/sources.list with the following repositories:

deb http://mirrors.gandi.net/ubuntu-old gutsy main universe multiverse
deb http://mirrors.gandi.net/ubuntu-old gutsy-security main universe multiverse
deb http://mirrors.gandi.net/ubuntu-old gutsy-updates main
deb http://mirrors.gandi.net/gandi/ubuntu gutsy main

edit: problem is now fixed

Slight Delays in Hosting Operations

Leland Vandervort — Wed, 15 Apr 2009 09:39:00 +0200

We will be performing scheduled maintenance this morning on our hosting API.

As a result, during this time, basic administrative operations (creation, stop/start of VMs, attachment/detachment of disks, addition/removal of memory/cpu) may be delayed by between 15 and 30 minutes.

We apologise for any inconvenience caused.

edit: maintenance is now finished and did not delay your operations

Maintenance on Gandimail servers.

aegiap — Fri, 27 Mar 2009 17:18:00 +0100

We will be performing maintenance on the Gandi Mail service tonight (Friday 27th March) at 11h00 (CET) for approximately one hour. Access to mailboxes with IMAP or POP3 may be unreachable. Reception of emails will not be affected.

The technical team of Gandi.

Network Routing Modification 25 March 2009

Leland Vandervort — Wed, 25 Mar 2009 13:01:00 +0100

We will be performing two minor modifications to our internet routing policy this afternoon at 14:30 CEST and 15:30 CEST. During these times customers may experience a very brief (60 to 90 seconds) interruption to Gandi services, but we will endeavour to minimise the impact insofar as possible.

We apologise for any inconvenience.

UPDATE: This modification has been completed.

Gandi Mail : Reduced Performance

Leland Vandervort — Wed, 25 Mar 2009 11:50:00 +0100

We have encountered an incident on the web mail platform which is impacting the performance of the system. Our technical team is currently working to resolve the issue.

Update : The issue is now fixed. (02h00 CET)

We apologise for any inconvenience caused.

Gandi Mail : Maintenance

Leland Vandervort — Mon, 23 Mar 2009 16:11:00 +0100

We will be performing maintenance on the Gandi Mail platform on Tuesday 24 March from 11:00 to 11:30 PM CEST. During this time, customers may experience some minor disruption to the mail service.

We apologise in advance for any inconvenience this may cause.

Problems with GandiBlogs

Ryan — Mon, 23 Mar 2009 10:22:00 +0100

Since 7:00 this morning (Paris time), the blogs have been responding slowly due to a heavy load on the database. The problem has been identified and is currently being corrected by our engineers. Please accept our apologies for the inconvenience.

Update 11:30 (Paris time): Blog access has returned to normal