Gandi IWI Blog

  • Version français
  • English version
  • English version

To content | To menu | To search

Maintenance

Entries feed

Thursday 2 July 2009

Explanation of the outage that lasted 4 hours

Dear Customers,

As some of you may have noticed, we experienced a power outage from 11:30 AM to 3:30 PM (CET) today. We apologise for the inconvenience caused.

The company which manages the datacenter, that houses one of our server rooms performed maintenance this morning on some of their electrical equipment. Unfortunately, a human error by the provider undertaking the maintenance led to a total blackout of the datacenter for several minutes. The outage impacted many major companies on the internet (Dailymotion, Skyblog, Pixmania, etc.), as well as a portion of our customers.

Our system automatically transfers data to another server room in the event of a problem and it was working normally until the threshold was reached too quickly. There are several reasons for this: a massive outage on a scale seldom seen, combined with our choice to build-up the other server locations gradually rather than all at once, a server rebooting process that still requires perfecting and the success of our free server operation...in short, a chain of events caused this outage for some of you.

It goes without saying that we will examine what we could have been done at our end to prevent this, for example, a quicker building up of the other server locations (the hosting operation helps this enormously), and the continued optimization of the rebooting script (in the event that the electricity gods or the datacenter's subcontractor are still out to get us).

Additionally, we will ensure we refund you for today, and are at your disposal for any additional requests you may have. We will be adding one day to the expiration date for all of the affected hosting resources throughout the day.

Please accept our sincere apologies for this inconvenience. We are working to ensure that our infrastructure is built-up in a way which assures good performance and excellent management, so that ALL of you can benefit from our technology.

Technical note: In the event of a Gandi AI server, if your server responds but one of your services (web, ftp, etc.) is not working correctly, you can fix this by simply stopping and starting your server from Gandi's website. In the event that you have an expert server, we recommend that you activate your account's emergency console and execute a fsck command on your disks.

click here to comment this news

Wednesday 1 July 2009

Maintenance on Gandi Network Infrastructure

We are performing some minor maintenance on the access and distribution layers of the Gandi Network.  During this time, users may experience slight interruptions of connectivity for the hosting services, but these should last only at most a minute or so for the network to restabilise.

We apologise for any inconvenience.

Update: This maintenance is now completed. (15:15 CEST)

Tuesday 30 June 2009

Late mail delivery for 27% of email mailboxes

Following a minor incident on the GandiMail platform, 27% of the email mailboxes have be experiencing late delivery (5h maximum). No emails will be lost.

update 05h15 : the delay on the mail delivery is now resorbed.

Please accept our apologies for any inconvience this may have caused.

Monday 8 June 2009

Customer Service Disruption [resolved]

Hello, Due to a problem with the fiber optic network at our headquarters, the customer service department is unavailable until further notice. Please accept our apologies for any inconvenience this may cause.

UPDATEThe support team once more has access to all of its tools, though has a large backlog to recover. We are doing everything possible to return to an acceptable quality of service.

Saturday 6 June 2009

Incident on the hosting.

Due to a network equipment failure, some virtual machines are now unreachable. Our staff is currently working on fixing this issue. We apologise for any inconvenience.

12:15 AM : All maintenance are completed.

Thursday 4 June 2009

Network Infrastructure Upgrades - 4-5-6 June 2009

We will be performing upgrades to our optical network infrastructure during the nights of 4-5 and 5-6 June 2009 to install new optical mulitplexers. During this activity, customers may experience several short periods of interruption to Gandi services lasting approximately five minutes each. The work, however, will be performed in a staged manner to minimise the impact on service. This work will further optimise our network infrastructure as well as providing additional capacity for growth in the coming months and years so that we can endeavour to provide the highest quality service for our customers.

We apologise in advance for any inconvenience that this work may cause.

Update: 6/6/09 07h00: These upgrades have been successfully completed.

Tuesday 26 May 2009

Interruption in the reception of English support mails

Dear all,

Due to the recent implementation of a new software application for the management of your support requests, we have encountered a problem whereas mails sent from the support form on the English version of our website were not being sent to our support team over the past few days.

However, because we have a record of mails that were sent via the form, we have been able to recover these "lost" mails and re-inject them into our support processing tool. Our team is working overtime to assure that we get your questions replied to as soon as possible. If your problem is particularly urgent, please contact us again using the online form (which now works), by including the keyword URGENT in the "domain name" or "IP" field.

It should also go without saying that customers whose activity was financially affected by the non-response to one of these tickets will be able to write to us concerning an eventual compensation, which we will examine on a case-by-case basis.

We ask that you forgive us for this unique service interruption, and accept our apologies for any inconvenience this may have caused.

Sincerely,
The team at Gandi

UPDATE 28/05/09 : We have finished replying to each and every mail that was sent to us during that period. We expect to be within the usual response time shortly.

Friday 22 May 2009

.cc .tv .name domains unavailable from the registries

Due to the unavailabilty of the registry responsible for the .CC, .TV, and .NAME domains, new domain creations are delayed, and real-time operations (whois, DNS changes, etc.) are temporarily unavailable.

We apologise for any inconvenience.

EDIT : The three registries are now available. We apologized for the inconvenience.

Tuesday 12 May 2009

Gandi Mail: maintenance

As a result of an issue detected on one of the Gandi mail file servers, we are performing urgent maintenance on the platform.

During this maintenance, users may experience some service interruptions.

We apologise for any inconvenience this may cause.

Update: 02:18 pm -- maintenance is still in progress.

Update: 08:47 pm : After the end of the maintenance of the storage unit gandimail service was unreachable due to high load. The situation should be completly corrected in the middle of the night. Some maintenance will take place during the night to prevent the problem from happening again.

Update: 11:13 pm: There is no delay in delivery any more. We should perform some maintenance during the night therefor some brief indisponibility could occur.

Update 12/05 06:41am: All maintenance are completed.

Update 12/05 11:22am: Some of our users keep experiencing connection problems to their inbox, and / or unusually long delays in the reception of their emails. Our technical team is currently working on fixing these problems.

Update 12/05 : Maintenance is now complete.

Thursday 7 May 2009

Billing processing delays

Hello,

Our billing system has been encountering some problems following a last night's maintenance operation by our payment partner. As a result, there has been a delay in the completion of some of your orders. We are currently working on the problem and will get it corrected as soon as possible.

Please accept our apologies for any inconvenience this may have caused.

UPDATE 10;12AM CET: The problem has been corrected. All of the orders will be completed in a few minutes.

Tuesday 5 May 2009

Incident with an edge core router.

We are currently encountering an incident with one of our core edge routers and will need to restart the affected equipment. This will cause some of our transit and peering routes to flap for several minutes whilst the network stablilises. Some users may encouter a short but temporary loss of service while routes reconverge.

We apologise for any inconvenience.

Updated (15:00) : The problem appears to be with a memeber at the PaNAP peering exchange in Paris. For the moment we have shut down our port to the PaNAP until the issue there is resolved, at which time we will bring our peering sessions at PaNAP back up.

Updated (18:00): The problem at the PaNAP has been resolved and our BGP peering sessions are now restored.

Monday 4 May 2009

Gandi Blog: service disruption

We have identified an issue with one of the servers responsible for hosting blog data. We are currently working to correct the problem.

We apologise for any inconvenience.

Edit : Gandiblog are now ok

Wednesday 29 April 2009

Slight disruptions to access Gandimail (POP3)

The access to our Gandimail service in POP3 is slightly problematic. We are actively working on stabilizing this access.

The technical team

Tuesday 21 April 2009

Slower creation time for GandiAI servers

The creation and update of servers using GandiAI is currently taking longer than usual due to an issue with the package server. We are working to resolve the issue.

The technical team.

edit: Following ubuntu gutsy 7.10 end of life and removal from the official repositories, we are setting up a new dedicated mirror. It is currently synching and will be fully available in a few hours.

GandiAI installations will occur at this time with no intervention.

Experts users will benefit from this repository by upgrading /etc/apt/sources.list with the following repositories:

deb http://mirrors.gandi.net/ubuntu-old gutsy main universe multiverse
deb http://mirrors.gandi.net/ubuntu-old gutsy-security main universe multiverse
deb http://mirrors.gandi.net/ubuntu-old gutsy-updates main
deb http://mirrors.gandi.net/gandi/ubuntu gutsy main

edit: problem is now fixed

Wednesday 15 April 2009

Slight Delays in Hosting Operations

We will be performing scheduled maintenance this morning on our hosting API.

As a result, during this time, basic administrative operations (creation, stop/start of VMs, attachment/detachment of disks, addition/removal of memory/cpu) may be delayed by between 15 and 30 minutes.

We apologise for any inconvenience caused.

edit: maintenance is now finished and did not delay your operations

Friday 27 March 2009

Maintenance on Gandimail servers.

We will be performing maintenance on the Gandi Mail service tonight (Friday 27th March) at 11h00 (CET) for approximately one hour. Access to mailboxes with IMAP or POP3 may be unreachable. Reception of emails will not be affected.

The technical team of Gandi.

Wednesday 25 March 2009

Network Routing Modification 25 March 2009

We will be performing two minor modifications to our internet routing policy this afternoon at 14:30 CEST and 15:30 CEST. During these times customers may experience a very brief (60 to 90 seconds) interruption to Gandi services, but we will endeavour to minimise the impact insofar as possible.

We apologise for any inconvenience.

UPDATE: This modification has been completed.

Gandi Mail : Reduced Performance

We have encountered an incident on the web mail platform which is impacting the performance of the system. Our technical team is currently working to resolve the issue.

Update : The issue is now fixed. (02h00 CET)

We apologise for any inconvenience caused.

Monday 23 March 2009

Gandi Mail : Maintenance

We will be performing maintenance on the Gandi Mail platform on Tuesday 24 March from 11:00 to 11:30 PM CEST. During this time, customers may experience some minor disruption to the mail service.

We apologise in advance for any inconvenience this may cause.

Problems with GandiBlogs

Since 7:00 this morning (Paris time), the blogs have been responding slowly due to a heavy load on the database. The problem has been identified and is currently being corrected by our engineers. Please accept our apologies for the inconvenience.

Update 11:30 (Paris time): Blog access has returned to normal

Thursday 19 March 2009

Network Infrastructure Upgrades - 20 March 2009.

We will be performing a number of significant upgrades to our network and routing infrastructure during the early hours of 20 March 2009, beginning at 2am CET. During this engineering work, Gandi services may be subject to short periods of disruption for approximately two hours. We apologise in advance for any inconvenience that may be caused.

UPDATE: This work as been completed.

Wednesday 18 March 2009

Temporary Disruption of Hosting Services

As a result of an anomaly encountered on our hosting management system, purchases and configurations for hosting shares are temporarily unavailable. We are currently working to resolve the issue.

We apologies for any inconvenience caused.

UPDATE: The problem has been resolved.

Thursday 12 March 2009

Technical incident with the blogs and groups

One of our fileservers encountered a hardware issue which affected blogs and the groups portal for almost 20 minutes.

The problem is now fixed.

We apologize for any inconvenience.

Wednesday 11 March 2009

Gandi Mail : Maintenance

We are going to proceed with some maintenance on gandimail platform between 22:00 PM CEST and 23:00 PM CEST. During this period some minor interruption may be encountered while accessing your mailbox.

We apologize for any inconvenience this may cause.

EDIT: Done

Friday 27 February 2009

Filer inaccessibility

Following a problem on a storage unit, the hard disks of 215 virtual servers have been rendered inaccessible. We are working on a resolution to the problem. The disks will soon be accessible.

Please accept our apologies for any inconvenience this may have caused.

The Gandi team

- page 1 of 3