Messages & Announcements

  • 2010-07-19:  Firefly planned outage July 19, 2010
    Category:  Maintenance

    We will schedule this outage Monday, July 19, 2010, starting 11:30 AM. This outage may potentially last until late in the day. We will send out an announcement once this patch has been applied.


    On July 2, 2010, we discovered a bug with the Panasas storage on Firefly. We sent out a message not to delete files larger than 2.8GB on /home or 7GB on /work as the storage blades on our Panasas storage would crash.

    Panasas has now made a fix available for this issue. We will need to schedule an outage on Monday, July 19, 2010 to apply this patch. While we understand that this means yet another outage for Firefly, this is a critical issue that needs to be solved to prevent potential data loss on /home and /work.

    We will schedule this outage Monday, July 19, 2010, starting 11:30 AM. This outage may potentially last until late in the day. We will send out an announcement once this patch has been applied.


    We apologize for the inconvenience caused by this outage.

  • 2010-07-12:  Firefly available for use
    Category:  System Failure

    Firefly is now available for use. The filesystem repairs ran over the weekend and completed successfully. We do not anticipate any data loss.

    During this outage, your jobs may or may not have completed successfully. Please check your jobs and resubmit them if they failed.

    While Firefly is available, the software bug in the Panasas storage device remains. The current estimate of when the bug fix will be available is this Friday (July 16, 2010). We will once again need to take Firefly offline to apply that software fix. We will send out an announcement when we have more information.


    Firefly is now available for use. The filesystem repairs ran over the weekend and completed successfully. We do not anticipate any data loss.

    During this outage, your jobs may or may not have completed successfully. Please check your jobs and resubmit them if they failed.

    While Firefly is available, the software bug in the Panasas storage device remains. The current estimate of when the bug fix will be available is this Friday (July 16, 2010). We will once again need to take Firefly offline to apply that software fix. We will send out an announcement when we have more information.

  • 2010-07-09:  Firefly remains unavailable
    Category:  System Failure

    Firefly remains unavailable today. We are still making progress on the filesystem repairs, but for now, our best estimate of when Firefly may be available again is Monday, July 12, 2010.


    Firefly remains unavailable today. We are still making progress on the filesystem repairs, but for now, our best estimate of when Firefly may be available again is Monday, July 12, 2010. We will send out an announcement at that time.

  • 2010-07-08:  FIREFLY DOWN
    Category:  System Failure

    Firefly's storage system has failed due to the previous announced bug. We are shutting down SSH. You will be notified when the storage is repaired.


    Firefly's storage system has failed due to the previous announced bug. We are shutting down SSH. You will be notified when the storage is repaired.

  • 2010-07-08:  Firefly /work space unavailable (again)
    Category:  System Failure

    Firefly's /work space is once again unavailable. Our storage system lost a storage blade and the system preemptively took the /work volume offline to prevent further issues.

    We are working with our storage vendor to resolve this issue.


    Firefly's /work space is once again unavailable. Our storage system lost a storage blade and the system preemptively took the /work volume offline to prevent further issues.

    We are working with our storage vendor to resolve this issue.

Pages