Triton Resource Web Home

Updated 11:00 a.m. Monday, February 11, 2013: Switch Maintenance Complete

The switch work is completed, and Triton should be fully accessible again.

Posted 10:30 p.m. Monday, February 11, 2013: Short Downtime for Switch Updates Starts Approx. 9:30 a.m.

Required switch maintenance is currently underway on Triton. Users will be unable to connect to Triton during the brief downtime. A follow-up will be posted when the sytem is accessible again. This will not interfere with running jobs.

For current Triton Resource users with outstanding balances of computing time, we are adopting the following policies:

1. Users with balances of computing time purchased prior to January 1, 2012 will receive credits on the new system on a one-for-one basis (1 SU on Triton Resource = 1 SU on TSCC). Converted SUs will expire on March 1, 2014.
2. Users with computing time purchased on or after January 1, 2012 will receive credits on the new system based on the value of their remaining credits and the new rate on TSCC ($0.03 per SU for UCSD users - rate subject to approval by UCSD Recharge Committee - so users will receive approximately 2 SU on TSCC for every SU on Triton Resource). Converted SUs will expire on March 1, 2014.
3. For external (non-UCSD) users with outstanding balances, we will be sending out amendments to contracts or service agreements over the next two months.
4. Current trial account balances will be canceled and trial users must re-apply on TSCC. New trial accounts will offer 250 SUs for a 90 day trial period.
5. Remaining balances from special programs (e.g., TRO, SDSC Affiliates) will not be transferred to TSCC.

Updated 3:00 p.m. Monday, October 15, 2012: Maintenance Completed Ahead of Schedule

The work on Oasis has been completed, and Triton is back in service. Queued jobs should begin running soon; let us know if you see any problems.

Updated 11:15 a.m. Monday, October 15, 2012: Login Nodes Will Stay Online

/oasis/triton/scratch has been unmounted across Triton, and the hardware work is getting underway. We were able to unmount from the login nodes without rebooting, so plan to leave them up for the duration.

Updated 7:45 a.m. Monday, October 15, 2012: Full-day Downtime Expected

Just a reminder that Triton will be offline for hardware work today Monday, October 15th. Any jobs submitted with a walltime that stretches into that period will be held until the system is back online.

Posted 10:00 a.m. Tuesday, October 2, 2012: Full-day Downtime Expected

Triton will be going offline on Monday, October 15th to replace the RAID controllers on the Lustre filesystem. There are a lot of them, so the current plan is to have the system offline for most of the day. I'll post timing details as the day approaches, but you should figure that Triton will be unavailable from 8:00 a.m. until sometime late in the afternoon.

Updated 1:00 p.m. Tuesday, August 14, 2012: User Access Re-enabled Today

Although the copy-back still isn't done, the filesystem has been mounted across Triton. Members of the jcsung, stsi-group, and nmalloy-mss groups should be aware that the contents of your directories are still incomplete, and those users should delay accessing data until they are directly informed that the copying process has finished. All other users, please feel free to begin using the filesystem.

There's still a bit of clean-up to do—a process is being run to make sure that all files belong to group scratch and that the setgid bit is set for all directories. That will take a while to complete, but, with any luck, won't cause difficulties in using the filesystem while it runs.

The new mount point is /oasis/triton/scratch. Remember that new policies now take effect for the parallel filesystem, including a 90-day purge policy. Please report any problems you encounter, and thank you for your patience.

Updated 2:45 p.m. Monday, August 13, 2012: Final Data Copy Almost Complete

We had a handful of dropped connections over the weekend which has slowed the data copy, but we're down to the last two directory trees. We will post as soon as the filesystem is available.

Updated 5:30 p.m. Thursday, August 9, 2012: Final Data Copy Underway

Copy of data back to the rebuilt parallel filesystem has begun. That process will likely take about a day and a half. We'll let you know as soon as the filesystem is remounted and available for use.

Updated 4:00 p.m. Wednesday, August 8, 2012: Final Performance Tests Underway

We're doing a final test to check concurrent bandwidth to the parallel filesystem. Assuming that checks out, we will begin copying back data later this afternoon—a process that should complete on Friday. At that time, the new PFS should be accessible from Triton for running jobs.

Updated 2:00 p.m. Thursday, August 2, 2012: Maintenance Extended Through Weekend

After reconfiguring and rebuilding the system, testing showed that the data transfer from two of the data servers is extremely show. The vendor has tried replacing drives and controllers, but the problem persists. Diagnosis is continuing.

Once the problem is fixed and we've verified the performance of the file system, it will take about 36 hours to transfer the contents of /phase1 onto the rebuilt system. So, the parallel file system is likely to remain offline at least until Monday, August 6. Apologies for the continued delay; we will post as soon as we have more information.

Updated 2:15 p.m. Thursday, July 19, 2012: Week-long Phase 1 Maintenance Starts Tomorrow

This is a final reminder that the Phase1 PFS will be going offline tomorrow morning. Umounting will commence at approximately 8:00 a.m. One or both login machines may require reboots to clear hung mounts; if so, a warning will be sent to the triton-discuss list before starting the reboot.

Posted 2:00 p.m. Monday, July 16, 2012: Triton will continue to run without access to /phase1

A reminder that Phase1 will be down for a week beginning this Friday. Any purging you can do of data from there that you no longer need would be appreciated.

Posted 11:00 a.m. Thursday, May 27, 2012: Entire cluster affected, restart in progress

Yesterday's power outage, which severely affected the entire UCSD campus, caused a loss of power to the SDSC machine room and took Triton offline. Power was restored by 2 a.m. and we are in the process of bringing systems back online. The outage was caused by a transformer fire in the main substation that serves the UCSD campus.

Please note that there are several services (network, disk resources for filesystems, etc.) that need to be restored before the compute cluster can be brought online. We will post to the discussion list and this web site when the system is fully functional again.

Updated 4:15 p.m. Monday, May 7, 2012: Phase 1 reconfigured and in service

Triton is back up. We'll be monitoring /phase1 operation and performance closely; please post if you encounter any problems, and thanks for your patience.

Updated 3:00 p.m. Monday, May 7, 2012: Phase 1 still down, no estimate of completion time

Work continues on bringing /phase1 back up fully--no updated estimate at this time as to when Triton will be back online. Our apologies for the delay. We've had a card failure on one of the /phase1 servers, which lengthened the downtime.

Posted 3:00 p.m. Friday, May 4, 2012: Phase 1 reboot requires scheduler suspension

We're going to take Triton down for some brief maintenance on Monday, May 7. We'll be rebooting the /phase1 fileservers and doing some rewiring on the Myrinet switch. Estimated downtime is one hour; we'll likely reboot the login nodes at the same time to clear any mounts that go over the Myrinet.

We have a reservation in place to prevent jobs from starting during this maintenance period. Please post to the discussion list if you have any questions.

Updated 10:30 a.m. Monday, Mar. 12, 2012: Users Can Re-login to Update Tools Path

We've made the jump to Maui and will be watching closely during the day. Scheduler-related tools such as showq are now found in /opt/maui/bin instead of /opt/moab/bin. Logging out of Triton and back in will be sufficient to pick up the replacement versions, or you can modify your PATH variable directly. Please post if you notice any problems—most likely a submitted job that stays queued for no apparent reason.

Updated 2:30 p.m. Thursday, Mar. 8, 2012: Scheduler Upgrade Postponed to Monday, 3/12/2012

We're postponing the final switch from Moab to Maui until Monday, 3/12. At that point we'll remove Moab from the login nodes, so users will need to reset their paths to pick up the tools in /opt/maui/bin.

Updated 5:35 p.m. Wednesday, Mar. 7, 2012: Scheduler Software Upgrade Delayed

We've been unable to get Maui running this afternoon, and we're going to pick up trying in the morning. In the meantime, we've restarted Moab so that jobs can proceed. We'll follow up with any status changes.

Posted 1:05 p.m. Tuesday, Mar. 6, 2012: Scheduler Software Being Replaced

We're planning a brief downtime for Triton Wednesday, 3/7/2012, at 2:00 p.m. for some scheduler work--our Moab license is expiring, and we're opting to switch to the Maui scheduler instead of renewing. The change-over shouldn't affect any running jobs, but there will be an interruption in your ability to submit new ones. After the change is made, you'll need to reset your path on the login nodes to pick up Maui's versions of the scheduler tools (e.g. showq), rather than Moab's. An easy way to do this is to log out and back in.

We'll post follow-up messages as the work proceeds.

Updated 9:05 a.m. Monday, Feb. 13, 2012: SDSC Switch Reboot Completed Successfully

The switch reboot is completed, and jobs are running on Triton again. We'll be doing testing this morning to see if the reboot fixed the problem with writes to /phase1.

Triton Unavailable Monday, Feb. 13 from 8 a.m. to 9 a.m. PT

Updated 1:00 p.m. Friday, Feb. 10, 2012: SDSC Switch Reboot Scheduled for Feb. 13 8-9 a.m.

In order to fix a problem some people are having with large writes to /phase1, we'll need to reboot our network switch. We have a one-hour downtime scheduled for Monday, 2/13, at 8:00 a.m. for this purpose. Newly-submitted jobs that run into that time period will be queued until after the downtime.

For questions, please email the Triton Discussion List, triton-discuss@sdsc.edu.

Triton is available again after /phase1 interface work

Updated 6:00 a.m. Tuesday, Feb. 7, 2012: SDSC Switch Maintenance Scheduled for Feb. 6 9 a.m. - 9 p.m.

Triton is available again. If you run into any issues, please email the Triton Discussion List, triton-discuss@sdsc.edu.

Triton Unavailable Monday, Feb. 6, 2012 from 9 am to 9 pm PT

Posted 1:00 p.m. Friday, Feb. 3, 2012: SDSC Switch Maintenance Scheduled for Feb. 6 9 a.m. - 9 p.m.

Triton will be unavailable next Monday for work on the /phase1 interface. Any jobs that extend into Monday will stay queued until the work is completed. We've set aside a full day, but we hope to have it done by 5 p.m.

For questions, please email the Triton Discussion List, triton-discuss@sdsc.edu.

Posted 3:00 p.m. Thursday, Sept. 22, 2011: Data Oasis Now Referred to as Phase One

SDSC has announced a new 800 TB Lustre filesystem, the first phase of a large high-performance filesystem that will be available to Triton users early next year. Phase One is available from Triton now, and currently has 16 storage servers, each with 4 object storage targets (64 OSTs total) and a peak measured bandwidth of 12.5 GB/s.

All current Triton users have an assigned directory in /phase1 which is accessible now. As of today Sept 22, 2011 the old filesystem (/oasis) is read-only. All users are requested to move their data from /oasis to /phase1 by Oct 10, 2011 at which point /oasis will be retired from service. Users are requested to use compute nodes or the alternate Triton login node (triton-38) to move their data between filesystems to avoid overloading a single client machine. Please contact the Triton Discussion List if you need assistance.

The /phase1 filesystem is intended to be high performance scratch storage and is subject to a purge policy. The filesystem is not backed up and should not be used for long-term storage. Users are reminded that any important data must be moved to their own local storage resources.

Updated 8:00 a.m. Wednesday, Sept. 21, 2011: SDSC Switch Maintenance Completed Successfully

The upgraded switch maintenance was completed at approximately 8:30 p.m. Tuesday, Sept. 20. Triton queues should again be accessible from external locations.

Updated 10:30 a.m. Tuesday, Sept. 20, 2011: Maintenance to Continue This Evening

After a longer than expected delay yesterday, the two lustre systems are finally remounted around Triton and the queues are pushing through jobs again. We will make every effort to keep this evening's outage time to a minimum.

From 6:00-9:00 tonight the network folks will be upgrading the switches that connect SDSC to the outside world. During that time, if you're located somewhere outside SDSC, you're likely to be unable to connect to Triton. The good news is that this work shouldn't affect connectivity within SDSC, so jobs in the queues at 6:00 p.m. should be able to run without difficulty--you just won't be able to see the results until outside connections are restored. If you have jobs that happen to need outside connections (e.g. they rely on an externally-mounted disk), it would be a good idea to put a hold on them until after the work is complete. See http://status.externsdsc.org/ for status updates.

Posted 9:30 a.m. Monday, Sept. 19, 2011: Due to Reopen Around Noon

The triton batch queues will are unavailable today, Monday, September 19th, until around noon for some switch work. As part of this work, both /phase1 and /oasis will be unmounted for the duration.

We will post as soon as /phase1 and /oasis are remounted.

Posted 12:30 p.m. Friday, Sept. 16, 2011: Reservations will prevent some jobs from being scheduled until after completion of downtime

There is a system-wide reservation in place for Data Oasis maintenance on Monday, Sept. 19 from 9 a.m. to noon. Jobs which specify end times after the start of this maintenance will not be scheduled until after it ends. There is also a reservation for Tuesday, Sept. 20 from 6 p.m. to 10 p.m. with the same effect on scheduling.

Updated 6:30 p.m. Tuesday, Sept. 13, 2011: Rebooting Both Login Nodes Around 8:00 p.m. PT

We need to reboot/reinstall the login nodes once more to clear some hung processes and fix a problem with the MPI installation. We will start by shutting off new logins to the alternate login node (triton-38). That will probably cycle by 8:00 p.m. After it comes back up, we'll do the same for the primary login. Apologies for the instability; with luck, this will be the last service interruption we'll see in the aftermath of Thursday's outage.

Updated 11:45 p.m. Sunday, Sept. 11, 2011: Primary Login Node Available Again

triton-login is back up; triton-38 was also down. Details will be forthcoming as we learn more about the cause.

Updated 12:00 p.m. Sunday, Sept. 11, 2011: Alternate Login Node in Use Until Further Notice

Our main login machine, triton-login, seems to be down. Please use our alternate login, triton-38.sdsc.edu, until further notice.

Updated 12:00 p.m. Saturday, Sept. 10, 2011: Triton Back Online

Triton has mostly recovered from the faceplant it took during the power outage. As mentioned, we have temporarily lost a file server (but not the data stored on it), and we had to replace the cluster management server, which has some lingering reprecussions. Here's what we know is presently *not* working:

• Changing management hardware invalidated most of our software licenses. Yesterday, we replaced the ones we could. The PGI compilers aren't working at this point; you may run into an application or two that we haven't tested yet.
• None of the /projects directories is mounting. The Triton-hosted /project directories are hosted on the failed server. The one piece of data that we seem to have lost completely is the list of partitions from external servers that we were mounting under /projects. If your project had an external mount, please send us the server and mount point, and we'll start rebuilding the list.
• A handful of the batch nodes (tcc-*) are still down. We have likely lost some local disks and switch ports.

We'll be working on these problems, plus anything else we discover, over the next week. Please use Triton with some caution during this shake-out, and post any problems that you encounter.

Updated 3:10 p.m. Friday, Sept. 9, 2011: Mid-afternoon update

Triton management node (New Hardware) is up and running and is busily building the rest of the nodes. The major issue we see is that one of the Project NFS servers refuses to boot. That particular issue will not be solved until Monday or Tuesday.

The specific projects that are affected are mounted under /projects:

       cgl-group
       frazer-group
       geogrid-aist
       liai-group
       nrnb-group
       ren-lab
       biogem-lab
       camera
       camera-lab
       crbs-group
       gleeson-lab
       lca-group
       mmiller-group
       sarkar-lab
       zhang-lab

If you have data in these directories, we have backups of your data as of approximately 4:00 a.m. on Sept. 8. If you need access to your replicated data, please contact Phil Papadopoulos or Jim Hayes. We can give you some options (read only, read/write, etc.), but we would want to talk to you individually. Our expectation is that the data on the primary Projects server is intact, we simply can't see it until hardware is addressed.

Home area data (e.g. /home/user) is stored on different servers and is unaffected.

All data partitions on /oasis and /phase1 have been checked and look to be in good shape, too.

We hope to have Triton otherwise restored by the end of the day (but it may go into the weekend). So far, the rebuild of is going smoothly.

Posted 1:30 p.m. Friday, Sept. 9, 2011: Damaged hardware will be replaced today

The power outage wreaked havoc with Triton's management node. We're replacing the hardware and rebuilding. We'll let you know when things are back in operation.

User data (held on different systems) are all intact.

Updated 11:00 a.m. Wednesday, June 29, 2011: Maintenance completed at 4:00 p.m.

Triton is available again after a longer-than-expected downtime. We've been struggling mostly with getting the updated batch scheduler software to work as desired; it seems like most of the kinks are worked out now.

Feel free to log in and resume working. Please keep a somewhat closer-than-usual eye on your jobs for the first few days, and post to Triton Discuss if you encounter anything that doesn't look right.

We appreciate your patience.

Updated 8:00 a.m. Tuesday, June 28, 2011: Short delay in upgrade procedure

We've run into some difficulties getting Triton to stand up properly after the software upgrade. We're continuing to work on it, and will post a follow-up when the system becomes available again. At this point, we do not expect to have it available until sometime Tuesday. We apologize for the delay.

Posted 2:00 p.m. Thursday, June 23, 2011: OS upgrade and new software will be available

Triton will be offline for a software upgrade next Monday, June 27th. We'll be moving from Rocks v5.3/CentOS v5.4 to Rocks v5.4/CentOS v5.6; additional new and updated applications are listed below. We've blocked out 8:00-5:00 Monday for the upgrade, but the actual downtime should be much shorter. Check this website, the mailing list, or the Triton Twitter feed for a follow-up message when the system is available again.

Please note that, unlike other recent downtimes, jobs in the queue when the system goes down will need to be resubmitted once it comes back up. Please contact the Triton Discuss mailing list if you have any questions.

New applications in /opt:

Updated applications in /opt:

Apologies for the extreme delay. Triton is back up with /oasis mounted, the login nodes are reopened, and the queues have started running again. Please post a note to the Discussion List if you run into any problems.

Updated 9:30 a.m. Tuesday, May 24: Data Oasis Communications Pending

Update on the 5/23 physical relocation: the move was completed in good time yesterday; however, Data Oasis continues to exhibit communications failures. Triton will remain offline until the problem is resolved. We will post follow-ups as we find out more.

Updated 10:30 a.m. Tuesday, May 10: All-Day Downtime Set for May 23

To make room for some new hardware, the Data Oasis servers and associated switch are going to be moved to SDSC's other machine room on Monday, May 23rd. Between the move, re-racking, and re-cabling, this will be a more involved process than the switch work we did a few weeks ago; we're figuring that we'll likely have close to a full day's downtime. The Triton submission queues will be down during this period, and access to the Triton login nodes may be shut off as well. We'll post updates as more information becomes available.

Updated 12:30 p.m. Monday, April 25: All Nodes Access Data Oasis

Switch maintenance is complete, and access between Triton and Data Oasis has been restored. The hold on jobs has been removed, so submissions should begin moving through the queues again. Please report any problems to the Triton Discussion List (triton-discuss@sdsc.edu).

Updated 3:15 p.m. Thursday, April 21: 8 a.m. Start for Switch Maintenance

Triton will be down for maintenance from 8 a.m. to 1 p.m. PT on Monday, April 25, 2011. Work will be performed on one of the switches that provides access to Data Oasis. All jobs scheduled to be running during this window will remain queued until after the connection to /oasis is restored.

Currently, there is no plan to shut off access to the Triton login nodes. However, they will probably be rebooted on short notice once the switch work is complete. Access to user home filesystems should not be affected, however most /projects filesystems will be offline during the outage.

Updated 3:00 p.m. Thursday, February 3, 2011: Brief Period of Unavailability Via Login Node

The upgrade to file servers supporting the home filesystem was completed at approximately 2:30 p.m. today.

Posted 12:05 p.m. Thursday, February 3: Brief Period of Unavailability Via Login Node

We are performing a maintenance on the home filesystem, which has caused Triton's login node to be inaccessible for a short time. We will notify through this page, the Discusson List, and Twitter when the system access is back online.

Posted 3:30 p.m. Wednesday, January 19, 2011: Students Learning to Supercompute via Triton

Triton has become a teaching resource as well as a research one in its first year on campus. Read the latest press release at the SDSC News Center.

Posted 10:30 a.m. Saturday, November 27: Lack of Cooperation Dooms Open Policy

Because of recurring issues with users simply consuming disk space without regard to space available, we have had to modify our operative, user-friendly space allocation which allowed users to expand to space needed and then contract after usage. That open policy has failed.

Hard quotas now exist on all home areas. The standard allocation is 100 GB per user. Currently, users who are consuming more than 100 GB have had their quotas set to accomodate active space as of 11/19/2010 so that users can effect cleanup of their home areas.

If you are consuming more than 100 gigabytes of space, you must either

• reduce your on-disk usage to 100 gigabytes or less



OR

• contact the TAPP Manager for a space request

At the time the policy change was instituted, there were about 60 users consuming over 100 GB of home area space. If you have not been consuming more than 100 GB of space on a long-term basis, or have been attentively clearing out your overage, we thank you!

Posted 8:30 a.m. Friday, October 29: Last Chance to Get Your Data

Last weekend's changes seem to have finally settled down Mirage. We hope you've had a chance to copy any data you'd like to keep to a new home on Data Oasis.

We plan to permanently disconnect Mirage from Triton this coming Monday, November 1st. Remember that the disks from Mirage will be erased and reused, so anything that hasn't been copied will be unrecoverable.

Posted 9:30 p.m. Tuesday, October 26, 2010: All User Data Must Be Copied to Oasis or Lost

Within the next couple days, all Mirage disks will be offlined from Triton. Any user data stored there will no longer be accessible. If you have yet to migrate all your essential data to Oasis, please inform the discussion list immediately so arrangements can be made to preserve your data.

It is recommended to use in interactive node to migrate your data to Oasis rather than a session on the login node. Here are some suggestions for how to perform that procedure:

1. Get and interactive shell via the qsub command.

   % qsub -I

2. Copy files

   There are many ways to do this, perhaps the slowest being cp -R. One simple and preferrable method is:

   % cd /mirage/<username>

   % tar cf - * | ( cd /oasis/<username>; tar xvfBp -)

You might follow the above tar with an rsync, for example:

rsync /mirage/<username>/ /oasis/<username>/

Note that rsync syntax is rather sensitive. The trailing slashes on the command above are important.

Using tar will cause reads to be well buffered, putting Lustre into more of its comfort zone for "big" files. If you have many small files, there is not an efficient way to move data, mostly because Lustre is not very efficient on small files.

Many other efficient methods are available for large files.

Updated 4:30 p.m. Tuesday, October 12: Transition of Data Is In Progress

As of today, Data Oasis is available for use. Users should being moving their data from Mirage and updating job scripts to write to /oasis immediately. On about Oct. 29, Mirage will be completely removed from Triton and user data there will no longer be available.

We'll start the process of flipping /mirage to read-only on Friday afternoon, Oct. 15, finishing sometime Monday. Unfortunately, this will require yet another round of rebooting, but we'll try to keep disruptions to a minimum.

The scratch directory for jobs can be found at /oasis/scratch/<login>/$PBS_JOBID We still need to add the code to set an environment variable to this path and to clean out the directories after three days.

Updated 1 p.m. Monday, October 11: Transition To Data Oasis Delayed

Due to complications with Mirage, the transition will start tomorrow, Tuesday, October 12. Sorry for any inconvenience this may cause. Please adhere to the below schedule, except that it be offset by 24 hours.

Posted 2 p.m. Friday, October 8: Transition To Begin Monday, October 11

Over the past couple of weeks the Triton team has been testing the first incarnation of Data Oasis (DO), a new parallel filesystem intended to replace Mirage. We're ready now to go live with the system. Given the fun we've experienced with Mirage over the past 12 hours, it looks like the timing is good.

This version of DO will have approximately 250 terabytes usable capacity, more than doubling the amount of disk space available on Mirage. DO will run the latest version of the Lustre filesystem; word on the street and our own experience during testing both indicate that we should see significant improvements in stability by making the upgrade.

Next Monday (10/11) we'll begin a two-week transition period to move from Mirage to DO. Both filesystems will be mounted across Triton, mirage at /mirage as usual, and DO at /oasis. As with /mirage, you will find a directory on /oasis named after your login id. Please modify any references you have in scripts, etc. to refer to the new filesystem, and start copying any data you want to keep from Mirage to DO. After three days (Thursday 10/14 — long enough to let any jobs running Sunday to complete), we'll remount Mirage read-only so that jobs won't be able to write any new data to it.

Unlike Mirage, we plan to reserve 50 terabytes on DO for job scratch space. Each job will have the associated scratch directory /oasis/scratch/<login>/<job#> that can be used to place data temporarily. (We'll set a job-specific environment variable to reference this directory.) These directories and their contents will be purged automatically three days after the job completes — this really is scratch space.

Also unlike Mirage, we'll be placing per-user quotas on DO usage, with the goal of avoiding the performance degradation we saw on Mirage when usage got into the high 90-percent range. Details will follow, but each user will have at least enough space to hold their current Mirage usage.

After the two-week transition period, Mirage will leave Triton and its disks will be recycled to other uses, so any data that hasn't been copied over really will be gone. As with Mirage, making backups of data on DO will be the responsibility of the users.

Please post any questions you have about this transition to the Triton Discussion List (triton-discuss@sdsc.edu), and we'll do our best to clarify and help in the move. There should be a /oasis set up sometime late Sunday afternoon; feel free to get an early start on transferring your data.

Posted 12:00 p.m. Friday, October 8: New Results from Storage Upgrade

These numbers are for up to 4 nodes using 32 cores. Tests were also run on up to 128 nodes and 1024 cores.

     Nodes       Cores        Max Write        Max Read
        1         8           378.12 MiB/sec   578.18 MiB/sec
        2         8           601.66 MiB/sec   849.80 MiB/sec
        4         8           744.38 MiB/sec   981.06 MiB/sec
        4         16          740.49 MiB/sec   1066.38 MiB/sec
        4         32          565.95 MiB/sec   1070.39 MiB/sec

The peak performance for this set was:

Please visit the Data Oasis page for more information.

Posted 5:00 p.m. Tuesday, Sept. 28: Performance Expectations Exceeded During Phase 0 Certification

Performance testing on the first phase (Phase 0) of Data Oasis over the last several days achieved approximately 3.5 gigabytes per second (GB/s) on writes and about 7.6GB/s on reads using a 2 terabyte file and 512 clients.

In networking terms, this was 64 gigabits per second, or an incredible 80% of the theoretical channel-bonded link. Another way to look at this data is:

8000/8*1135 (our best 1 OSS number) = 8000/9050 = 88% scaling efficiency

Our goal was 7 billion bytes per second, so we beat that by 15%.

In related news, the Phase 1 Data Oasis RFP has been published. Oasis should be expanding in about two months. Our goal for Phase 1 will be approximately five times the sustained speeds achieved in Phase 0.

We hope to have Phase 0 available to all Triton users within a few days.

Posted 3:30 p.m. Wednesday, Sept. 22: Campus Users Will Lose Access to Triton while Router is Upgraded

On Tuesday, October 5th, 2010 from 5 p.m. until approximately 7:30 p.m., SDSC and UCSD networking teams will update routes to utilize the new MX960 router at SDSC, and retire the older T320 router (known as dolphin). Network routes between SDSC and UCSD will be affected, and there may be connectivity issues to some hosts.

During the maintenance, UCSD Triton users will be unable to access Triton resources. Users connecting to UCSD or SDSC from external hosts should continue to have uninterrupted access to Triton and campus networks.

If you have questions or concerns, please email the Triton Discussion List (triton-discuss@sdsc.edu) or SDSC Network Support.

Posted 9:30 a.m. Tuesday, Sept. 21: Users Asked to Remove Files

We've hit a critical point on /home file usage, where there isn't enough free space for the system overhead involved in deleting files. We have recently freed approximately 1.3 terabytes, which provided headroom for cleaning up the filesystem.

We have about 530 users sharing around 44 terabytes of home space. That works out to about 83 gigabytes per user. Because not all users need that much space (we have a number of idle accounts that contain little or no data), we've so far avoided putting quotas on the system. However, if you're using considerably more than that — say, 830 gigabytes or more — then you're consuming more than a fair share of the resource. Please shift your collected data off of Triton so that we have enough room for everyone to operate.

We continue to seek additional ways to relieve the space crunch. We've shifted some users to another server, freeing up about 30 terabytes from the primary system. However, there will always be a hard limit, no matter how much disk we throw at it, as seen in the 98% usage of the 100 terabytes on /mirage. Users offloading their data onto other resources is the only long-term solution.

Update (9:30 a.m. Monday, Sept. 13): Non-UCSD personnel being temporarily denied use of software

Due to restrictions in our licensing agreement with The Mathworks, we are currently forced to limit Triton users access to both client and server MATLAB licenses strictly to UCSD users.

We regret any inconvenience this causes. We are working with the company to relax the restriction so that MATLAB may again be available to all users. We will announce policy changes when we reach a new agreement.

Update (9:30 a.m. Thursday, Sept. 2): Filesystem Capacity at 98%

/mirage has hit 98% of capacity. At that high a level of usage, performance bogs down and reliability can get a bit shaky. Please make a pass through your data on the filesystem and remove files you no longer need.

Update (1:30 a.m. Saturday, July 10): Triton_RC3 Rolls Now on Download Page

Many of the source rolls from the Rocks 5.3 upgrade completed on May 18 are now posted for download on the Triton Download Page. You can also find information on how to build a cluster like Triton on our Build Your Own page.

Update (3:30 p.m. Tuesday, June 23): Benefits and effects of new filesystem

Triton will soon support a faster, more reliable, higher capacity parallel filesystem. Some of the features and benefits are listed below. We will provide more specific data at the conclusion of testing.

Expected User Benefits

• Filesystem capacity will increase from 100 terabytes to 281 terabytes
• Redundant metadata servers will provide increased reliability; 384 terabytes total capacity
• Filesystem software upgrade to version 1.8.3 will add new features and bug fixes
• Enterprise-class server hardware and interconnect fabric will provide better performance
• System will support quotas

Anticipated User Impacts

• Filesystem name will change from /mirage to /oasis - this may affect scripts with hard-coded paths
• Phase 0: both /mirage and /oasis will be online at same time for about 30 days
• /mirage data will be destroyed after migration grace period and /mirage will be unmounted

The new hardware and software versions are currently being tested on reserved nodes of Triton. Availability to the production nodes is expected within a few weeks.

Update (5:30 p.m. Wednesday, June 9): TRITON_RC3 Upgrade Details

Following are some particulars regarding the affected packages and systems.

Updated system software:

Rocks          v5.1   --> v5.3
CentOS         v5.2   --> v5.4
PGI compiler   v8.0   --> v10.5
Lustre client  v1.6.6 --> v1.8.3
Myrinet driver v1.2.8 --> v1.2.12
Moab           v5.3.5 --> v5.3.7 (TORQUE roll update)

New applications (some of these have been in /beta; all will now have a permanent home in /opt):

  BEAST  1.5.2
  APBS  1.2.1
  LAMMPS 28Nov09
  NAMD  2.7b1
  NWChem 5.1.1
  Open Motif  2.3.2
  PDT 3.15
  TAU 2.19
  SciPy  0.7.1rc3
  FFTW v2.1.5 (in addition to v3.2.1)

In addition, performance and administrative gains include:

• Users have more refined environment control since most applications now use loadable modules rather than an auto-loaded environment.
• A significant refactoring of the library layout delivers easier maintenance for static and dynamic mpich/mpich2/openmpi libraries with Intel PGI compilers.
• Several applications now use faster Intel compiled binaries.
• The /home area file space has increased, and disk space has been repartitioned to provide better support for large applications.
• The Myrinet driver upgrade provides fixes for SRAM parity detection in the NIC. It also improves Progress on Codes with many outstanding (unmatched) messages.

Update (1:00 p.m. Monday, May 3): New Support Feature! Get approval for your job to run longer than 72 hours!

For jobs requiring more than Triton's 72-hour wallclock limit, users may now request an exception to allow those jobs to be scheduled and run. Please make your request through the discussion mailman list (triton-discuss@sdsc.edu) and system administrators will make the provisions necessary to support your request.

+ Login Node Patching Monday, Sept. 13

+ Intermittent File System Access on Thursday, August 12, 2010

+ Triton Nodes Back Online Monday, August 2

+ Partial Outage of Triton Nodes on August 1, 2010

+ Triton software upgrade on May 18

+ Free MATLAB seminars on campus Thursday, May 13

+ Storage outage successul this weekend

+ Storage outage 9 p.m. Apr 23rd — 11 a.m. Apr 25th

+ Triton Research Opportunities (TRO) Program — Call for Proposals

+ New Policy to Limit Login Node Access

+ Continuing Triton Access for UCSD Researchers

+ Computing in the New Year

+ UCSD Year-end Holiday Campus Closure

+ New Triton Login Nodes

+ New Triton Hardware

+ Triton Software Rolls Available

+ Home Filesystem Issues Nov. 9 - Dec. 7

+ Early Adopter Phase Ended October 5

+ RC2 System Software Upgrade 18 Sept 2009

+ Cluster Reinstallation 23 Jul 2009 (TRITON_RC1)

+ General Status of Triton Resource

+ Early Adopter User Accounts