In the event that your ReadyNAS is not functioning properly, it is quite possible that you are suffering from a failing or flaky disk.  Under normal circumstances, the ReadyNAS should detect a failed disk and kick it from the array.  In those cases, you will also receive alerts from the NAS and see information in the log files.  In some instances, however, the disk may be having intermittent read and/or write problems that can cause all sorts of weird issues ranging from a decrease in performance to data re-syncs.  It is also possible that you may not receive any alerts or notification of disk trouble, making it difficult to determine if the disks are actually the problem.

In order to effectively troubleshoot the problem, you will need to test the disks in order to eliminate or confirm that they are the source of the problem.  Testing the disks requires the following:

1. Installation of the suspect disk in a SATA-capable PC:
2. The hard disk manufacturer’s diagnostic utility (i.e. SeaTools for Seagate drives).
3. A few hours to run a thorough test for each disk.
4. 

Most of the hard disk vendors offer the utility in a number of versions, such as a bootable CD ISO version, a bootable floppy disk version, as well as a version that can be run from within Windows.  As far as I’m aware, only the Windows version allows the suspect drive to be installed in a portable USB enclosure.  Most of the other versions of the utilities will require you to connect the drive directly to the SATA channel.  Please check with your disk vendor.

SeaTools for Windows

SeaTools for Windows tests SATA, USB, 1394, ATA (PATA/IDE) and SCSI drives. It installs onto your system. SeaTools for Windows is completely data safe. If the drive passes SeaTools for Windows, your troubleshooting can move to other areas.

SeaTools for DOS

SeaTools for DOS tests SATA or ATA drives from a bootable CD-ROM or floppy. It can test a SATA (Serial ATA) or older ATA (PATA/IDE) interface hard drive. Because the software boots to its own operating system you can test your drive regardless of the OS installed on it. You can even test a new or completely blank drive. In addition, this version offers limited repair and data erasure.

The Old “pull one-disk-at-a-time” Trick:

Mac users, as well as those users that only have a notebook/laptop/netbook may need to borrow a friend’s computer or take it to a local computer shop for testing or revert to the old “pull one-disk-at-a-time” trick:

1. Power off the NAS.
2. Remove drive 1 (leave 2, 3, 4 in)
3. Power on the NAS.
4. See if it boots.
5. If not, start over at step 1 and replace disk 1 and remove disk 2. Repeat with all drives.

During the boot process, be sure to monitor the NAS with RAIDar.

If you can’t access your data -or- if RAIDar prompts you to configure your NAS (i.e. factory default), power off the NAS before the 10 minute window expires.

Additional Information:

1. Be sure to only select disks that appear on the Hardware Compatibility List. Using non-approved disks can result in various issues ranging from disks dropping out of the array to catastrophic data loss.
2. If you are going to use HCL-approved disks that were used elsewhere (such as a computer or other device), be advised that the ReadyNAS will initialize the disk and wipe any data that exists on the disk. In order to get the NAS to initialize the disks, you may need to delete any/all partitions from the disk as the ReadyNAS may detect a valid partition signature on the disk and refuse to initialize it. Even if the disk was working fine previously, you should still take the time to test it.
   

   Delete Partition in Windows

3. 4K sector alignment vs. 512-byte (x86-based units: Ultra, NVX, Pro, 2100, 3100, 3200, 4200 & sparc-based units: Duo, X6/600, NV/NV+, 1000s/1100)
4. Increasing LCC counts – WDIDLE3 utility

The Slowest Network Link

One of the problems is that many people fail to take into account that network performance is limited to the speed of the slowest link. Even though the ReadyNAS has a gigabit interface, if your router or network switch does not support gigabit, you will only get speeds that match the network device (typically 100 Mbps for most devices).

You also need to consider the speed on the network card on the PC.  If the NIC on the PC is not gigabit or you’re  using a wireless connection, performance will also be reduced.

In order to achieve maximum performance, all device connections would need to be wired gigabit connections. This requires:

• gigabit ethernet cards in each computer
• a gigabit-capable router or switch
• category 5e or 6 ethernet cable
• optional jumbo frame support

If you’re looking to upgrade any of your network equipment, be sure to check the Hardware Compatibility List to make sure that the equipment is supported by the ReadyNAS.

Bits-per-second vs. Bytes-per-second

Network throughput is generally measured in bits-per-second (i.e. 100 Mbps, 54 Mbps, etc.) and disk I/O is typically measured in bytes-per-second (i.e. MB/s). When measuring performance, pay close attention to the big B (as in Bytes) vs. the little b (as in bits). A hard-wired connection of 100 Mbps is theoretically capable of 12.5 MB/s (12.5 MB/s x 8 bits-per-byte = 100 Mbps), however, network overhead typically reduces it to about 80% (9 MB/s on average). Same for wireless 802.11g, however, the overhead is much greater and results in about 22 Mbps max (or approximately 3 MB/s).

The only way to maximize performance is to make all links wired gigabit, which should increase disk I/O to it’s maximum.

Typical Performance Expectations

There are a number of factors that can affect performance, such as the size, type and number of files, as well as the protocol used (CIFS, NFS, AFP, FTP, HTTP, etc.).  Under ideal conditions, a typical user should be able to achieve the following performance:

1. Wireless 802.11g: 2-3 MB/s average (Vista / Windows 7 users should read this)
2. Wireless 802.11n: up to 6 or 7 MB/s average
3. Fast ethernet (wired 100 Mbps): Max 12.5 MB/s – average 9-10 MB/s
4. Gigabit ethernet (wired 1000 Mbps):
   • Click on the link for each model to get detailed performance specs & hardware used
   • ReadyNAS Duo/NV+: 25 – 40 MB/s
   • ReadyNAS NVX/2100: 70-80 MB/s
   • ReadyNAS Ultra4/Ultra6: 80-90 MB/s (depending on model)
   • ReadyNAS Pro/3100/3200: 100+ MB/s
5. 10 GbE (wired 10,000 MB/s):
   • ReadyNAS 4200: 852 MB/s (using 802.3 ad Teaming, drag & drop 20 GB file over CIFS)

ReadyNAS Device Limitations

Performance is also limited by the CPU in the ReadyNAS device itself.  The ReadyNAS devices that use the sparc-based IT3107 CPU are limited to a maximum speed of about 25-40 MB/s, where as the Intel-based models like the Pro can surpass 100 MB/s. There are also a number of tweaks that you can make in order to optimize performance on the NAS.  Be sure to read the article on optimizing performance: http://www.readynas.com/?p=310

Desktop PC Limitations

Another limitation people fail to take into consideration is the performance of the desktop client computer that they’re using.  The disk I/O of most desktop-class PCs are limited to about 60 MB/s and would never be able to reach the capabilities of a product like the ReadyNAS Pro, unless your PC was configured for some sort of RAID.  Some might wonder what the benefit of owning a ReadyNAS Pro is if you will never be able to max out the performance.  A single PC might not be able to max out the Pro, however, if you have multiple gigabit clients simultaneously reading and writing to the Pro, it will be able to keep up.  Using 802.3ad LAG, the ReadyNAS Pro has achieved 240 MB/s read speeds and 140 MB/s write speeds when being accessed with multiple Macs.

Next Steps

Okay, so you’ve checked everything and you’re still not getting the performance you expect.  What should you try next?

• Check for network errors in Frontview > Network > Interfaces > Show Errors.
• Direct-connection between PC and NAS. This will eliminate any incompatible network equipment.
• Replace ethernet cables with new or known-working CAT 5e cables.  A flaky cable on the NAS or PC can cause all sorts of grief.
• Disable or uninstall any security software.  Some security software can be overly agressive.
• Try a different protocol.  If using CIFS/SMB (Windows), try copying files using FTP to see if the performance is comparable.
• Try a different computer.  If possible, try using a different computer to eliminate the PC as the source of the trouble.
• Try a different operating system.  If your PC has multiple OSes (or you have a Live Linux CD), try booting into another OS.
• Check hard disks. Flaky hard disks can cause intermittent read/write issues that can slow the NAS to a crawl.
• Check your RAM memory. If you have upgraded your memory, you may want to re-install the original memory to eliminate the RAM as the culprit.
• Factory Default. Sometimes the OS on the NAS becomes corrupt and can be corrected by resetting the unit.  Be sure to backup any data on the unit.
• Contact support. If you’ve exhausted all of the above, you should download your log files and contact support.

http://www.youtube.com/watch?v=Jr9j-MRTxz8

Generally speaking, the ReadyNAS devices are quite easy to setup. Most units come with the drives pre-installed and configured, however, there are a few models that are available in a diskless version (notably the Pioneer Editions of the Pro and NVX, as well as some models of the Duo and NV+). Basically, you just install the disks, turn the unit on and configure the NAS.  In my experience, however, I found it best to do a few things before configuring the NAS to avoid problems down the road.

This guide makes the assumption that you will be purchasing your own disks, but even if the disks came pre-installed, I recommend taking the time to follow this procedure.

Selecting and Testing Your Disks

1. Be sure to only select disks that appear on the Hardware Compatibility List.  Using non-approved disks can result in various issues ranging from disks dropping out of the array to catastrophic data loss.
2. Before installing the disks in the array, please take the time to test the disks using the vendor tools.  Disks can get damaged during shipping and handling, as well as just be flaky straight from the factory.  You will need to install each disk into a SATA-capable PC in order to test the disks.  If any of the disks fail the vendor test then you should replace the drive before proceeding.
3. If you are going to use HCL-approved disks that were used elsewhere (such as a computer or other device), be advised that the ReadyNAS will initialize the disk and wipe any data that exists on the disk. In order to get the NAS to initialize the disks, you may need to delete any/all partitions from the disk as the ReadyNAS may detect a valid partition signature on the disk and refuse to initialize it.  Even if the disk was working fine previously, you should still take the time to test it.

Installation

Now that the disks are tested and verified to be error free, you can install them into the ReadyNAS.  The ReadyNAS should come with a disk tray for each slot, as well as 4 screws for each tray.

1. Install the disks into the trays as per the Quick Start Guide.
2. Install the latest version of RAIDar on your computer.
3. Once the disks are installed into the chassis and RAIDar is installed on your computer, perform a factory default:
   • ReadyNAS Duo, NV+, 1100 – http://www.readynas.com/forum/faq.php#How_do_I_reset_the_ReadyNAS_to_factory_default%3F
   • ReadyNAS Pro & NVX – http://home.bott.ca/webserver/?p=252
   • ReadyNAS 2100 – http://home.bott.ca/webserver/?p=180
   • ReadyNAS 3200 – http://home.bott.ca/webserver/?p=281
4. After the factory default process begins, you will have a 10-minute window to connect to the ReadyNAS using RAIDar to configure the unit for X-RAID or Flex-RAID.  Select your desired level of RAID and complete the configuration.  Please note that the process of initializing each disk and creating the volume can take a number of hours to complete.
5. After basic configuration, but before copying any data, it is recommended to upgrade to the latest version of firmware.

Configuration

The configuration process is quite easy for the creation of users and shares, however, there are some configuration issues that can cause problems if you’re not careful.  Many access problems stem from some sort of naming conflict, such as creating a share with the same name as a user account:

1. Password for the Admin Account on the ReadyNAS: You should change the admin password on the ReadyNAS from the default setting (netgear1).  The admin password should be different than any password used in Windows.
2. Hostname for the NAS: By default, the ReadyNAS will be named something like “NAS-xx-xx-xx” (where xx-xx-xx is the last 3 segments of the MAC address).  If you decide to change it, the hostname should be a unique name on the network that is not the same as any other share, username or computer on the network.
3. User Accounts on the NAS: You should create a user account for each user that requires access to the ReadyNAS.  Usernames must be unique and cannot be the same as any share or computername on the NAS or network.  When you create each user account on the NAS, a private user share will be created for that user.
4. Share Names on the NAS: Share names must be unique and cannot be the same as any username or computername on the NAS or network. For example, you should not create a share with the same name as one of the user accounts, as the NAS is configured to create a private share for each user at the time of account creation.
5. User Accounts in Windows: In order to provide seamless/transparent access from Windows to the ReadyNAS, it is recommended that your Windows Login Account be the same as your ReadyNAS User Account (same username and password).  This way, when the NAS challenges Windows for credentials, Windows will send the current username and password to the NAS for authentication and the user will have the appropriate level of access that has been granted.  It is also recommended that you not login to Windows using an account called “admin”, as this may create problems when connecting to the NAS.

There are a number of additional unofficial guides on this site that can help if you run into problems such as poor wireless performance in Vista/Windows 7 or a NAS disappearing from the network, as well as how to use the NAS for WINS or DHCP or Wake-on-LAN.

The most important thing to keep in mind is that although network attached storage devices can help protect against a failed disk, they are still not immune to other sorts of problems like multiple disk failures, accidental or intentional deletions, fire, flood or theft.  It is always advisable to make sure that you have multiple backups: http://www.readynas.com/?p=3153.

Additionally, many of our public access terminals have security software installed that prevents software from being installed or automatically rolls-back the machine to a known state after each reboot. In order to allow software to be installed/updated, the security software must be disabled.

Using the ReadyNAS as a wake-on-lan device requires a few things:

• A PC that supports wake-on-lan
• Some basic linux knowledge (the ReadyNAS is based on Debian Linux)
• SSH access to your NAS (and possibly the APT add-on for Duo, NV+ and 1100 owners)
• Installing the wakeonlan application from the Debian repositories
• Some sort of shell script to run to wake the PC

At work, I have written a few scripts that allow me to wake the PCs at my remote sites.  The basic process is as follows:

1. SSH into branch NAS:

- ssh root@192.168.129.2 for Grantham-NAS
- ssh root@192.168.131.2 for Merritt-NAS

2. At the command prompt type:

wakeonlan -f /backup/utilites/wakeonlan/grantham.wol
wakeonlan -f /backup/utilites/wakeonlan/merritt.wol

The grantham.wol and merritt.wol files are just text files that I wrote up that contain the MAC address (00:13:72:xx:xx:xx), subnet broadcast address (192.168.129.255) and port number (7) as well as some comments (#) for each PC on the network:

Epsilon:/backup/utilities/wakeonlan# cat grantham.wol

# Mac Address       IP-Address  Computer Name
# GRAN_CIRC1
00:13:72:xx:xx:xx       192.168.129.255 7
# 192.168.129.59

# GRAN_CIRC2
00:13:72:xx:xx:xx      192.168.129.255 7
# 192.168.129.62

# GRAN_CIRC3
00:13:72:xx:xx:xx   192.168.129.255 7
# 192.168.129.65

Other notes:

I also use the ReadyNAS as a DHCP server.  I have another write-up here that offers some insight on how to do it, as well as add additional options, such as static leases.  If you’re already using the ReadyNAS as a DHCP server, you can utilize some commands to get useful information for your script, such as the MAC address:

1. Viewing active leases to get the MAC address:

Grantham-NAS:~# dumpleases -f /var/lib/misc/udhcpd.leases
Mac Address       IP-Address      Expires in
00:13:72:xx:xx:xx 192.168.129.55  6 days, 20 hours, 31 minutes, 0 seconds
00:13:72:xx:xx:xx 192.168.129.53  6 days, 18 hours, 34 minutes, 16 seconds
00:13:72:xx:xx:xx 192.168.129.54  6 days, 6 hours, 10 minutes, 39 seconds
00:13:72:xx:xx:xx 192.168.129.67  6 days, 18 hours, 51 minutes, 49 seconds
00:13:72:xx:xx:xx 192.168.129.56  6 days, 18 hours, 41 minutes, 46 seconds
00:13:72:xx:xx:xx 192.168.129.71  6 days, 18 hours, 53 minutes, 48 seconds
00:13:72:xx:xx:xx 192.168.129.61  6 days, 23 hours, 52 minutes, 47 seconds
00:13:72:xx:xx:xx 192.168.129.73  6 days, 23 hours, 45 minutes, 6 seconds
00:13:72:xx:xx:xx 192.168.129.58  6 days, 23 hours, 36 minutes, 4 seconds
00:13:72:xx:xx:xx 192.168.129.69  6 days, 23 hours, 13 minutes, 45 seconds
00:13:72:xx:xx:xx 192.168.129.70  6 days, 23 hours, 32 minutes, 28 seconds
00:13:72:xx:xx:xx 192.168.129.50  6 days, 18 hours, 40 minutes, 26 seconds
00:13:72:xx:xx:xx 192.168.129.51  6 days, 18 hours, 40 minutes, 3 seconds
00:13:72:xx:xx:xx 192.168.129.66  6 days, 6 hours, 31 seconds
00:13:72:xx:xx:xx 192.168.129.63  6 days, 18 hours, 41 minutes, 17 seconds
00:13:72:xx:xx:xx 192.168.129.52  6 days, 1 hours, 34 minutes, 0 seconds
00:13:72:xx:xx:xx 192.168.129.57  6 days, 20 hours, 9 minutes, 46 seconds
00:13:72:xx:xx:xx 192.168.129.62  6 days, 23 hours, 4 minutes, 5 seconds
00:00:00:00:00:00 192.168.129.59  expired
00:00:00:00:00:00 192.168.129.60  expired
00:13:72:xx:xx:xx 192.168.129.64  expired
00:13:72:xx:xx:xx 192.168.129.65  6 days, 23 hours, 24 minutes, 4 seconds

2. I also temporarily updated the udhcpd.conf file with auto_time 60 to update the lease file to every minute while testing.

Grantham-NAS:~# cat /etc/udhcpd.conf
# This file is auto-generated.  Do not modify!
auto_time       60
interface       eth0
option wins     192.168.128.25
option dns      192.168.128.25
option dns      192.168.128.35
option domain   mydomain.ca
option lease    604800
option subnet   255.255.255.0
option router   192.168.129.1
start   192.168.129.50
end     192.168.129.199

3. Getting MAC addresses from within Windows:

From command line type “getmac /s computername”

C:\>getmac /S gran_circ1

Physical Address    Transport Name
=================== ==========================================================
00:13:72:xx:xx:xx   \Device\Tcpip_{0E96113B-5F1C-4200-8F81-4D0E83E39AE1}

At this point, the computers will begin to boot and I use open-source tools such as VNC and iTALC to remotely control the computers and disable any security software.

• 16 KB block size for improved performance (vs. 4 KB blocks on v.3 firmware)*
• online expansion*
• support for volumes up to 64 TB (vs. 5 TB on v.3)
• support for larger drives
• and a bunch of new services and features (timemachine, bittorrent, ReadyDLNA, Vault, Remote, add-ons galore, etc.)

It is quite easy to upgrade from v.3 to v.4 — just install the latest version of firmware and you’re done (after backing up your data, just to be safe).  The downside is that upgrading from v.3 to v.4 does not change the block size or provide online expansion which can result in a decrease in performance.  The only way to get all of the benefits of v.4 is to perform a factory default, which entails the following:

• backup and verify data
• upgrade to v.4
• backup all configuration settings
• perform a factory default of the NAS
• restore settings
• restore data

Have a look at this thread for some ideas on how to accomplish a firmware and disk upgrade:

http://www.readynas.com/forum/viewtopic.php?f=65&t=32732&p=180019#p180019

* these options are only available after performing a factory default on v.4

Note: The block size only affects the sparc-based units.  If you have a Pro, then block size is (and always has been) 4 K. The current version of RAIDiator supports up to 64 TB volumes on both the x86-based units and the sparc-based units.

FWIW, on all Intel x86-based units (Pro, NVX, 2100 and 3200), the block size is 4 k.

Sparc-based units (600, X6, NV, NV+, 1000s, 1100) running version 3.x of RAIDiator the block size is 4 k.  New sparc-based units (Duo, NV+, 1100) running version 4.x of RAIDiator and older models that have been factory defaulted with version 4 will have a block size of 16 k.

Sparc-based units that have been upgraded to version 4 from version 3 without a factory default will still have 4 k blocks.

To verify your block size on sparc-based units, download the system_log.zip from FrontView > Status > Logs > Download All Logs.
Open the zip file, then view the volume.log.

Look for a line that says Block size.

RAIDiator 3 Upgrade:

Block size:               4096

RAIDiator 4 Factory:

Block size:               16384

LED blink behavior for 2 disk systems is three quick blinks of all disk LEDs and the backup LED, followed by a 1 second delay, followed by a number of slow blinks.  The number of slow blinks will be the error code.

Current error codes:
1 – Vendor mismatch
2 – No disks detected
3 – Bad contents on root partition of disks
4 – Flash error
5 – Unsupported RAID configuration

Quoting Skywalker:

There is no “proprietary” filesystem running on any ReadyNAS.  It is “straight”, ordinary, unmodified EXT3.  You can even run all the e2fsprogs (debugfs, e2fsck, etc) on a 16KB ReadyNAS filesystem on a standard Linux PC with no modifications.  Mounting a 16KB ReadyNAS filesystem on an x86 PC requires about the same amount of work as mounting an NTFS filesystem read/write on Linux — using a FUSE driver to mount it.

The ext2fuse driver won’t come on any LiveCDs,  so you’d need a full distro running with the necessary build tools installed.

Ubuntu, Fedora, Debian… it doesn’t really matter.  Once you get those installed, you need to get the ext2fuse source code from sourceforge.net.

1. Using Debian, you can install the build tools by issuing the following command as root:

apt-get install build-essential

2. Download the ext2fuse package and extract it:

tar xvzf ext2fuse-src-0.8.1.tar.gz

3. Change to the directory of the extracted program
4. Run the following command:

./configure

5. Before compiling, open src/Makefile in an editor.
6. Remove the lines that look like this:

ext2fuse_DEPENDENCIES = ../lib/et/libcom_err.a \
../lib/ext2fs/libext2fs.a

7. Change this line:

ext2fuse_LDADD = ../lib/et/libcom_err.a ../lib/ext2fs/libext2fs.a

8. To this:

ext2fuse_LDADD = -lcom_err -lext2fs

9. Then build the executable by running make:

make

10. Next, install the package by running make install:

make install

11. After compiling and installing the package, you’ll need to load the fuse module:

modprobe fuse

12. Next, run the following commands to recover the volume group information and then activate it:

vgscan
vgchange -ay c

13. Create a directory to mount the array:

mkdir /mnt/lvm

14. Mount the array:

ext2fuse /dev/c/c /mnt/lvm

15. If successful, you should see:

/dev/c/c is to be mounted at /mnt/lvm
fuse-ext2 initialized for device: /dev/c/c
block size is 16384

16. At this point, your ReadyNAS data volume should be mounted on /mnt/lvm and you should be able to access your data.

I haven’t tried this myself (I gave my NV+ to my dad), but welcome any feedback on making it better.

Update (May 22, 2010) – Have a look at this for additional information:

http://greyproc.blogspot.com/2010/04/readynas-600-raid-recovery-with-ubuntu.html

Primary ReadyNAS :
ReadyNAS Pro Business Edition : 1024 MB RAM : 6 x 500 GB Seagate 320 NS : Raidiator 4.2.5 : SSH Enabled : WordPress (powers this site)

Backup ReadyNAS:
ReadyNAS NVX Business Edition : 1024 MB RAM : 2 x 500 GB Seagate 320 NS : 2 x 1 TB WDC WD1002FBYS-02A6B0 : Raidiator 4.2.13 : SSH Enabled

Router: D-Link DGL-4500 Gigabit Wireless a/b/g/n
Switch: Netgear GS108T : 8 Port Gigabit Managed Switch

Desktop: Dell OptiPlex GX520 : P4-3.0 GHz : 1 GB RAM : BroadCom BCM5751 : Ubuntu 10.04
Laptop: Dell Inspiron 6400 : Core Duo T2400 : 2 GB RAM : ATI x1400 : Multi-boot : Win 7 Pro : XP Pro : XP MCE : Windows 2003 Server : Vista Ultimate : Ubuntu Linux
Others: iMac DV SE : OSX Panther :: HP iPAQ 4350 : PocketPC 2003 :: iPod Touch 32 GB

HDTV: LG 42LG70 42″ 1080p 120Hz Flat-Panel LCD HDTV
Media Player: Netgear EVA-8000 Digital Entertainer HD

UPS: TrippLite OMNISMART 1000LCD UPS

Work

Primary ReadyNAS:
ReadyNAS 2100 RNRX-4410 : 1024 MB RAM : 4 x 1 TB Seagate : Raidiator 4.2.5 : SSH Enabled

Remote Backup ReadyNAS #1:
ReadyNAS 2100 RNRX-4410 : 1024 MB RAM : 4 x 1 TB Western Digital : Raidiator 4.2.5 : SSH Enabled

Remote Backup ReadyNAS #2:
ReadyNAS 2100 RNRX-4410 : 1024 MB RAM : 4 x 1 TB Western Digital : Raidiator 4.2.5 : SSH Enabled

At work, I backup just over a dozen servers (primarily Windows 2000/2003 and RedHat Linux) using rsync to push the data to the Primary ReadyNAS 2100.  Every night, the primary ReadyNAS is scheduled to take a snapshot to freeze the data.  Shortly thereafter, the other 2 remote ReadyNAS devices then pull a backup of the primary NAS over rsync.

The ReadyNAS 2100s are integrated into Windows AD and the 2 backup units are also configured to provide DHCP services to the remote locations.

The big story today is about Microsoft subsidiary Danger losing all T-Mobile Sidekick customer data from their servers. Danger is the company noted for the T-Mobile Sidekick, the revolution in cloud mobile, and most memorably, almost everybody living in 90210 having to get new phone numbers because of Paris Hilton. Valued T-Mobile Sidekick customers received a notice today from the company updating them on the “data disruption” problem. The good news is that data is no longer being disrupted. The bad news is that there is no data left to be disrupted.

… as Sidekick users found out today, and ironically, as 7,500 users of online backup provider Carbonite found out after the company lost their backups (Carbonite can take some comfort in that they now rank very well for ‘data loss’ in search engines because of the incident. What do they say about bad publicity?). In the Danger case, it appears from initial speculation that the data was lost because they attempted to upgrade a storage array without backing it up first. Here is a case of smart and rational people who do this for a living at one of the best companies in the world, and they didn’t even bother making a backup…

When it happens to you, will your data be safe?

There are a variety of reasons that people buy NAS devices, such as the ReadyNAS.  Many families have multiple computers and are looking for a way to provide easy, centralized access to their data.  Other folks have suffered lost data due to hard drive failures and want an easy way to keep their data backed up.

Many NAS devices incorporate RAID (redundant array of inexpensive/independent disks) into their design.  RAID is designed to allow users to recover from a single disk failure by striping the data across multiple disks.  In the event that a single disk fails, the RAID array can rebuild the missing data by performing a parity calculation and the user can still have un-interrupted access to their data.  When they replace the failed disk with a new one, the RAID array will automatically rebuild itself and continue to offer a protected data volume.

Misconceptions

Misconception 1:

Many people think that storing their data on a RAID-protected NAS is an adequate backup.  In fact, it is not.  Many people use their NAS as a central server and store the only copy of their data right on the NAS.  Things like fire, flood, theft, accidental or intentional deletions, user error or multiple hard drive failures can all lead to catastrophic data loss. If the only copy of your data is on the NAS, it is not a backup!

Misconception 2:

Many people use client-based software to backup the data on their PC to the NAS.  This is a good thing because now they can maintain multiple copies of their data.  The most current copy is on their PC and, if they’re using versioning software, they can maintain a number of versions of their files on the NAS.

NTI Shadow with Versioning

The only problem with this method is that all of your data is stored at the same location.  A fire, flood, theft or other natural disaster will take your PC, NAS and whatever data resides on them.  If all of your data is stored at the same location, it is not a secure backup.

Misconception 3:

The third misconception is that you can use one of the RAID drives as a backup by pulling it out and storing it off-site (kind of like a backup tape).  This is definitely not a recommended method of maintaining a backup or your data. corndog from SmallNetBuilder.com sums it up perfectly:

…using RAID-1 hot-pull as a method of backup is insane. I’ve posted previously about this. There are a few reasons why you should never ever do this. First – RAID is designed to be a “first level” method of data protection. That means it is supposed to protect the live data on your system. You still need to do backups, but RAID is supposed to protect against most problems, so you only need to go to your backups in the case of a real catastrophe. RAID protects the live running data – there is no guarantee about disks that are pulled. By design, pulled disks are crap. RAID makes no effort to ensure that the data on those disks is meaningful in any way.

Second, when you pull a disk, there are multiple levels of commit that you have just broken. What guarantee do you have that the NAS wasn’t right in the middle of saving a file right when you pulled the disk? Just because you made sure all your clients were not writing anything doesn’t mean the NAS wasn’t clearing a cache or something – you simply do not know. When you do not know, you should EXPECT corruption.

Third, your file system is not cleanly dismounted. Think about it. Say for instance you want to keep a safe copy of your Windows data somewhere, and you are planning to pull the disk that is in your Windows machine. Wouldn’t you make sure you did a clean shutdown and the drive is dismounted properly before pulling out the disk? You wouldn’t just pull the power plug and kill the machine, then pull the disk right? That’s exactly what you are doing to the contents of the disk when you hot-pull a RAID1 drive.

Fourth, think of what RAID1 is all about – protection against drive failure. Now, think about how hard your drives are working during a typical day. Failure is most likely when the drive is most busy. If each day you hot-pull that disk, not only have you just walked away from redundancy until your new replacement disk is all mirrored again, you’ve also just put your drives to work – HEAVY work – remirroring. Guaranteed your disks are working waaaay harder during the re-mirror process than at any other time during the day. So just when they are most likely to fail (when they are working the hardest) you have also intentionally removed the RAID protection at this same time too – that’s just plain CRAZY!

Fifth, when you hot-pull a drive, it is powering off right in the middle of running – it has not been safetied, and it is MOVING (i.e. sliding out of your NAS). No matter how carefully and how smoothly you pull that drive, compared to the fine tolerances of space between the heads and the platter, you are giving it a massively rocky ride. Are you sure you aren’t causing the heads to crash onto the platter while you are doing this? Where have you ever read that it is safe for the disk to hot-pull a good drive? Hot-pull is meant for drives that are already dead. Plain and simple.

Bottom line – when RAID was designed, it was never intended as a method of hot-pull-backup. For the reasons outlined above, it should never be done. Think about it this way: Backups are important. They protect your precious data. Backup solutions (such as backup software or synchronizing tools like rsync) are designed specifically for doing exactly this job – backing up your important stuff. If your data is really so important, why would you rely on a method that was never intended to even work, to protect it, when there are other, much better methods that were actually designed to do this job properly? For example, why would you try to put new tires on your car while driving down the road, when there are perfectly good garages you could pull into, and they can do it right?

I regularly see people on many different forums complaining about trying to use RAID1-hot-pull as a backup method and they always seem to have problems – makes me shake my head in disbelief every time. They mustn’t really care about their data, or they would use a proper backup method.

Proper Backups

A proper backup consists of multiple copies of data stored in different locations.  If the only backup of your NAS sits on the shelf beside your NAS and there’s a fire or theft, chances are your backup is going to be lost as well.  At my place of employment, I backup over a dozen servers to a primary NAS at regular intervals throughout the day.  Every night, I sync the data on the primary NAS to 2 separate NASes located at our remote locations.  In the event of a major catastrophe, I have multiple backups maintained at multiple locations.

I’m not suggesting that this is an affordable or practical solution for the home user or SOHO-user, however, a couple of large capacity external USB drives can be utilized to provide rotating backups that are kept off-site.  There are also services provided specifically for the ReadyNAS like ReadyNAS Vault that offer a secure, remote facility to store your data.

There are data restoration companies like DriveSavers who can recover data from failed or damaged drives for the price of a new car — give or take a few thousand dollars.  They can’t recover data from a stolen device, though.

For a few hundred dollars, you can make sure that your data is safe.  Just be sure to regularly backup your data, verify that your backup procedure works and that you can restore your data, and then keep the backup in a different location.