Document Outline

• Sun StorEdge™ 3900 and 6900 Series 2.0 Troubleshooting Guide
  • Contents
  • 1. Introduction1
  • 2. General Troubleshooting Procedures3
  • 3. Troubleshooting Tools23
  • 4. Troubleshooting Ethernet Hubs35
  • 5. Troubleshooting the Fibre Channel (FC) Links37
  • 6. Troubleshooting Host Devices67
  • 7. Troubleshooting Switches73
  • 8. Troubleshooting the Sun StorEdge T3+ Array Devices87
  • 9. Troubleshooting Virtualization Engine Devices107
  • 10. Troubleshooting Using Microsoft Windows 2000137
  • 11. Example of Fault Isolation147
  • A. Virtualization Engine References155
  • B. Configuration Utility Error Messages163
    • List of Figures
    • List of Tables
    • Preface
  • How This Book Is Organized
  • Using UNIX Commands
  • Typographic Conventions
  • Shell Prompts
  • Related Documentation
  • Accessing Sun Documentation Online
  • Sun Welcomes Your Comments
    • 1
    • Introduction
      • TABLE11 Sun StorEdge 3900 and 6900 Series Configurations
    • Predictive Failure Analysis (PFA) Capabilities
      • 2
      • General Troubleshooting Procedures
    • High-Level Troubleshooting Tasks
      • 1. Discover the error by checking one or more of the following messages or files:
      • 2. Determine the extent of the problem by using one or more of the following methods:
      • 3. Check the status of a Sun StorEdge T3+ array by using one or more of the following methods:
      • 4. Check the status of the Sun StorEdge network FC switch-8 and switch-16 switches using the foll...
      • 5. Check the status of the virtualization engine using one or more of the following methods:
      • 6. Quiesce the I/O along the path to be tested using one of the following methods:
      • 7. Test and isolate field-replaceable units (FRUs) using the following tools:
      • 8. Verify the fix using the following tools:
      • 9. Return the path to service with one of the following methods:
    • Host-Side Troubleshooting
    • Storage Service Processor-Side Troubleshooting
    • Verifying the Configuration Settings
      • To Verify Configuration Settings
        • 1. Run one of the following scripts:
          • CODEEXAMPLE21 checkdefaultconfig(1M) Output
        • 2. If anything is marked FAIL, check the /var/adm/log/SEcfglog file for the details of the failure.
        • 3. Fix the FAIL condition, and then verify the settings again.
      • Clearing the Lock File
        • To Clear the Lock File
          • 1. Type the following command:
          • 2. Monitor the /var/adm/log/SEcfglog file to see when the savevemap(1M) process successfully exits.
            • CODEEXAMPLE22 savevemap(1M) Output
    • Sun StorEdge 6900 Series Multipathing Example
      • FIGURE21 Sun StorEdge 6900 Series Logical View
      • FIGURE22 Primary Data Paths to the Alternate Master
      • FIGURE23 Primary Data Paths to the Master Sun StorEdge T3+ Array
      • FIGURE24 Path Failure—Before the Second Tier of Switches
      • FIGURE25 Path Failure—I/O Routed Through Both HBAs
    • Multipathing Options in the Sun StorEdge 6900 Series
      • Manually Halting the I/O
        • To Quiesce the I/O
          • 1. Determine the path you want to disable.
          • 2. Type:
        • To Unconfigure the c2 Path
          • 1. Type:
          • 2. Using the Storage Automated Diagnostic Environment GUI Topology, determine which virtualizatio...
          • 3. Use the worldwide name (WWN) of the virtualization engine that is in the unconfigure command, ...
          • 4. Verify that the I/O has halted.
      • Suspending the I/O
        • To Put the c2 Path Back into Production
          • 1. Type:
          • 2. Verify that I/O has resumed on all paths.
      • To View the Dynamic Multi-Pathing (DMP) Properties
        • 1. Type:
        • 2. Use the luxadm(1M) command to display further information about the underlying LUN.
        • To Put the DMP-Enabled Paths Back into Production
          • 1. Type:
          • 2. Verify that the path has been reenabled by typing:
          • 3
          • Troubleshooting Tools
    • Storage Automated Diagnostic Environment 2.2
      • Example Topology
        • FIGURE31 Storage Automated Diagnostic Environment Example Topology
      • Generating Component-Specific Event Grids
        • To Customize an Event Report
          • 1. Choose the Event Grid link on the the Storage Automated Diagnostic Environment Help menu.
          • 2. Select the criteria from the Storage Automated Diagnostic Environment event grid, like the one...
            • TABLE31 Event Grid Sorting Criteria
    • Microsoft Windows 2000 System Errors
      • FIGURE32 Microsoft Windows 2000 Event Properties System Log
    • Command Line Test Examples
      • qlctest(1M)
        • CODEEXAMPLE31 qlctest(1M)
      • switchtest(1M)
        • CODEEXAMPLE32 switchtest(1M)
    • Monitoring Sun StorEdge T3 and T3+ Arrays Using the Explorer Data Collection Utility
      • To Install the Explorer Data Collection Utility on the Storage Service Processor
        • 1. Type:
        • 2. When you are prompted for site-specific information during the installation process, you can o...
        • 3. Before running the Explorer Data Collection Utility, make sure that the switch and Sun StorEdg...
        • Example
          • CODEEXAMPLE33 Editing Switch Information Using vi
          • 4. Type Sun StorEdge T3+ array information in the /opt/SUNWexplo/etc/ t3input.txt file.
          • 5. Type the password for your specific site.
            • CODEEXAMPLE34 Editing Sun StorEdge T3+ Array Information Using vi
    • Monitoring Host Bus Adapters (HBAs) Using QLogic SANblade Manager
      • FIGURE33 Qlogic SANblade Manager HBA Driver and Firmware Versions
      • FIGURE34 QLogic SANblade Manager Diagnostics
      • 4
      • Troubleshooting Ethernet Hubs
        • 5
      • Troubleshooting the Fibre Channel (FC) Links
    • FC Links
      • TABLE51 FC Links
      • FC Link Diagrams
        • FIGURE51 Sun StorEdge 3900 Series FC Link Diagram
        • TABLE52 Ax to Bx FC Links.
          • FIGURE52 Sun StorEdge 6900 Series FC Link Diagram
    • Troubleshooting the A1 or B1 FC Link
      • FIGURE53 Data Host Notification of Intermittent Problems
      • FIGURE54 Data Host Notification of Severe Link Error
      • FIGURE55 Storage Service Processor Notification
      • Verifying the Data Host
        • CODEEXAMPLE51 luxadm(1M) Display
        • CODEEXAMPLE52 cfgadm -al Display
      • FRU Tests Available for the A1 or B1 FC Link Segment
        • CODEEXAMPLE53 switchtest(1M) Called With Options
      • To Isolate the A1 or B1 FC Link
        • 1. Quiesce the I/O on the A1 or B1 FC link path.
        • 2. Run switchtest(1M) or qlctest(1M) to test the entire link.
        • 3. Break the connection by uncabling the link.
        • 4. Insert a loopback connector into the switch port.
        • 5. Rerun switchtest.
          • a. If switchtest fails, replace the gigabit interface converter (GBIC) and rerun switchtest.
          • b. If switchtest fails again, replace the switch.
        • 6. Insert a loopback connector into the HBA.
        • 7. Run qlctest.
          • a. If the qlctest test fails, replace the HBA.
          • b. If the qlctest test passes, replace the cable.
        • 8. Recable the entire link.
        • 9. Run switchtest or qlctest to validate the fix.
        • 10. Put the path back into production.
    • Troubleshooting the A2 or B2 FC Link
      • FIGURE56 A2 or B2 FC Link Host-Side Event
      • FIGURE57 A2 or B2 FC Link Storage Service Processor-Side Event
      • Verifying the Data Host
        • CODEEXAMPLE54 cfgadm -al
      • Verifying the A2 or B2 FC Link
      • FRU Tests Available for the A2 or B2 FC Link Segment
      • To Isolate the A2 or B2 FC Link
        • 1. Quiesce the I/O on the A2 or B2 FC link path.
        • 2. Break the connection by uncabling the link.
        • 3. Insert the loopback connector in to the switch port.
        • 4. Run switchtest:
          • a. If the test fails, replace the GBIC and rerun switchtest.
          • b. If the test fails again, replace the switch.
        • 5. If the switch and the GBIC show no errors, replace the remaining components in the following o...
          • a. Replace the virtualization engine-side GBIC, recable the link, and monitor the link for errors.
          • b. Replace the cable, recable the link, and monitor the link for errors.
          • c. Replace the virtualization engine, restore the virtualization engine settings, recable the lin...
        • 6. Return the path to production.
    • Troubleshooting the A3 or B3 FC Link
      • FIGURE58 A3 or B3 FC Link Host-Side Event
      • FIGURE59 A3 or B3 FC Link Storage Service Processor-Side Event
      • FIGURE510 A3 or B3 FC Link Storage Service Processor-Side Event
      • Verifying the Data Host
        • CODEEXAMPLE55 Devices in the “Connected” State
        • CODEEXAMPLE56 DMP Error Message
      • Verifying the Storage Service Processor-Side
      • FRU Tests Available for the A3 or B3 FC Link Segment
      • To Isolate the A3 or B3 FC Link
        • 1. Quiesce the I/O on the A3 or B3 FC link path (refer to “Quiescing the I/O on the A3 or B3 Link...
        • 2. Break the connection by uncabling the link.
        • 3. Insert the loopback connector in to the switch port.
        • 4. Run switchtest:
          • a. If the test fails, replace the GBIC and rerun switchtest.
          • b. If the test fails again, replace the switch.
        • 5. If the switch or the GBIC shows no errors, replace the remaining components in the following o...
          • a. Replace the virtualization engine-side GBIC, recable the link, and monitor the link for errors.
          • b. Replace the cable, recable the link, and monitor the link for errors.
          • c. Replace the virtualization engine, restore the virtualization engine settings, recable the lin...
        • 6. Return the path to production.
        • Quiescing the I/O on the A3 or B3 Link
          • 1. Determine the path you want to disable.
          • 2. Disable the path by typing the following:
          • 3. Verify that the path is disabled:
        • Suspending the I/O on the A3 to B3 Link
    • Troubleshooting the A4 or B4 FC Link
      • FIGURE511 A4 or B4 FC Link Data-Host Notification
      • FIGURE512 Storage Service Processor-Side Notification
      • Verifying the Data Host
        • Sun StorEdge 3900 Series
        • Sun StorEdge 6900 Series
          • CODEEXAMPLE57 Failed Path Marked Offline
          • CODEEXAMPLE58 Failed Path Marked Unusable
      • FRU Tests Available for the A4 or B4 FC Link Segment
      • To Isolate the A4 or B4 FC Link
        • 1. Quiesce the I/O on the A4 or B4 FC link path.
        • 2. Run linktest(1M) from the Storage Automated Diagnostic Environment GUI to isolate suspected fa...
        • 1. Quiesce the I/O on the A4 or B4 FC link path.
        • 2. Run switchtest(1M) to test the entire link (re-create the problem).
        • 3. Break the connection by uncabling the link.
        • 4. Insert the loopback connector in to the switch port.
        • 5. Rerun switchtest.
          • a. If switchtest fails, replace the GBIC and rerun switchtest.
          • b. If the test fails again, replace the switch.
        • 6. If switchtest passes, assume that the suspect components are the cable and the Sun StorEdge T3...
          • a. Replace the cable.
          • b. Rerun switchtest.
        • 7. If the test fails again, replace the Sun StorEdge T3+ array controller.
        • 8. Return the path to production.
        • 9. Return the Sun StorEdge T3+ array LUNs to the correct controllers, if a failover occurred. (De...
        • 6
        • Troubleshooting Host Devices
    • Using the Host Event Grid
      • To Access the Host Event Grid
        • 1. From the Storage Automated Diagnostic Environment Help menu, choose the Event Grid link.
        • 2. FIGURE61 shows the Host Event Grid, from which you can select related criteria for the event...
          • FIGURE61 Sample Host Event Grid
          • TABLE61 Storage Automated Diagnostic Environment Event Grid for the Host
      • To Replace the Master Host
        • 1. Install the SUNWstade package on a new master host.
        • 2. Run /opt/SUNWstade/bin/ras_install on the new master host.
        • 3. Configure the host as the master host.
        • 4. Connect to the master server’s GUI at http://<servername>:7654
        • 5. Choose System Utilities -> Recover Config.
          • a. In the Recover Config window, enter the IP address of any alternate master or slave monitoring...
          • b. Make sure the checkboxes for Recover config and Reset slave to this master are checked.
          • c. Click Recover.
        • 6. Choose Maintenance -> General Maintenance.
          • a. Ensure that all host and device settings are recovered correctly.
          • b. Refer to Chapter 3 of the Storage Automated Diagnostic Environment User’s Guide for detailed i...
        • 7. Choose Maintenance -> General Maintenance -> Start/Stop Agent to start the agent on the master...
      • To Replace the Alternate Master or Slave Monitoring Host
        • 1. Choose Maintenance -> General Maintenance -> Maintain Hosts.
        • 2. In the Maintain Hosts window, from the Existing Hosts list, select the host to be replaced and...
        • 3. Install the new host.
        • 4. Install the SUNWstade package on the new host.
        • 5. Run /opt/SUNWstade/bin/ras_install.
        • 6. Configure the host as a slave.
        • 7. Choose Maintenance -> General Maintenance -> Maintain Hosts.
        • 8. In the Maintain Hosts window, select the new host.
        • 9. Configure the options as needed.
        • 10. Choose Maintenance -> Topology Maintenance -> Topology Snapshot.
          • a. In the Topology Snapshot window, select the new host.
          • b. Click the Create and Retrieve Selected Topologies button.
          • c. Click the Merge and Push Master Topology button.
        • 7
        • Troubleshooting Switches
    • About the Switches
      • Zone Modifications
        • 1. Offline the T ports (interswitch links).
        • 2. Offline the virtualization engine ports.
        • 3. Modify the zone on one switch while the other switch continues to run.
        • 4. Online the T ports (interswitch links).
        • 5. Allow the zone database to merge.
        • 6. Online the virtualization engine ports.
      • Switchless Configurations
      • Diagnosing and Troubleshooting Switch Hardware Problems
        • 1. To compare the current configuration to the default configuration, type:
        • 2. To compare the current switch configuration to the most recently saved map file, type:
        • 3. To display the current switch configuration, type:
        • 4. To restore the configuration from the saved map file back to the default switch configuration,...
    • Using the Switch Event Grid
      • To Use the Switch Event Grid
        • 1. From the Storage Automated Diagnostic Environment Help menu, select the Event Grid link.
        • 2. FIGURE71 shows the Switch Event Grid, from which you can select related criteria for the eve...
          • FIGURE71 Switch Event Grid
          • TABLE71 Storage Automated Diagnostic Environment Event Grid for 1 Gbit Switches
            • 1. Check the Topology GUI for any link errors.
            • 2. Quiesce I/O on the link
            • 3. Run linktest on the link to isolate the failing FRU.
            • 1. Check to see if the switch has been reset.
            • 2. Check the power going to the switch.
            • 1. Check Ethernet connectivity to the switch.
            • 2. Verify that the switch is booted correctly with no POST errors.
            • 3. Verify that the switch Test Mode is set for normal operations.
            • 4. Verify the TCP/IP settings on switch by way of Forced PROM Mode access.
            • 5. Replace switch, if needed.
            • 1. Verify cables, GBICs, and connections along the FC path.
            • 2. Check the Storage Automated Diagnostic Environment SAN Topology GUI to identify failing segmen...
            • 3. Verify the correct FC switch configuration.
          • TABLE72 Storage Automated Diagnostic Environment Event Grid for 2 GBit Switches
            • 1. Check to see if the switch has been reset.
            • 2. Check the power going to the switch.
            • 1. Check Ethernet connectivity to the switch.
            • 2. Verify that the switch is booted correctly with no POST errors.
            • 3. Verify that the switch Test Mode is set for normal operations.
            • 4. Verify the TCP/IP settings on switch by way of Forced PROM Mode access.
            • 5. Replace switch, if needed.
            • 1. Verify cables, GBICs, and connections along the FC path.
            • 2. Check the Storage Automated Diagnostic Environment SAN Topology GUI to identify failing segmen...
            • 3. Verify the correct FC switch configuration.
    • setupswitch Exit Values
      • 8
      • Troubleshooting the Sun StorEdge T3+ Array Devices
    • Troubleshooting the T1 or T2 Data Path
      • Notification Events
        • FIGURE81 Storage Service Processor Event
        • FIGURE82 Virtualization Engine Alert
        • To Verify the Storage Service Processor
          • 1. Run the Sun StorEdge T3+ array port listmap command to see the failover event.
          • 2. Compare the virtualization engine configuration to a saved configuration by running runsecfg(1M).
          • 3. Choose Verify Virtualization Engine Map.
            • FIGURE83 Manage Configuration Files Menu
        • FRU Tests Available for the T1 or T2 Data Path FRU
          • FIGURE84 Example Link Test Text Output from the Storage Automated Diagnostic Environment
        • To Isolate the T1 or T2 Data Path
          • 1. Run linktest from the Storage Automated Diagnostic Environment for a guided isolation procedure.
          • 2. After replacing the failed FRU, run failbackt3path , if needed.
    • Sun StorEdge T3+ Array Event Grid
      • To Use the Sun StorEdge T3+ Array Event Grid
        • 1. From the Storage Automated Diagnostic Environment Help menu, click the Event Grid link.
        • 2. Select the criteria from the Storage Automated Diagnostic Environment event grid, like the one...
          • FIGURE85 Sun StorEdge T3+ Array Event Grid
          • TABLE81 Storage Automated Diagnostic Environment Event Grid for the Sun StorEdge T3+ Array
            • 1. Open a Telnet session to the affected Sun StorEdge T3+ array.
            • 2. Verify disk state in fru stat, fru list, and vol stat.
            • 1. Open a Telnet session to the affected Sun StorEdge T3+ array.
            • 2. Verify the loopcard state with fru stat.
            • 3. Verify the matching firmware with the other loopcard.
            • 4. Reenable the loopcard if possible (enable u (encid)|[1|2] ).
            • 5. Replace the loopcard if necessary.
            • 6. Reenable the disk if possible
            • 7. Replace the disk, if necessary.
            • 1. Open a Telnet session to the affected Sun StorEdge T3+ array.
            • 2. Run refresh -s to verify the battery state.
            • 3. Replace the battery, if necessary.
            • 1. Open a Telnet session to the affected Sun StorEdge T3+ array.
            • 2. Verify the fan state with fru stat.
            • 3. Replace the power cooling unit, if necessary.
            • 1. Open a Telnet session to the affected Sun StorEdge T3+ array.
            • 2. Verify power cooling unit state in fru stat.
            • 3. Replace power cooling unit, if necessary.
            • 1. Open a Telnet session to the affected Sun StorEdge T3+ array.
            • 2. Verify that the power cooling unit state is in fru stat
            • 3. Replace the power cooling unit, if necessary.
            • 1. Verify luxadm with the command line (luxadm probe, luxadm display)
            • 2. Verify cables, GBICs, and connections along the data path.
            • 3. Check the Storage Automated Diagnostic Environment SAN Topology GUI to identify the failing se...
            • 4. Verify the correct FC switch configuration, if applicable.
            • 1. Check the Ethernet connectivity to the affected Sun StorEdge T3+ array.
            • 2. Verify that the Sun StorEdge T3+ array is booted correctly.
            • 3. Verify the correct TCP/IP settings on the Sun StorEdge T3+ array .
            • 4. Increase the http timeout.
            • 5. Ping timeout in Utilities
            • 1. Open a Telnet session to the affected Sun StorEdge T3+ array.
            • 2. Verify the controller state with fru_stat and sys_stat.
            • 3. Re-enable the controller if possible (enable u)
            • 4. Run logger - dmprstlog from a serial port session on the affected controller. The output from ...
            • 5. Replace the controller as indicated by the NVRAM failure code.
            • 1. Open a Telnet session to the affected Sun StorEdge T3+ array.
            • 2. Verify the disk state with vol_stat, fru_stat, and fru_list.
            • 3. Replace the disk if necessary.
            • 1. Open a Telnet session to the affected Sun StorEdge T3+ array.
            • 2. Verify loopcard state with fru stat.
            • 3. Verify matching firmware with other loopcard.
            • 4. Reenable the loopcard, if possible with: (enable u(encid)|[1|2|])
            • 5. Replace the loopcard, if necessary.
            • 1. Open a Telnet session to the affected Sun StorEdge T3+ array.
            • 2. Verify the status of the LUNs with vol_mode or vol_stat.
            • 1. Check the power supply and cables.
            • 2. Replace PCU, if necessary.
        • 9
        • Troubleshooting Virtualization Engine Devices
    • About the Virtualization Engine
    • Virtualization Engine Diagnostics
      • Service Request Numbers (SRNs)
      • Service and Diagnostic Codes
      • Retrieving Service Information
        • CLI Interface
      • Error Log Analysis Commands
        • To Display the Log Files and Retrieve SRNs
          • Example
        • To Clear the Log
    • Virtualization Engine LEDs
      • TABLE91 Virtualization Engine LEDs
      • Power LED Codes
        • FIGURE91 Virtualization Engine Front Panel LEDs
      • Interpreting LED Service and Diagnostic Codes
        • TABLE92 LED Diagnostic Codes
      • Back Panel Features
        • FIGURE92 Virtualization Engine Back Panel
      • Ethernet Port LEDs
        • TABLE93 Speed, Activity, and Validity of the Link
    • FC Link Error Status Report
      • TABLE94 Virtualization Engine Statistical Data
      • To Check the FC Link Error Status Manually
        • 1. Use the svstat command to take a reading, as shown in CODEEXAMPLE91.
        • 2. Within the next few minutes, take another reading.
          • CODEEXAMPLE91 FC Link Error Status Example
    • Translating Host-Device Names
      • CODEEXAMPLE92 luxadm Output for a Host Device
    • Displaying the VLUN Serial Number
      • To Display Devices That are Not Sun StorEdge Traffic Manager (MPxIO)-Enabled
        • 1. Use the format -e command.
        • 2. Type the number of the disk on which you are working at the format prompt.
        • 3. Type inquiry at the scsi prompt.
        • 4. Find the VLUN serial number in the Inquiry displayed list.
      • To Display Sun StorEdge Traffic Manager (MPxIO)-Enabled Devices
    • Viewing the Virtualization Engine Map
      • 1. To view the virtualization engine map, type:
      • 2. Optionally open a Telnet session to the virtualization engine and run the runsecfg utility to ...
      • To Failback the Virtualization Engine
        • 1. Verify that the T3+ array active path needs to be restored by viewing a live snapshot of the v...
          • CODEEXAMPLE93 Multipath Drive Summary
        • 2. If the Sun StorEdge T3+ array LUNS have failed over, run the command found in CODEEXAMPLE95...
          • CODEEXAMPLE94 Sun StorEdge T3+ array and Disk Pool Name
          • a. If no failures occur, the command exits with no output.
          • b. If failures occur, you might see one of the following messages:
            • CODEEXAMPLE95 Sun StorEdge T3+ Array Failure Codes
        • 3. If you encounter the return code 586, check the switches sw2a and sw2b and make sure the ports...
          • a. t3b0 should be plugged in to port 2 (on a 1 Gbit switch) of both sw2a and sw2b (port 1 on a 2 ...
          • b. The virtualization engine should be plugged in to port 1 (on a 1 Gbit switch) of the same two ...
        • 4. Run the showswitch(1M) command for sw2a and sw2b.
        • 5. Look at the output sections "Port Status" and "Name Server" to see if the ports are online. Th...
          • CODEEXAMPLE96 Error-Free Online Switch Ports
        • 6. If either port 1 or port 2 is offline, check the GBICs and cables.
        • 7. If a Sun StorEdge T3+ array switch port is offline, log in to the Sun StorEdge T3+ array and l...
          • CODEEXAMPLE97 Status of Sun StorEdge T3+ Array Controllers and Port List
        • 8. If either controller is in a disabled state or if either port is offline, refer to the Sun Sto...
        • 9. After the problem has been corrected, repeat Step 2.
    • Manually Clearing and Restoring the SAN Database
      • To Reset the SAN Database on Both Virtualization Engines
        • 1. Type:
        • 2. To disable the switch ports associated with the vehostname, type:
        • 3. Open a Telnet session into vehostname and clear the SAN database by entering 9 at the prompt.
        • 4. Select Q to exit the telnet session.
        • 5. To enable the switch ports associated with the vehostname, type:
        • 6. To reset the virtualization engine and force it to synchronize with its partner virtualization...
        • To Reset the SAN Database on a Single Virtualization Engine
          • 1. To disconnect the virtualization engine’s device-side FC cables, type:
          • 2. Open a Telnet session to the virtualization engine specified in Step 1.
          • 3. Enter the password.
          • 4. Type 9 to clear the SAN database.
          • 5. To reconnect the virtualization engine’s device-side FC cables, type:
          • 6. Type B to reboot the virtualization engine.
    • Restarting the slicd Daemon
      • To Restart the slicd Daemon
        • 1. Check whether the slicd daemon is running:
        • 2. Use the ipcs(1) command to check for any message queues, shared memory, or semaphores still in...
        • 3. Remove the segments by typing the following:
        • 4. To restart the slicd for the v1 virtualization engine, type:
        • 5. Confirm that the slicd daemon is running:
        • 6. If the second virtualization engine fails, power cycle the virtualization engines and make sur...
          • a. Open a Telnet session into v1_hostname.
          • b. Display the Vital Product Data (VPD) by entering .1 at the prompt.
    • Diagnosing a creatediskpools(1M) Failure
      • 1. Inspect the SUNWsecfg log file (/var/adm/log/SEcfglog) to see if any errors are indicated.
      • 2. Run the showswitch(1M) command for sw2a and sw2b.
      • 3. After corrective action has been successfully completed, run the following command:
    • Virtualization Engine Event Grid
      • To Use the Virtualization Engine Event Grid
        • FIGURE93 Virtualization Engine Event Grid
        • TABLE95 Storage Automated Diagnostic Environment Event Grid for Virtualization Engine
          • 1. Check the Sun StorEdge T3+ array for current LUN ownership.
          • 2. Use the SUNWsecfg utility on the Storage Service Processor to fail LUNs back to the correct co...
          • 1. Check power to the virtualization engine.
          • 2. Check Ethernet connectivity to the virtualization engine.
          • 3. Check the status of the slicd daemon.
          • 4. Make sure the virtualization engine is booted correctly.
          • 5. Verify the correct TCP/IP settings on the virtualization engine.
          • 6. Replace the virtualization engine, if necessary.
          • 7. Run ipcs(1) and ipcrm(1) to clean up old semaphore.
          • 1. Check the status of the slicd daemon.
          • 2. Check the power on the virtualization engine.
          • 3. Make sure the virtualization engine is booted correctly.
          • 4. Verify that the TCP/IP settings on the virtualization engine are correct.
          • 5. Check the T3 message log for failover conditions in the Sun StorEdge T3+ array.
          • 6. Replace the virtualization engine, if necessary.
          • 1. Check the status of the slicd daemon.
          • 2. Check the power on the virtualization engine.
          • 3. Make sure the virtualization engine is booted correctly.
          • 4. Verify that the TCP/IP settings on the virtualization engine are correct.
          • 5. Check the T3 message log for failover conditions in the Sun StorEdge T3+ array.
          • 6. Replace the virtualization engine, if necessary.
        • 10
        • Troubleshooting Using Microsoft Windows 2000
    • General Notes
    • Troubleshooting Tasks Using Microsoft Windows 2000
      • Launching the Sun StorEdge T3+ Array Failover Driver GUI
        • FIGURE101 Launching the Sun StorEdge T3+ Array Failover Driver
      • Checking the Version of the Sun StorEdge T3+ Array Failover Driver
        • FIGURE102 Sun StorEdge T3+ Array Failover Driver Versions 2.0.0.123 and 2.1.0.104
        • To Use the Sun StorEdge T3+ Array Failover Driver GUI
          • 1. Make sure the Sun StorEdge T3+ Array failover driver is loaded. From the Microsoft Windows 200...
          • 2. Ensure the "Jafo" driver is in a Running and OK state.
          • 3. Launch the Sun StorEdge T3+ Array Failover Driver GUI using instructions found in “Launching t...
            • FIGURE103 Healthy Sun StorEdge 3900 series system, shown using Multipath Configurator
          • 4. Compare the healthy Sun StorEdge 3900 series system to a system that has experienced a LUN fai...
            • FIGURE104 Sun StorEdge 3900 series system with a LUN failover, shown using Multipath Configurator
          • 5. To further check the affected Sun StorEdge T3+ array:
            • a. Right-click the Sun StorEdge T3+ array in the failed path.
            • b. Select Array Properties.
              • FIGURE105 Multipath Configurator Array Properties
            • c. To view details about the Sun StorEdge T3+ Array paths, click the Details button.
              • FIGURE106 Multipath Configurator LUN Properties Detail
        • To Use the Sun StorEdge T3+ Array Failover Driver Command Line Interface (CLI)
          • FIGURE107 Sun StorEdge T3+ Array Failover Driver CLI Output for the Sun StorEdge 3900 Series
          • FIGURE108 Sun StorEdge T3+ Array Failover Driver CLI Example Output for the Sun StorEdge 6900 S...
          • TABLE101 Tips for Interpreting Sun StorEdge 6910 Series CLI Output
          • 11
          • Example of Fault Isolation
            • 1. Discover the Error
              • FIGURE111 Alerts Display Using the Storage Automated Diagnostic Environment
            • 2. Check the Sun StorEdge T3+ Array Failover Driver
              • FIGURE112 Drilling Down for Sun StorEdge T3+ Array Failover Driver Fault Detail
            • 3. Check the HBA
              • FIGURE113 Fault Confirmation Using QLogic SunBlade
            • 4. Isolate the components in the path.
              • FIGURE114 Diagnostics Using QLogic SunBlade
            • 5. Again, temporarily remove the cable from the switch port in question, insert a loopback connec...
              • FIGURE115 Storage Automated Diagnostic Environment Test from Topology
              • FIGURE116 Storage Automated Diagnostic Environment Test from Topology Pull-Down Menu
              • FIGURE117 Storage Automated Diagnostic Environment Test from Topology Test Detail
              • FIGURE118 Successful Switch Test Results
            • 6. Recover the problem with the GBIC or the switch.
              • a. Recable the link between the HBA and switch.
              • b. Use the Sun StorEdge T3+ Array Failover Driver GUI for the Sun StorEdge 3900 series system, or...
                • FIGURE119 Multipath Recovery using the Sun StorEdge T3+ Array Multipath Configurator
                • FIGURE1110 Recovered Paths
            • A
          • Virtualization Engine References
    • SRN Reference
      • TABLEA1 SRN Reference
        • 1. Backup the drive data.
        • 2. Destroy the mirror drive where the failure has occurred.
        • 3. Format the drives using mode 14.
        • 4. Create a new mirror drive.
        • 5. Reassign the old SCSI ID and LUN to the new mirror drive.
        • 6. Restore the data.
        • 1. Check the condition of the virtualization engine.
        • 2. Check the cabling between the virtualization engine and daemon server.
    • SRN/SNMP Single Point-of-Failure Descriptions
      • TABLEA2 SRN/SNMP Single Point-of-Failure Table
    • Port Communication Numbers
      • TABLEA3 Port CommunicationNumbers
    • Virtualization Engine Service Codes
      • TABLEA4 Virtualization Engine Service Codes—0 -399 Host-Side Interface Driver Errors
      • TABLEA5 Virtualization Engine Service Codes—400-599 Device-Side Interface Driver Errors
      • B
      • Configuration Utility Error Messages
    • Virtualization Engine Error Messages
      • TABLEB1 Virtualization Engine Error Messages
        • 1. Run listavailable -v (which returns the status of individual virtualization engines)
        • 2. Check for the lock file directly by using ls -la /opt/SUNWsecfg/etc (look for .v1.lock or .v2....
        • 3. If the lock is set in error, use the removelocks -v command to clear.
        • 1. Run startslicd and then showlogs -e 50 to determine why startslicd could not start the daemon.
        • 2. Reset or power off the virtualization engine if the problem persists.
        • 1. Set the VEPASSWD environment variable with the proper value.
        • 2. Try to login again.
        • 1. Log in to the virtualization engine and verify that the device, host, and network settings are...
        • 2. Make sure the virtualization engine hardware is not in ERROR 50 mode.
        • 3. If required, power cycle the virtualization engine hardware, or disable the host side switch p...
        • 4. Run the setupve -n ve_name command and enable the switch port.
        • 1. Determine the host name associated with ${initiator} by using the showvemap -n ${vepair} -f co...
        • 2. Run the command resetve -n vename.
        • 1. Run the checkvemap command again.
        • 2. If this fails, check the status of both virtualization engines.
        • 3. If there is an error condition, see AppendixA for corrective action.
        • 1. If a zone name is assigned, run the rmvezone command.
        • 2. If errors still exist, run sadapter alias -d $vepair -r $initiator -a $zone -n “ “.
        • 3. Run savemap -n $vepair.
        • 1. Run the showvemap -n $vepair command to verify that the disk pool was created properly.
        • 2. If the disk pool is unavailable, run creatediskpools -n $t3name.
        • 3. If that fails, check the Sun StorEdge T3+ array for unmounted volumes or path failures, by run...
        • 1. Run checkt3mount -n $t3name -l ALL to see the mount status of the volume.
        • 2. For further information about problems with the underlying Sun StorEdge T3+ array, run checkt3...
        • 1. Check the status of both virtualization engines.
        • 2. If an error condition exists, refer to AppendixA for corrective action.
        • 3. Run the restorevemap command again.
        • 1. Check the status of the virtualization engine and reset, if necessary.
        • 2. Run the setdefaultconfig command again.
        • 1. Run setupve -n ve_hostname -v (verbose mode).
        • 2. Check the errors.
        • 3. Run checkve -n ve_hostname.
    • Switch Error Messages
      • TABLEB2 Sun StorEdge Network FC Switch Error Messages
        • 1. Check listavailable -s to see if another switch command might be updating the configuration.
        • 2. If the switch in question does not appear, check for the existence of the lock file directly b...
        • 3. If the lock is set in error, use the removelocks -s command to clear it.
        • 1. Reset the switch.
        • 2. Rerun the appropriate switch command.
        • 1. Set the SWLOGIN and SWPASSWD environment variables to the correct switch login id and password.
        • 2. Re-run the switch command.
        • 1. Select View Logs or see $LOGFILE for more details.
        • 2. Rerun setupswitch on the specified $switch.
        • 1. Attempt to activate an existing zone set.
        • 2. /opt/SUNWsecfg/flib/sanbox2 -x switchip get_zoneset_list
        • 3. From that list, select the zoneset you want to be active. It should be named something similar...
        • 4. /opt/SUNWsecfg/flib/sanbox2 -x switchip activate_zoneset zoneset
        • 5. If you are still having problems, rerun setupswitch.
        • 6. Rerun the initial command you attempted to run.
        • 1. cp /opt/SUNWsecfg/etc/ ”switch”.map /opt/SUNWsecfg/etc/ ”switch”.save
        • 2. Run saveswitch -s switch
        • 3. Manually edit the configurable items in the /opt/SUNWsecfg/etc/ ”switch”.map file to valid val...
        • 4. Rerun restoreswitch -s switch.
        • 1. Wait several minutes.
        • 2. Run ping $switch.
        • 3. If errors persist, manually power cycle the switch.
        • 1. Wait several minutes.
        • 2. Run ping $switch
        • 3. If errors persist, manually power cycle the switch.
        • 1. Check the switch chassis IDs of all switches in the SAN.
        • 2. Verify that each ID is unique.
        • 3. After the chassis IDs have been established, override the switch chassis IDs with the followin...
    • Sun StorEdge T3+ Array Partner Group Error Messages
      • TABLEB3 Sun StorEdge T3+ Array Error Messages
        • 1. Check the current Sun StorEdge T3+ array configuration with the showt3 -n <t3> command.
        • 2. Verify whether the configuration is corrupted or has changed.
        • 3. Refer to the raid.cfg files in /opt/SUNWsecfg/etc to determine if the configuration commands a...
        • 1. Replace the failed FRUs.
        • 2. Restore the Sun StorEdge T3+ array configuration with the restoret3config -f -n t3_name command.
        • 1. Refer to the T3 default/custom configuration table in the Sun StorEdge 3900 and 6900 Series 2....
        • 2. Use showt3-n t3_name to display the present configuration.
        • 3. Check the Sun StorEdge T3+ firmware version (it should be version 2.01.00 or higher). Upgrade ...
        • 1. Check the Sun StorEdge T3+ network connection.
        • 2. Check with Ping $t3_name command to determine if the Sun StorEdge T3+ array is operating.
        • 1. Check the Sun StorEdge T3+ master and alternate master network connection.
        • 2. Check with Ping $t3_name command to determine if the Sun StorEdge T3+ array is operating.
        • 1. Check whether any other secfg utility is running.
        • 2. If an secfg utility is running, allow it to finish.
        • 1. Verify that the snapshot files are saved and have read permissions in the /opt/SUNWsecfg/etc/t...
        • 2. If the snapshot files are not available, create them using the savet3config command.
        • 1. Run the showt3 -n command to verify that the requested LUN exists on the Sun StorEdge T3+ array.
        • 2. Confirm that the Sun StorEdge T3+ array configuration matches standard configurations.
        • 1. Check the Sun StorEdge T3+ array firmware level and verify it is 2.01.00 or higher.
        • 2. Check if volume slicing is supported and enabled on the Sun StorEdge T3+ array.
        • 1. Check the Sun StorEdge T3+ array firmware level and verify it is 2.01.00 or higher.
        • 2. Upgrade the firmware, if required.
        • 1. Check to see if any secfg T3+ commands are being executed.
        • 2. If the commands are executing, wait for them to complete.
        • 3. Run creatediskpools -n t3name.
        • 1. Check the Sun StorEdge T3+ configuration with the showt3 -n t3_name command.
        • 2. Refer to the Sun StorEdge T3 and T3+ documentation.
        • 1. Check for snapshot files in the /opt/SUNWsecfg/etc/t3_name/ directory.
        • 2. If the snapshot files are not found, use the modifyt3config command to configure the Sun StorE...
        • 1. Check the volume status using the checkt3mount or showt3 command.
        • 2. If unmounted, use the restoret3config command to mount.
        • 1. If the configuration is different from standard Sun StorEdge T3+ array configuration, run the ...
        • 2. Use the modifyt3config command to reconfigure the device.
        • 1. Create a Sun StorEdge T3+ array slice.
        • 2. Before setting permissions, use the createt3slice command.
    • Other Error Messages
      • TABLEB4 Other SUNWsecfg Error Messages
      • Abbreviations and Acronyms
      • ASIC
      • CLI
      • CRC
      • DAS
      • EOF
      • FC
      • FC-ELS
      • FRU
      • GBIC
      • GUI
      • HBA
      • ISL
      • LED
      • LUN
      • MAC
      • NSCC
      • PCU
      • PDU
      • PFA
      • POST
      • RAID
      • RARP
      • RFE
      • RSS
      • SAN
      • SCSI
      • SLIC
      • SNMP
      • SPOF
      • SRN
      • SRS
      • SSP
      • SVE
      • TCP/IP
      • VLUN
      • WWN
        • Index