Consolidated Etched test-harness error codes — MLT / 1X, L10, Common and L11 —
with revision history and original author per code. Field names and enum values follow
th_registry.yaml (TH Error Code Specification v0.3 §7, §11.1), the source of truth.
th_registry.yaml: never renumber,
never reuse a code — a change of meaning requires a new code. The -SxQy
suffix carries severity and quick action; the stable identity is the TH-<BLOCK>-<NNNN> prefix.
Code tables below are refreshed from the registry snapshots. The spreadsheet export and its verbatim CSVs are unchanged.
| File | What it is | Size |
|---|---|---|
| etched_error_code.xlsx | The source workbook — a mirror of the original spreadsheet. Same tabs in the same order, same columns, same rows, nothing added or reordered. This is the file to edit and commit. | 47 KB |
| etched_error_code_annotated.xlsx | The joined view: version, original author, severity and quick-action names, owner, since, and registry status, with the code tables refreshed from the current registries. | 173 KB |
data/source/revision_history.csv mlt_1x.csv l10.csv l11.csv field_definitions.csv dri_ownership.csv suite_run_log.csv | Each source tab as CSV, verbatim — the diffable form of the workbook above. | 7 files |
data/revision_history.csv mlt_1x.csv l10.csv common.csv l11.csv field_definitions.csv error_id_encoding.csv dri_ownership.csv source_data_notes.csv enum_legends.csv | Joined tabs as CSV. MLT / 1X, L10 and Common are refreshed from the registries. L11 is the previous spreadsheet draft. | 10 files |
| sheet.md | Verbatim export of the Google Sheet the earlier tables were built from. | 104 KB |
| th_registry.yaml MLT/mlt_th_registry.yaml L10/l10_th_registry.yaml common_th_registry.yaml | Snapshots of host/system_test/error_codes at sw@65cf3520a606. | 4 files |
From the source spreadsheet, oldest revision first.
| Version | Comment | Author |
|---|---|---|
| 0.1 | Draft | Ulysses Kao |
| 0.2 | update the test case and error code for SLT 1X | Ulysses Kao |
| 0.3 | Update the error code | Ulysses Kao |
| 0.4 | Refresh from host/system_test/error_codes at sw@65cf3520a606 | supercomputing-sw |
| Field | Description |
|---|---|
Error Code ID | The key ID of Error code |
Message | The possible error message |
Quick_Action | How does the user or operate handle this issues immediately |
Recover_Step | Detail recover steps for this error |
Source | Error from which test or software |
Error_Type | Error types |
Component | Relative components |
Test_Case | Jira IDs of test cases, allow multiple IDs, separate by “,” |
Bugs | Jira IDs of bugs, allow multiple IDs, separate by “,” |
Count | Count of happen |
Last_happen | Datetime of last happen |
Character layout of the 12-hex-digit EC- identifier used by the L11 block.
| Char # | Describe | Define |
|---|---|---|
0 | Priority | 0~2: Priority 0~2 3~F: Reserve |
1 | Quick action | 0: No act 1: Cold Reboot 2: Warm Reboot 3: Soft power off 4: Hard shutdown 5: Disable component 6: Undefined 7: Hot remove 8: Hot switch 9~F: Reserve |
2~3 | Source | FF: Forward from out of Etched 00: MTS 01: SLT 02: 1X Test 03: L10 Test 04: L11 Test 05: Kayak 06: RMS 07: PDU Daemon 08: CDU Daemon 09: BMC 0A: BIOS 0B~FE reserve |
4~5 | Error Type | 00: Undefine 01: Initial 02: Connection 03: Provision 04: Config (HW/SW/FW) 05: Unit Test 06: Function 07: Stress 08: Sensors 0A~FF: reserve |
6~7 | Component | 00: Undfine 01: Soho chip 02: CPU 03: DIMM 04: PDU 05: CDU 06: RMS 07: BMC 08: BIOS 09: Storage 0A: ToR Switch 0B: Manage switch 0C: Temp 0D: Fan 0E: PSU 0F~0xFF: reserve |
8~11 | Serial number | The serial number of error code From 0000~FFFF |
Values as used by category, severity and quick_action in the registries.
| Value | Name |
|---|---|
0 | CATEGORY_NONE |
1 | CATEGORY_NETWORK |
2 | CATEGORY_STORAGE |
3 | CATEGORY_MEMORY |
4 | CATEGORY_AUTH |
5 | CATEGORY_CONFIG |
6 | CATEGORY_HARDWARE |
7 | CATEGORY_SOFTWARE |
8 | CATEGORY_API |
9 | CATEGORY_RESOURCE |
10 | CATEGORY_SECURITY |
11 | CATEGORY_DATA |
12 | CATEGORY_INTERNAL |
| Value | Name |
|---|---|
0 | ERROR_NONE |
1 | ERROR_MONITOR |
2 | ERROR_ISOLATE |
3 | ERROR_UNKNOWN |
4 | ERROR_TRIAGE |
5 | ERROR_CONFIG |
6 | ERROR_RESET |
| Value | Name |
|---|---|
0 | QA_NO_ACT |
1 | QA_COLD_REBOOT |
2 | QA_WARM_REBOOT |
3 | QA_SOFT_POWER_OFF |
4 | QA_HARD_SHUTDOWN |
5 | QA_DISABLE_COMPONENT |
6 | QA_RETRY |
7 | QA_HOT_REMOVE |
8 | QA_HOT_SWITCH |
9 | QA_ESCALATE |
Refreshed from MLT/mlt_th_registry.yaml and 1X-only leftovers. 78 codes.
| Error Code ID | Ver | Doc Rev | Original Author | Name | Message | Severity | Quick Action | Recover / Troubleshooting Procedure | Error Type | Component | Test Case | Source | Bugs | Owner | Since |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
TH-C2C-0001-S2Q9 | v1 | 0.2 | Mykola Servetnyk | C2C_LINKUP_FAILED | C2C lanes failed to train or link up within the timeout: RX equalization did not converge after its configured attempts, or the link-up test reported one or more datalinks/lanes down. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.C2C.SOHU_SERDES | C2cLinkupTestCaseC2cLinkupMultiChipTestCaseC2cThroughputTestCaseC2cThroughputMultiChipTestCase | 1X Module Test | ETCH-33573: C2cLinkupTestCase Failure | supercomputing-sw | harness 2026.216 |
TH-C2C-0002-S2Q9 | v1 | 0.2 | Mykola Servetnyk | C2C_LOOPBACK_DATA_INTEGRITY_MISMATCH | Bit-by-bit C2C loopback comparison found a data mismatch on at least one datalink/lane. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Data | 1X.C2C.SOHU_SERDES | C2cIntegrityTestCaseC2cIntegrityMultiChipTestCase | 1X Module Test | ETCH-33915: C2cIntegrityTestCase failure | supercomputing-sw | harness 2026.216 |
TH-C2C-0003-S2Q9 | v1 | 0.2 | Mykola Servetnyk | C2C_LOOPBACK_THROUGHPUT_BELOW_THRESHOLD | The large C2C transfer did not meet its acceptance criteria: measured throughput below the configured percentage of line rate (90% in the 1X suite), or the post-FEC uncorrected error fraction above its limit. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.C2C.SOHU_SERDES | C2cThroughputTestCaseC2cThroughputMultiChipTestCase | 1X Module Test | ETCH-32191: C2cIntegrityTestCase/C2cThroughputTestCase Failures | supercomputing-sw | harness 2026.216 |
TH-C2C-0004-S4Q9 | v1 | 0.2 | Mykola Servetnyk | C2C_LANE_MARGIN_BELOW_THRESHOLD | Worst per-lane minimum BER from the eye-diagram sweep exceeded the configured ber_threshold. The SERDES margin spec is not final, so this code is severity 4 (triage) rather than a straight isolate: confirm against the current spec before dispositioning a module. | S4 TRIAGE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.C2C.SOHU_SERDES | C2cLaneMarginTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-C2C-0005-S2Q9 | v1 | 0.3 | Mykola Servetnyk | C2C_PRBS_ERROR_THRESHOLD_EXCEEDED | The C2C PRBS run completed but the measured pre-FEC bit-error rate exceeded max_pre_fec_ber on at least one lane. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | CATEGORY_DATA | 1X.C2C.SOHU_SERDES | C2cPrbsLoopbackTestCaseC2cPrbsMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.216 |
TH-C2C-0009-S2Q9 | v1 | 0.4 | Ulysses Kao | C2C_LINK_STABILITY_FAILED | A C2C lane lost lock in the stability test and went link down. This is a risk for kayak and this should be investigated. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.C2C.SOHU_SERDES | C2cLinkStabilityMultiChipTestCase | MLT / 1X Module Test | — | accelerator-sw | harness 2026.229 |
TH-DMA-0001-S2Q6 | v1 | 0.2 | Sam Jiang | DMA_DATA_PATH_FAIL | Host-chip DMA data-path test failure: dma_test_app reported a data verification or March C+ failure on an SRAM region (RopeSram / CentralSram), or failed in a way its output does not attribute to a more specific PCIe/HBM failure mode (fallback for this test). | S2 ISOLATE | Q6 RETRY | Suspected test issue (single chip test timeout took down the station), retest once. | Hardware | 1X.DMA.HOST_CHIP_PATH | SohuDmaTestCase | 1X Module Test | ETCH-35873: SohuDmaTestCase failure | supercomputing-sw | harness 2026.216 |
TH-FW-0001-S2Q9 | v1 | 0.2 | Mykola Servetnyk | SOHU_FIRMWARE_UPDATE_FAILED | The Sohu application firmware update did not complete: loading or executing the image package raised, the ASIC did not become pingable afterwards, or the ASIC came back without reporting application firmware. For the hot-reload case it also covers the pre-update eligibility outcomes, where nothing was flashed at all — the chip is running a legacy bootloader, is in bootloader mode with no application PLDM endpoint, or the precheck could not read the running versions over RPC/proxy. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Config | 1X.FW.SOHU_APP_FIRMWARE | SohuFirmwareUpdateTestCaseSohuHotReloadFirmwareUpdateTestCaseSohuSecureBootRecoveryUpdateTestCase | 1X Module Test | ETCH-34843: SohuHotReloadFirmwareUpdateTestCase failure | supercomputing-sw | harness 2026.216 |
TH-FW-0002-S4Q9 | v1 | 0.2 | Mykola Servetnyk | SOHU_FIRMWARE_IMAGE_HASH_MISMATCH | The ASIC booted application firmware after the update, but the running image hash from get_version_info() does not match the hash in the image package that was just staged. | S4 TRIAGE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Config | 1X.FW.SOHU_APP_FIRMWARE | SohuFirmwareUpdateTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-FW-0003-S2Q6 | v1 | 0.2 | Aidan Holm | BOOTLOADER_PROGRAMMING_FAILED | Bootloader programming failed (flash program fail or no ping after program). | S2 ISOLATE | Q6 RETRY | Verify that the green power indicators on the Sohu module are ON. Verify that both the PCIe cable and power cable are connected correctly and securely. If there is no trouble found, retest once. | Hardware | 1X.FW.BOOTLOADER | ProgramSecureBootloaderTestCaseBootloaderResultTestCase | 1X Module Test | ETCH-35585: BootloaderResultTestCase failures | supercomputing-sw | harness 2026.217 |
TH-FW-0004-S4Q6 | v1 | 0.2 | Aidan Holm | JTAG_IMAGE_LOAD_VERIFY_FAILED | JTAG image load/verify failed (ICCM verify mismatch). | S4 TRIAGE | Q6 RETRY | Suspected test issue, retest once. | Hardware | 1X.FW.JTAG | ProgramSecureBootloaderTestCaseBootloaderResultTestCase | 1X Module Test | ETCH-35586: ProgramSecureBootloaderTestCase failure | supercomputing-sw | harness 2026.217 |
TH-FW-0005-S5Q2 | v1 | 0.2 | Ulysses Kao | FIXTURE_FIRMWARE_UPDATE_FAILED | Fixture firmware update failed (golden VBB / retimer): OpenOCD flash, ST-Link adapter, post-flash UART/RPC ready check, or application firmware-hash verification did not succeed. | S5 CONFIG | Q2 WARM_REBOOT | Reboot Devkit, retest once. | Config | 1X.FW.FIXTURE | VbbFirmwareUpdateTestCaseRetimerFirmwareUpdateTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.217 |
TH-FW-0006-S5Q9 | v1 | 0.2 | Mykola Servetnyk | BOARD_DVFS_PROFILE_FAILED | Board DVFS profile detection or application failed. | S5 CONFIG | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Config | 1X.FW.DVFS_PROFILE | DetectBoardDvfsProfileTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-FW-0007-S2Q6 | v1 | 0.2 | Ulysses Kao | SECURE_BOOT_RECOVERY_UPDATE_TIMEOUT | The PLDM Update Agent did not complete within pldm_update_timeout_s during a secure-boot recovery update - the device is likely not responding over the Sohu UART in recovery mode. | S2 ISOLATE | Q6 RETRY | Verify UART cabling and that recovery straps engage; rerun once; preserve logs and route to FW triage | Config | MLT.FW.SOHU_CHIP | SohuSecureBootRecoveryUpdateTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-FW-0008-S2Q6 | v1 | 0.2 | Ulysses Kao | SECURE_BOOT_RECOVERY_UPDATE_FAIL | The VFIO app firmware update through secure-boot recovery mode failed - the recovery strap sequence failed or the PLDM Update Agent reported an update failure (non-zero pldm_ua exit, update rejected by the device). Tooling/deployment gaps are TH-HAR-0003; invalid args are TH-HAR-0002. | S2 ISOLATE | Q6 RETRY | Verify UART cabling/seating and rerun recovery-update subset once; then preserve unit + pldm_ua log and verify package vs release; route to FW triage - do NOT keep retrying as device may be mid-update | Config | MLT.FW.SOHU_CHIP | SohuSecureBootRecoveryUpdateTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-FW-0009-S5Q9 | v1 | 0.2 | James Chou | SOHU_HOT_RELOAD_NOT_ELIGIBLE | The hot reload could not start, so no firmware was written: the chip is running a legacy bootloader rather than secure_bootloader, it is in bootloader mode with no application PLDM endpoint, or the precheck could not read the running versions over RPC/proxy. The suite must run secure-boot recovery or bootstrap before this test case. Distinct from TH-FW-0001, which covers an update that was attempted and failed. | S5 CONFIG | Q9 ESCALATE | Escalate to Etched with the eligibility outcome and the running image name; nothing was flashed, so do not route the unit to rework or FA. | Config | 1X.FW.SOHU_APP_FIRMWARE | SohuHotReloadFirmwareUpdateTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.217 |
TH-FW-0018-S5Q9 | v1 | 0.3 | Mykola Servetnyk | SOHU_FIRMWARE_VERSION_MISMATCH | The running Sohu firmware is not the image the suite requires: application firmware was expected, but the chip reported only a bootloader slot. The inverse (application firmware running when the step did not expect it) is a suite sequencing problem and stays unclassified. | S5 CONFIG | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | CATEGORY_CONFIG | 1X.FW.SOHU_APP_FIRMWARE | CheckFirmwareVersionTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HBM-0001-S2Q9 | v1 | 0.2 | Jack Chou | HBM_PMBIST_FAILURE | HBM PMBIST failure. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Memory | 1X.HBM.STACK | SohuPmbistTestCase | 1X Module Test | ETCH-33885: Power Virus failures [was SohuPmbistTestCase failures] | supercomputing-sw | harness 2026.216 |
TH-HBM-0003-S4Q9 | v1 | 0.2 | Jack Chou | HBM_EYE_MARGIN_BELOW_THRESHOLD | HBM training / eye margin below threshold (tentative). | S4 TRIAGE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Memory | 1X.HBM.STACK | SohuRdqsSweepTrainingTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-HBM-0004-S2Q6 | v1 | 0.2 | Jack Chou | HBM_MTC_STRESS_FAILURE | HBM MTC stress test failure. | S2 ISOLATE | Q6 RETRY | Suspected test issue, retest once. | Memory | 1X.HBM.STACK | SohuMtcStressTestCase | 1X Module Test | ETCH-35299: SohuMtcStressTestCase failures | supercomputing-sw | harness 2026.216 |
TH-HBM-0005-S2Q9 retired | v1 | 0.2 | Ulysses Kao | HBM_BANDWIDTH_BELOW_THRESHOLD | PCIe-to-HBM read or write throughput did not reach the configured floor. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Memory | 1X.HBM.STACK | — | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-HBM-0006-S2Q9 | v1 | 0.2 | James Chou | HBM_LANE_REPAIR_FAILED | Lane repair established that one or more HBM pseudo-channels cannot be repaired within spare coverage. Covers a completed sweep with per-target errors (fail_fast=False), and fail_fast raises that prove unrepairability (exhausted candidate byte pairs, multi-bit / aword / DWORD LFSR unrepairable, or applied repairs still failing verification). Distinct from TH-HBM-0008, which covers an aborted sweep where repairability was never established (RPC / firmware / setup). | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Memory | 1X.HBM.STACK | SohuLaneRepairTestCase | 1X Module Test | ETCH-35587: SohuLaneRepairTestCase failure | supercomputing-sw | harness 2026.216 |
TH-HBM-0007-S2Q9 | v1 | 0.2 | James Chou | HBM_DEVICE_ID_READ_FAIL | The IEEE 1500 DRAM device ID could not be read from every HBM stack, or the reported stack topology did not match the expected stack count. Covers an unreadable device ID (RPC error, timeout, or unexpected response) as well as a missing channel id or wrong number of stacks in the reported system info. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Data | 1X.HBM.STACK | SohuHbmDeviceIdTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-HBM-0008-S4Q9 | v1 | 0.2 | James Chou | HBM_LANE_REPAIR_INCOMPLETE | The lane-repair detection sweep did not run to completion, so whether a passing repair exists was never established. Covers unexpected detection abort (RPC / TypeError / firmware reload after clearing persisted repairs) before an unrepairable verdict. Distinct from TH-HBM-0006, which covers proven unrepairability (exhausted candidates, multi-bit / aword / DWORD LFSR unrepairable, or repairs that still fail verification). | S4 TRIAGE | Q9 ESCALATE | Escalate to Etched with the detection exception and the partial repair map; repairability is unknown rather than failed, so do not route the unit to FA. | Memory | 1X.HBM.STACK | SohuLaneRepairTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.217 |
TH-HBM-0014-S5Q9 | v1 | 0.4 | Ulysses Kao | HBM_NOT_TRAINED_FOR_REQUIRED_SPEED | The chip does not carry HBM DCDL training covering every pseudo-channel of every HBM at the data rate the suite requires, or is running a different rate. Also covers a Flash init-config overlay that does not decode at all, which means no persisted training exists. Firmware refuses an untrained module its trained production point, and nothing else in the suite fails on such a chip, so without this check an untrained socket ships. The emitting case only reads the persisted records and the running speed bin; it changes nothing on the chip. | S5 CONFIG | Q9 ESCALATE | Confirm the unit's board/chip swap history since its last HBM training run. Request an HBM training run at the required data rate for the HBMs named by the untrained bitmask, then rerun this suite. If training completes and the socket still reports untrained HBMs, escalate to Etched with the suite logs. | Config | 1X.HBM.STACK | — | MLT / 1X Module Test | — | supercomputing-sw | harness 2026.245 |
TH-MEZZ-0001-S2Q6 | v1 | 0.2 | Aidan Holm | MEZZANINE_CPLD_ACCESS_WRITE_FAILED | Mezz/eval CPLD register access or write failure. | S2 ISOLATE | Q6 RETRY | Verify that the green power indicators on the Sohu module are ON. Verify that both the PCIe cable and power cable are connected correctly and securely. If there is no trouble found, retest once. | Hardware | 1X.MEZZ.CPLD | ProgramSecureBootloaderTestCaseBootloaderResultTestCase | 1X Module Test | ETCH-35586: ProgramSecureBootloaderTestCase failure | supercomputing-sw | harness 2026.217 |
TH-MEZZ-0002-S2Q6 | v1 | 0.2 | Aidan Holm | UNEXPECTED_MEZZANINE_I2C_ALERT | Unexpected mezzanine I2C alert asserted. | S2 ISOLATE | Q6 RETRY | Suspected test issue, retest once. | Hardware | 1X.MEZZ.I2C | MezzanineI2cAlertsTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.217 |
TH-MEZZ-0003-S2Q6 | v1 | 0.4 | Aidan Holm | EXPECTED_MEZZANINE_I2C_ALERT_NOT_OBSERVED | Expected mezzanine I2C alert was not observed. | S2 ISOLATE | Q6 RETRY | Suspected test issue, retest once. | Hardware | 1X.MEZZ.I2C | MezzanineI2cAlertsTestCase | MLT / 1X Module Test | — | supercomputing-sw | harness 2026.219 |
TH-PCIE-0001-S2Q6 | v1 | 0.2 | Jack Chou | PCIE_LINK_STATE_MISMATCH | PCIe link state mismatch: the Sohu endpoint link is not L0 / Gen5 / x16. | S2 ISOLATE | Q6 RETRY | Suspect PCIe cable not fully seated into PV1 board, reseat the cable and retest once. | Hardware | 1X.PCIE.LINK | SohuPcieAerCheckTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-PCIE-0002-S2Q6 | v1 | 0.2 | Jack Chou | PCIE_AER_ERRORS_DETECTED | PCIe AER errors detected (correctable or uncorrectable). | S2 ISOLATE | Q6 RETRY | Suspect PCIe cable not fully seated into PV1 board, reseat the cable and retest once. | Hardware | 1X.PCIE.LINK | SohuPcieAerCheckTestCase | 1X Module Test | ETCH-34150: Power Virus failures [was SohuPcieAerCheckTestCase failures] | supercomputing-sw | harness 2026.216 |
TH-PCIE-0003-S2Q9 | v1 | 0.2 | Sam Jiang | PCIE_CSRAM_DMA_THROUGHPUT_BELOW_THRESHOLD | PCIe->CSRAM DMA throughput below the qualified floor (suite-configured; 50 GB/s for the 1X module suite). Emitted by SohuDmaTestCase when its CentralSram throughput check fails in the active DMA flow. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.PCIE.CSRAM_DMA | SohuDmaTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-PCIE-0004-S2Q9 | v1 | 0.2 | Sam Jiang | PCIE_HBM_DMA_INTEGRITY_MISMATCH | PCIe->HBM DMA data integrity mismatch: dma_test_app read back data from an HBM region (EastHbm / WestHbm) that does not match the written random / bit-inverse pattern. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Data | 1X.PCIE.HBM_DMA | SohuDmaTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-PCIE-0005-S2Q9 | v1 | 0.2 | Sam Jiang | PCIE_HBM_DMA_THROUGHPUT_BELOW_THRESHOLD | PCIe->HBM DMA throughput below the qualified floor (suite-configured; 16 GB/s for the 1X module suite) on EastHbm / WestHbm transfers in dma_test_app. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.PCIE.HBM_DMA | SohuDmaTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-PCIE-0006-S2Q6 | v1 | 0.2 | Ulysses Kao | PCIE_SPEED_CHANGE_FAILED | PCIe speed change failed (Gen5<->Gen1 transition did not complete). | S2 ISOLATE | Q6 RETRY | Suspect PCIe cable not fully seated into PV1 board, reseat the cable and retest once. | Hardware | 1X.PCIE.LINK | — | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-PCIE-0007-S4Q6 | v1 | 0.2 | Jack Chou | PCIE_LANE_MARGIN_BELOW_THRESHOLD | PCIe lane margin below threshold (EH>25mV EW>0.2UI, tentative). | S4 TRIAGE | Q6 RETRY | Suspect PCIe cable not fully seated into PV1 board, reseat the cable and retest once. | Hardware | 1X.PCIE.LINK | SohuPcieLaneMarginTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-PCIE-0008-S2Q6 | v1 | 0.2 | Aidan Holm | SOHU_ENDPOINT_NOT_ENUMERATED | Sohu endpoint not enumerated (absent from PCIe tree / RPC proxy). | S2 ISOLATE | Q6 RETRY | Suspect PCIe cable not fully seated into PV1 board, reseat the cable and retest once. | Hardware | 1X.PCIE.ENUMERATION | PcieSetupTestCaseSohuVfioPingTestCaseSohuSaScreeningTestCase | 1X Module Test | ETCH-35872: Chips not enumerated by RPC proxy | supercomputing-sw | harness 2026.217 |
TH-PWR-0002-S4Q9 | v1 | 0.2 | Jack Chou | POWER_TELEMETRY_UNAVAILABLE | Power telemetry unavailable (VBB / VRM / VM sensor RPC / collectors not responding, or expected VM sensors missing from the reading response). | S4 TRIAGE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.PWR.VBB_POWER | SohuPowerTelemetryTestCaseSohuVrmTestCaseSohuVmTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-PWR-0003-S2Q9 | v1 | 0.2 | Jack Chou | POWER_DATA_OUT_OF_RANGE | Power data out of the qualified range, or CPLD reports an active VRM FSM fault on the mezzanine. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.PWR.VBB_POWER | SohuPowerTelemetryTestCaseSohuVrmTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-PWR-0004-S2Q9 | v1 | 0.2 | Ulysses Kao | VM_SENSOR_OUT_OF_TOLERANCE | VM sensor reading outside +/-5% of nominal. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.PWR.VM_SENSOR | SohuVmTestCase | 1X Module Test | ETCH-34151: SohuVmTestCase failure | supercomputing-sw | harness 2026.216 |
TH-PWR-0005-S2Q9 | v1 | 0.2 | Ulysses Kao | RAIL_ADJUST_READBACK_MISMATCH | Voltage rail adjustment not reflected in the VM readback. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.PWR.VM_SENSOR | SohuVmTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-PWR-0006-S4Q9 | v1 | 0.2 | James Chou | VMIN_ABOVE_RATED_LIMIT | Vmin above the rated limit for an SA/SAU rail. | S4 TRIAGE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.PWR.VMIN | SohuLlamaMttTestCase | 1X Module Test | ETCH-32910: SohuLlamaMttTestCase Failures | supercomputing-sw | harness 2026.216 |
TH-PWR-0007-S2Q9 | v1 | 0.2 | James Chou | UNEXPECTED_DVFS_RAMPDOWN | Unexpected DVFS rampdown event during a workload. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.PWR.DVFS | SohuLlamaMttTestCaseSohuPowerVirusTestCase | 1X Module Test | ETCH-32910: SohuLlamaMttTestCase Failures | supercomputing-sw | harness 2026.216 |
TH-PWR-0008-S2Q6 | v1 | 0.2 | Mykola Servetnyk | MODULE_POWER_ON_FAILURE | Module failed to power on (indicators off, rail fault or short). | S2 ISOLATE | Q6 RETRY | Verify that the green power indicators on the Sohu module are ON. Verify that both the PCIe cable and power cable are connected correctly and securely. If there is no trouble found, retest once. | Hardware | 1X.PWR.POWER_ON | BootloaderResultTestCase | 1X Module Test | ETCH-33918: SohuSecureBootRecoveryUpdateTestCase failure | supercomputing-sw | harness 2026.216 |
TH-SN-0001-S2Q9 | v1 | 0.2 | Sam Jiang | CHIP_ID_READ_OR_MISMATCH | The sohu_chip_id sysfs attribute of an enumerated Sohu endpoint could not be read or parsed, read back as all ones (device in reset), or reported a chip ID other than the one the resource configuration expects for that PCIe location. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Data | 1X.SN.CHIP_ID | SohuChipIdTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-SOHU-0001-S2Q9 | v1 | 0.2 | Aidan Holm | ASIC_RESET_DEASSERT_FAILED | ASIC reset deassert failed. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.SOHU.RESET | ProgramSecureBootloaderTestCaseBootloaderResultTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.217 |
TH-SOHU-0002-S2Q9 | v1 | 0.2 | James Chou | SOHU_NOT_PINGABLE | The Sohu device did not respond to a ping over the RPC proxy: the firmware is not running, not reachable, or the device dropped off the proxy. Emitted when a device that had been posting stops answering, for example the post-campaign sweep after a repeated-FLR run. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.SOHU.ASIC | SohuFlrTestCaseSohuVfioPingTestCaseSaSortRampupHbmSingleChipTestCaseSohuWeightLoadTestCase | 1X Module Test | ETCH-35300: SaSortRampupHbmSingleChipTestCase failures | supercomputing-sw | harness 2026.216 |
TH-SOHU-0003-S2Q9 | v1 | 0.2 | Ulysses Kao | PLDM_SENSOR_CHECK_FAILED | PLDM sensor discoverability or sanity check failed (boot-phase gate). | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.SOHU.PLDM | SohuSecureBootRecoveryUpdateTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-SOHU-0004-S2Q9 | v1 | 0.2 | James Chou | I2C_PERIPHERAL_READ_FAIL | An I2C transaction on the Sohu mezzanine failed. This covers bring-up (initializing the ASIC-debug mux or any of the four Sohu I2C controllers, and reading the mezzanine board type) as well as the per-device probes on I2C0-I2C3, where a probe either failed its transaction or returned data that did not match the expected device identifier/register value. The first failure ends the test; for a probe it identifies the bus, device, and address. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.SOHU.MEZZ_I2C | SohuI2cTestCase | 1X Module Test | ETCH-35872: Chips not enumerated by RPC proxy | supercomputing-sw | harness 2026.216 |
TH-SOHU-0005-S2Q9 | v1 | 0.2 | James Chou | GPIO_LOOPBACK_MISMATCH | An MCPU GPIO did not read back its driven value. Covers both halves of the test: the chip-ID strap pins reading a value other than the expected chip_id, and the per-pin output->input loopback failing to observe either the flipped value or the restored original. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.SOHU.MCPU_GPIO | SohuGpioTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-SOHU-0006-S2Q9 | v1 | 0.2 | Ulysses Kao | UART_PING_LOG_MISMATCH | UART ping log message absent or content mismatch. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.SOHU.UART | SohuUartTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-SOHU-0007-S2Q9 | v1 | 0.2 | James Chou | JTAG_READBACK_MISMATCH | A JTAG check did not produce the expected value, or could not be read at all: an ARC core cluster_id, the SA row/column build configuration, or a failure to select a core or read a register over the chain. The core1_cpu_reset read is recorded as a diagnostic measurement only and never fails the test. Failures before any JTAG transaction take the harness codes instead — an unknown jtag_interface is TH-HAR-0002, and no adapter matching serial_prefix or a missing OpenOCD binary are TH-HAR-0003. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.SOHU.JTAG | SohuJtagTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-SOHU-0008-S2Q9 | v1 | 0.2 | Sam Jiang | EFUSE_REGION_READ_FAILURE | Reading the eFuse region over RPC failed, so no init config could be decoded from the module. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.SOHU.EFUSE | SohuEfuseTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-SOHU-0009-S2Q9 | v1 | 0.2 | Sam Jiang | EFUSE_WS_FT_STATUS_INCOMPLETE | The eFuse read succeeded but a wafer-sort or final-test completion flag (wafer_sort_complete, final_test_complete, mbist_complete, sa_complete, sensor_complete or hbm_complete) is False, so the module carries no record of a completed WS/FT flow including MBIST and SA repair. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Data | 1X.SOHU.EFUSE | SohuEfuseTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-SOHU-0010-S2Q9 | v1 | 0.2 | Sam Jiang | BUS_CONNECTIVITY_CSR_ACCESS_FAIL | The mCPU-resident bus connectivity sweep reported a non-zero failed count: at least one CSR check across the SoC blocks (MCPU, ISC, IPU, CSRAM, WCU, GPIO, PAD, sensors hub, PCIe, SAMU, datalink, MACs) did not pass. A check fails either because a write/verify/restore readback did not return the written pattern, or because a read-only field did not hold its expected value. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.SOHU.SOC_BUS | SohuBusConnectivityTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-SOHU-0011-S2Q9 | v1 | 0.2 | Ulysses Kao | SRAM_TEST_FAIL | On-chip SRAM test failure (stuck bits / readback mismatch) - the RPC read/write/invert checks reported failing SRAM regions, or the March C+ binary exited non-zero. Failing SRAM names go in the message / affected_instances, never into new codes (spec §1.2). Note - a non-zero March C+ exit is not yet distinguishable from a launch error; refine when the binary exposes an exit-status contract. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Memory | MLT.SOHU.SOHU_CHIP | SohuSramMemoryTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-SOHU-0012-S2Q9 | v1 | 0.2 | Sam Jiang | SA_MATMUL_NUMERIC_MISMATCH | An SA matmul dispatcher test binary exited non-zero: systolic-array outputs did not match the funcsim reference. sa_sort screening tolerates a bounded number of defective SA columns, so a non-zero exit means the mismatches went beyond what the health map is allowed to mask out. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.SOHU.SA | SaSortRampupHbmSingleChipTestCase | 1X Module Test | ETCH-34131: SohuPowerVirusTestCase failures | supercomputing-sw | harness 2026.216 |
TH-SOHU-0014-S2Q9 | v1 | 0.2 | Sam Jiang | ATTENTION_SAU_NUMERIC_MISMATCH | An attention/SAU dispatcher test binary exited non-zero: one or more attention, KV-flush or SAU operations did not match the funcsim reference. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.SOHU.SAU | RunAllAttnTestsSingleChipTestCaseRunAllAttnHbmBypassTestsSingleChipTestCaseRunKvFlushTestsSingleChipTestCaseKvFlushAndAttentionSingleChipTestCaseKvFlushAndAttentionInterleavedSingleChipTestCase | 1X Module Test | ETCH-35940: 267693980000009 - Failed RunAllAttnTestsSingleChipTestCase | supercomputing-sw | harness 2026.216 |
TH-SOHU-0015-S2Q9 | v1 | 0.2 | Aidan Holm | WORKLOAD_HANG_OPERATION_TIMEOUT | Workload hang / operation timeout. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.SOHU.WORKLOAD | KayakTestCaseSohuPowerVirusTestCaseSohuLlamaMttTestCase | 1X Module Test | ETCH-35548: SohuPowerVirusThermalTestCase failures | supercomputing-sw | harness 2026.216 |
TH-SOHU-0016-S2Q9 | v1 | 0.2 | Ulysses Kao | WEIGHT_LOAD_SPOT_CHECK_MISMATCH | Weight-load spot check failed: HBM-resident weights did not match the on-disk model after load (weight_load_test_app non-zero exit). | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Data | 1X.SOHU.WEIGHT_LOAD | SohuWeightLoadTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-SOHU-0017-S2Q9 | v1 | 0.2 | James Chou | INFERENCE_DISPATCH_TIME_ABOVE_LIMIT | Sustained inference throughput on the million-token test did not reach the expected chip tokens/s (95% of the suite's expected_chip_tokens_per_second). Not wired yet: the check runs inside llama_mtt_app and the harness only sees the exit status, so a non-zero exit cannot be told apart from a model-creation or dataset failure. Emitting it needs the binary to signal the threshold failure distinctly, or the harness to parse MTT_results/metrics.txt. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.SOHU.ASIC | SohuLlamaMttTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-SOHU-0018-S2Q9 | v1 | 0.2 | Aidan Holm | END_TO_END_INFERENCE_NUMERIC_MISMATCH | End-to-end inference numeric mismatch vs reference. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.SOHU.INFERENCE | SohuLlama8bFp8PrefillTestCaseSohuLlama8bFp8ForwardTestCaseSohuLlamaForwardIteratedTestCaseKayakTestCase | 1X Module Test | ETCH-34695: SohuLlama8bFp8PrefillTestCase failures | supercomputing-sw | harness 2026.216 |
TH-SOHU-0019-S2Q9 | v1 | 0.2 | James Chou | FLR_FAILED | A Function Level Reset cycle did not complete cleanly: the reset request failed or went unanswered within flr_timeout_s (default 5.0s), or the device did not become available and pingable again within recovery_timeout_s (default 10.0s). Reported for the first failing iteration of the FLR campaign; per-iteration round-trip times are recorded as flr_round_trip_* measurements and are not compared against a limit. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.SOHU.PCIE_FLR | SohuFlrTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-SOHU-0020-S2Q9 | v1 | 0.2 | Ulysses Kao | SAU_PLL_CONFIG_FAIL | SAU/SA PLL configuration failure - firmware did not report a safe SAU PLL phase alignment after FLR, or a requested PLL glitch frequency did not read back within 5% of the target before the readback timeout. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | MLT.SOHU.SOHU_CHIP | SohuSauPllPhaseAlignmentTestCaseSetPllGlitchFreqTestCase | 1X Module Test | ETCH-35301: SohuVfioPingTestCase failures | supercomputing-sw | harness 2026.216 |
TH-SOHU-0021-S2Q9 | v1 | 0.2 | Jack Chou | DEFECTIVE_SA_COLUMNS_EXCEED_LIMIT | Defective SA column count exceeds the screening limit. Retry is futile for a defect map. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.SOHU.SA | SohuSaScreeningTestCaseSaSortRampupHbmSingleChipTestCase | 1X Module Test | ETCH-35581: SohuSaScreeningTestCase failures | supercomputing-sw | harness 2026.216 |
TH-SOHU-0022-S2Q9 | v1 | 0.2 | Jack Chou | PD_CHAIN_READ_FAILURE | Process detector (PD) chain read failure. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.SOHU.PD | SohuPdTestCase | 1X Module Test | ETCH-33887: SohuPdTestCase failure | supercomputing-sw | harness 2026.216 |
TH-SOHU-0023-S2Q9 | v1 | 0.2 | Ulysses Kao | CHIP_BIN_BELOW_THRESHOLD | Chip bin below the acceptance threshold / failed the binning criteria (e.g. unbinnable SA column mask count, or illegal bin transition). | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.SOHU.BINNING | SohuChipBinTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-SOHU-0024-S5Q9 | v1 | 0.2 | Ulysses Kao | SCREENING_DATA_MISSING | Required screening data missing or failed to load (SA mask / bin inputs), including eFuse/Flash init-config read failures and invalid screening bin encodings. | S5 CONFIG | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Config | 1X.SOHU.SCREENING_DATA | SohuChipBinTestCaseSohuSaScreeningTestCaseSaSortRampupHbmSingleChipTestCase | 1X Module Test | ETCH-34698 | supercomputing-sw | harness 2026.216 |
TH-SOHU-0025-S4Q9 | v1 | 0.2 | Ulysses Kao | CHIP_STATE_SNAPSHOT_FAILED | Chip state snapshot capture failed. | S4 TRIAGE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.SOHU.SNAPSHOT | SohuSnapTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-SOHU-0026-S2Q9 retired | v1 | 0.2 | Ulysses Kao | PLL_GLITCH_FREQUENCY_READBACK_FAILED | RETIRED (ETCH-38346): historically emitted when PLL glitch frequency did not read back within 5% of the target before the timeout. New emissions use TH-SOHU-0020 (SAU_PLL_CONFIG_FAIL). | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.SOHU.PLL | SetPllGlitchFreqTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-SOHU-0027-S2Q9 | v1 | 0.2 | Ulysses Kao | VFIO_PING_LATENCY_OUT_OF_SPEC | VFIO ping responses were received but none met the allowed latency - the device is pingable but its response latency is out of spec. Distinct from TH-SOHU-0002 (no response at all). | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.SOHU.ASIC | SohuVfioPingTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-THM-0001-S2Q6 | v1 | 0.2 | Ulysses Kao | THERMAL_READING_OUT_OF_RANGE | Thermal diode temperature reading out of the qualified range, reported invalid by the sensor, or an uncalibrated mezzanine diode reading below 0 C (bad solder joint indicator). Failing channels go in the message / affected_instances, never into new codes (spec §1.2). | S2 ISOLATE | Q6 RETRY | Inspect the IBC TIM and ensure that the Sohu module is properly seated and secured in the fixture. Also verify that the Sohu module top cold plate is connected to the HRM cooling loop. If there is no trouble found, retest once. | Hardware | MLT.THM.SOHU_CHIP | SohuThermalDiodeTestCaseSohuDtsTestCase | 1X Module Test | ETCH-35857: MLT DTS faiure Peak-to-peak deviation: 33029224 uC is out of bounds (30000000) | supercomputing-sw | harness 2026.216 |
TH-THM-0002-S2Q6 | v1 | 0.2 | Sam Jiang | DTS_SENSOR_VARIATION_EXCEEDED | The peak-to-peak spread across the usable Sohu DTS readings exceeded the configured deviation limit (20C at L10). Zeroed and faulted sensors are excluded from the spread, so this reflects real sensor-to-sensor variation. | S2 ISOLATE | Q6 RETRY | Inspect the IBC TIM and ensure that the Sohu module is properly seated and secured in the fixture. Also verify that the Sohu module top cold plate is connected to the HRM cooling loop. If there is no trouble found, retest once. | Hardware | 1X.THM.DTS | SohuDtsTestCase | 1X Module Test | ETCH-35857: MLT DTS faiure Peak-to-peak deviation: 33029224 uC is out of bounds (30000000) | supercomputing-sw | harness 2026.216 |
TH-THM-0003-S2Q9 | v1 | 0.2 | Sam Jiang | CTS_PROTECTION_NO_SHUTDOWN | The CTS trim code was driven to its maximum (lowest trip threshold) but the chip did not shut down: the CPLD cattrip latch bit in ASIC_CTRL1 was never set within the shutdown timeout. | S2 ISOLATE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.THM.CTS | SohuCtsTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-THM-0004-S2Q6 | v1 | 0.2 | Jack Chou | CATASTROPHIC_TRIP_DURING_STRESS | Catastrophic trip event during a stress workload. | S2 ISOLATE | Q6 RETRY | Inspect the IBC TIM and ensure that the Sohu module is properly seated and secured in the fixture. Also verify that the Sohu module top cold plate is connected to the HRM cooling loop. If there is no trouble found, retest once. | Hardware | 1X.THM.TRIP | SohuPowerVirusThermalTestCase | 1X Module Test | ETCH-35548: SohuPowerVirusThermalTestCase failures | supercomputing-sw | harness 2026.216 |
TH-THM-0005-S4Q9 | v1 | 0.2 | Sam Jiang | DTS_PTP_DEVIATION_OUT_OF_BOUNDS | DTS peak-to-peak deviation under power virus out of bounds (threshold under review; bounds may be raised). | S4 TRIAGE | Q9 ESCALATE | If any temperature reading is out of spec verify that the affected unit has completed the required 45-min TIM baking process. Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.THM.DTS | SohuDtsTestCaseSohuPowerVirusTestCase | 1X Module Test | ETCH-35857: MLT DTS faiure Peak-to-peak deviation: 33029224 uC is out of bounds (30000000) | supercomputing-sw | harness 2026.216 |
TH-THM-0006-S2Q9 | v1 | 0.2 | Jack Chou | DTS_PEAK_TEMPERATURE_EXCEEDS_MAX | Peak DTS temperature under power virus exceeds the expected maximum. | S2 ISOLATE | Q9 ESCALATE | If any temperature reading is out of spec verify that the affected unit has completed the required 45-min TIM baking process. Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | 1X.THM.DTS | SohuPowerVirusThermalTestCase | 1X Module Test | ETCH-34688: SohuPowerVirusThermalTestCase failures | supercomputing-sw | harness 2026.216 |
TH-THM-0007-S4Q9 | v1 | 0.2 | Ulysses Kao | THERMAL_TELEMETRY_UNAVAILABLE | The TMP464 thermal diode data could not be obtained - RPC/connection failure over the VBB UART, or a malformed response (missing temperature array, unexpected channel count). The thermal verdict is inconclusive. | S4 TRIAGE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Software | MLT.THM.SOHU_CHIP | SohuThermalDiodeTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-THM-0017-S2Q9 | v1 | 0.4 | Mykola Servetnyk | CPLD_CATTRIP_DETECTED | The mezzanine CPLD reports a cattrip: the MCU cattrip latch is set, MCU cattrip is live with VCC_1V2 good, or an HBM cattrip channel is active. Emitted by CpldDiagnosticsTestCase when run with fail_on_cattrip, so the verdict names the thermal event instead of the workload that died under it. | S2 ISOLATE | Q9 ESCALATE | If every slotted chip tripped, stop and check the station cooling (CDU status, coolant flow, HRM loop) before any retest. For a single chip, inspect the IBC TIM and reseat the module in the fixture; confirm the top cold plate is on the HRM loop. Escalate to Etched engineer with the CPLD reports and collector data. | Hardware | 1X.THM.TRIP | CpldDiagnosticsTestCase | MLT / 1X Module Test | — | supercomputing-sw | harness 2026.253 |
Refreshed from L10/l10_th_registry.yaml, with the packed 32-bit integer form. 228 codes.
| Error Code ID | Packed | Ver | Doc Rev | Original Author | Name | Message | Severity | Quick Action | Recover / Troubleshooting Procedure | Error Type | Component | Test Case | Source | Bugs | Owner | Since |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
TH-BIOS-0001-S5Q9 | 358154241 | v1 | 0.3 | Ulysses Kao | BIOS_EXPECTED_CONFIG_UNAVAILABLE | Expected BIOS version or boot-order configuration could not be loaded; no BIOS verdict is available. | S5 CONFIG | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Config | L10.BIOS.BIOS | CheckBiosVersionCheckBiosBootOrder | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BIOS-0002-S4Q9 | 390660098 | v1 | 0.3 | Ulysses Kao | BIOS_VERSION_QUERY_FAILED | The running BIOS version could not be queried. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.BIOS.BIOS | CheckBiosVersion | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BIOS-0003-S4Q9 | 390660099 | v1 | 0.3 | Ulysses Kao | BIOS_VERSION_OUTPUT_INVALID | BIOS version output was empty, incomplete, or could not be parsed. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.BIOS.BIOS | CheckBiosVersion | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BIOS-0004-S5Q9 | 358154244 | v1 | 0.3 | Ulysses Kao | BIOS_VERSION_MISMATCH | The running BIOS version does not match the expected L10 release. | S5 CONFIG | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.BIOS.BIOS | CheckBiosVersion | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BIOS-0005-S4Q9 | 390660101 | v1 | 0.3 | Ulysses Kao | BIOS_BOOT_ORDER_QUERY_FAILED | The configured BIOS boot order could not be queried. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.BIOS.BIOS | CheckBiosBootOrder | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BIOS-0006-S5Q9 | 358154246 | v1 | 0.3 | Ulysses Kao | BIOS_BOOT_ORDER_MISMATCH | The BIOS boot order does not match the required L10 configuration. | S5 CONFIG | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.BIOS.BIOS | CheckBiosBootOrder | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BIOS-0007-S4Q9 | 357105671 | v1 | 0.3 | Ulysses Kao | BIOS_UPDATE_FAILED | The server BIOS update command failed or did not complete successfully. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.BIOS.BIOS | SohuServerBiosUpdate | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BIOS-0008-S4Q9 | 357105672 | v1 | 0.3 | Ulysses Kao | BIOS_POST_UPDATE_VERIFY_FAILED | The BIOS did not report the requested version after update and reboot. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.BIOS.BIOS | SohuServerBiosUpdate | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BIOS-0009-S4Q9 | 357105673 | v1 | 0.4 | Ulysses Kao | BIOS_BOOT_ORDER_UPDATE_FAILED | Staging the required BIOS boot order via the BMC failed. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.BIOS.BIOS | CheckBiosBootOrder | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BMC-0001-S5Q9 | 358154241 | v1 | 0.3 | Ulysses Kao | BMC_EXPECTED_CONFIG_UNAVAILABLE | Expected BMC/product configuration could not be loaded; no BMC comparison verdict is available. | S5 CONFIG | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Config | L10.BMC.BMC | BmcCheckBmcCheckFanLocationBmcCheckFanControlBmcRecovery | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BMC-0002-S4Q9 | 390660098 | v1 | 0.3 | Ulysses Kao | BMC_INVENTORY_QUERY_FAILED | The BMC inventory or management query command failed. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.BMC.BMC | BmcCheck | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BMC-0003-S4Q9 | 390660099 | v1 | 0.3 | Ulysses Kao | BMC_INVENTORY_OUTPUT_INVALID | BMC inventory output was incomplete or could not be parsed. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.BMC.BMC | BmcCheck | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BMC-0004-S2Q9 | 455671812 | v1 | 0.3 | Ulysses Kao | BMC_IDENTITY_MISMATCH | BMC-reported platform vendor, product, or model identity does not match expected configuration. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Data | L10.BMC.BMC | BmcCheck | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BMC-0005-S2Q9 | 455671813 | v1 | 0.3 | Ulysses Kao | BMC_MANAGEMENT_ADDRESS_MISMATCH | The BMC management IP address or MAC address does not match the expected server configuration. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Data | L10.BMC.BMC | BmcCheck | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BMC-0006-S5Q9 | 358154246 | v1 | 0.3 | Ulysses Kao | BMC_FIRMWARE_VERSION_MISMATCH | The running BMC firmware version does not match the expected L10 release. | S5 CONFIG | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.BMC.BMC | BmcCheck | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BMC-0007-S4Q9 | 390660103 | v1 | 0.3 | Ulysses Kao | BMC_FAN_LOCATION_PROGRAM_FAILED | The BMC command to program fan locations failed. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Software | L10.BMC.BMC | BmcCheckFanLocation | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BMC-0008-S4Q9 | 390660104 | v1 | 0.3 | Ulysses Kao | BMC_FAN_LOCATION_READBACK_FAILED | Programmed fan-location data could not be read back from the BMC. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.BMC.BMC | BmcCheckFanLocation | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BMC-0009-S5Q9 | 358154249 | v1 | 0.3 | Ulysses Kao | BMC_FAN_CONFIG_UNAVAILABLE | BMC fan-zone, PWM, or sensor configuration needed by the fan-control test is missing or invalid. | S5 CONFIG | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Config | L10.BMC.BMC | BmcCheckFanControl | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BMC-0010-S4Q9 | 390660106 | v1 | 0.3 | Ulysses Kao | BMC_SENSOR_QUERY_FAILED | The BMC sensor query failed or returned no usable fan/thermal readings. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.BMC.BMC | BmcCheckFanControl | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BMC-0011-S4Q6 | 289800203 | v1 | 0.3 | Ulysses Kao | BMC_CONNECTIVITY_FAILED | BMC connectivity validation observed packet loss or could not complete the configured reachability check. | S4 TRIAGE | Q6 RETRY | Retry the affected L10 subset once and capture logs from the failing run. If persistent, isolate the unit; attach logs; route to FA; open Jira. | Network | L10.BMC.BMC | BmcConnectivityCheckBmcRecovery | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BMC-0012-S4Q9 | 289996812 | v1 | 0.3 | Ulysses Kao | BMC_SSH_UNAVAILABLE | An SSH session to the BMC could not be established for the requested operation. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Network | L10.BMC.BMC | SohuServerBmcFirmwareUpdateSohuServerBiosUpdatePsuFirmwareUpdateCpldFirmwareUpdateL10VBBUpdateBmcPowerCycle | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BMC-0013-S4Q9 | 390660109 | v1 | 0.3 | Ulysses Kao | BMC_UPDATE_IMAGE_STAGING_FAILED | The firmware image or update configuration could not be staged to the BMC. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.BMC.BMC | SohuServerBmcFirmwareUpdateSohuServerBiosUpdatePsuFirmwareUpdateCpldFirmwareUpdateL10VBBUpdate | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BMC-0014-S4Q9 | 357105678 | v1 | 0.3 | Ulysses Kao | BMC_FIRMWARE_UPDATE_COMMAND_FAILED | The BMC firmware update command failed or did not complete successfully. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.BMC.BMC | SohuServerBmcFirmwareUpdate | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BMC-0015-S4Q9 | 357105679 | v1 | 0.3 | Ulysses Kao | BMC_POST_UPDATE_VERIFY_FAILED | The BMC did not report the requested firmware version after update. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.BMC.BMC | SohuServerBmcFirmwareUpdate | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BMC-0016-S4Q9 | 357105680 | v1 | 0.3 | Ulysses Kao | BMC_POWER_CYCLE_INTERVAL_SET_FAILED | The requested BMC power-cycle interval could not be configured. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Config | L10.BMC.BMC | SetPowerCycleInterval | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BMC-0017-S4Q9 | 390660113 | v1 | 0.3 | Ulysses Kao | BMC_SYSTEM_EVENT_LOG_QUERY_FAILED | The BMC system-event log could not be read or parsed. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.BMC.BMC | StoreSystemLogStatesCheckSystemLogStates | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BMC-0018-S4Q9 | 373882898 | v1 | 0.3 | Ulysses Kao | BMC_CRITICAL_EVENT_DETECTED | A new critical BMC/SEL event was detected between the stored baseline and the post-test log state. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Hardware | L10.BMC.BMC | CheckSystemLogStates | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BMC-0019-S4Q9 | 390660115 | v1 | 0.3 | Ulysses Kao | BMC_RAW_COMMAND_FAILED | A required low-level BMC raw command failed or returned no response. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.BMC.BMC | BmcCheckBmcCheckFanLocationBmcCheckFanControl | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-BMC-0020-S5Q9 | 358154260 | v1 | 0.3 | Ulysses Kao | BMC_UPDATE_MODEL_CONFIG_UNAVAILABLE | The BMC update flow could not resolve the server model or target-specific update configuration. | S5 CONFIG | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Config | L10.BMC.BMC | SohuServerBmcFirmwareUpdateSohuServerBiosUpdatePsuFirmwareUpdateCpldFirmwareUpdateL10VBBUpdate | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-C2C-0001-S2Q9 | 371785729 | v1 | 0.3 | Ulysses Kao | C2C_LINKUP_FAILED | C2C lanes failed to train or link up within the timeout: RX equalization did not converge after its configured attempts, or the link-up test reported one or more datalinks/lanes down. | S2 ISOLATE | Q9 ESCALATE | Retry the C2C subset up to three times in loopback mode with the affected lanes. Reseat the loopback fixture/cable per station SOP; confirm the FIR sweep ran before link-up. Isolate the module; attach the link-up PoR summary and per-lane data; route to FA; open Jira. | CATEGORY_HARDWARE | L10.C2C.SOHU_SERDES | C2cLinkupMultiChipTestCaseC2cThroughputMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-C2C-0004-S4Q9 | 373882884 | v1 | 0.3 | Ulysses Kao | C2C_LANE_MARGIN_BELOW_THRESHOLD | Worst per-lane minimum BER from the eye-diagram sweep exceeded the configured ber_threshold; the SERDES margin spec is not final. | S4 TRIAGE | Q9 ESCALATE | Retry the lane-margin test up to three times; record the per-lane BER breakdown for each run. Route to SIVAL/platform triage with the per-lane BER matrices; do not isolate the module on this code alone until the SERDES margin spec is final. | CATEGORY_HARDWARE | L10.C2C.SOHU_SERDES | C2cFirCdrSweepMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-C2C-0005-S2Q9 | 455671813 | v1 | 0.3 | Ulysses Kao | C2C_PRBS_ERROR_THRESHOLD_EXCEEDED | The C2C PRBS run completed but the observed bit-error count or rate exceeded the configured acceptance limit. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | CATEGORY_DATA | L10.C2C.SOHU_SERDES | C2cPrbsMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-C2C-0006-S4Q9 | 390660102 | v1 | 0.3 | Ulysses Kao | C2C_SWEEP_OUTPUT_INVALID | The FIR/CDR or PRBS test did not produce complete, parseable lane results, so the C2C verdict is inconclusive. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.C2C.SOHU_SERDES | C2cFirCdrSweepMultiChipTestCaseC2cPrbsMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-C2C-0007-S2Q9 | 455671815 | v1 | 0.4 | Ulysses Kao | C2C_INTEGRITY_MISMATCH | Bit-by-bit C2C loopback comparison found a data mismatch on at least one datalink/lane. | S2 ISOLATE | Q9 ESCALATE | Capture the per-lane error plots BEFORE retrying, then retry the C2C subset up to three times. Verify the loopback lane map matches the fixture wiring; reseat the fixture per station SOP. Isolate the module; attach the per-lane mismatch data; route to FA; open Jira. | Data | L10.C2C.SOHU_SERDES | C2cIntegrityMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-C2C-0008-S2Q9 | 371785736 | v1 | 0.4 | Ulysses Kao | C2C_THROUGHPUT_BELOW_THRESHOLD | The large C2C transfer did not meet its acceptance criteria: measured throughput below the configured percentage of line rate (90% in the 1X suite), or the post-FEC uncorrected error fraction above its limit. | S2 ISOLATE | Q9 ESCALATE | Retry the C2C subset up to three times with the affected mode and lanes. Reseat the loopback fixture; confirm the CSRAM PLL boost applied during the transfer. Isolate the module; attach throughput and post-FEC counters; route to FA; open Jira. | Hardware | L10.C2C.SOHU_SERDES | C2cThroughputMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-CPU-0001-S2Q6 | 371589121 | v1 | 0.3 | Ulysses Kao | CPU_STRESS_FAIL | stress-ng CPU stress failed because a stressor/verification error occurred or the tool returned its workload-failure status. | S2 ISOLATE | Q6 RETRY | Verify station ambient/airflow nominal, rerun CPU subset. Inspect heatsink mounting and TIM per assembly SOP; remount if out of spec. Preserve unit; attach stress-ng logs + MCE excerpt; route to FA; open Jira. | CATEGORY_HARDWARE | L10.CPU.HOST_CPU | CpuStress | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-CPU-0003-S5Q9 | 391708675 | v1 | 0.3 | Ulysses Kao | CPU_STRESS_TOOL_UNAVAILABLE | stress-ng is not installed or cannot be executed on the target; no CPU verdict is available (ERROR disposition; excluded from DUT yield). | S5 CONFIG | Q9 ESCALATE | Install stress-ng on the target (or rerun the package-install step), rerun CPU subset. If provisioning image is missing the package, route to test-infra; open Jira. Do NOT route the unit to rework or FA. | CATEGORY_SOFTWARE | L10.CPU.HOST_CPU | CpuStress | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-CPU-0004-S4Q6 | 390463492 | v1 | 0.3 | Ulysses Kao | CPU_STRESS_EXECUTION_ERROR | stress-ng returned an execution, initialization, option, resource, unsupported-stressor, or signal error rather than a CPU workload verdict. | S4 TRIAGE | Q6 RETRY | Confirm no other workload active on the DUT, rerun CPU subset. Route logs to engineering triage with the stress-ng exit code and dmesg excerpt. | CATEGORY_SOFTWARE | L10.CPU.HOST_CPU | CpuStress | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-CPU-0005-S4Q9 | 390660101 | v1 | 0.3 | Ulysses Kao | CPU_INVENTORY_QUERY_FAILED | The CPU inventory command failed; no CPU BOM verdict is available. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.CPU.HOST_CPU | CheckCpu | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-CPU-0006-S4Q9 | 390660102 | v1 | 0.3 | Ulysses Kao | CPU_INVENTORY_OUTPUT_INVALID | The CPU inventory output was malformed or could not be parsed into processor records. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.CPU.HOST_CPU | CheckCpu | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-CPU-0007-S2Q9 | 371785735 | v1 | 0.3 | Ulysses Kao | CPU_COUNT_MISMATCH | The discovered processor count does not match the expected L10 BOM. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Hardware | L10.CPU.HOST_CPU | CheckCpu | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-CPU-0008-S2Q9 | 455671816 | v1 | 0.3 | Ulysses Kao | CPU_IDENTITY_MISMATCH | A processor part number or manufacturer does not match the expected L10 BOM. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Data | L10.CPU.HOST_CPU | CheckCpu | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-CPU-0009-S2Q9 | 371785737 | v1 | 0.3 | Ulysses Kao | CPU_TOPOLOGY_MISMATCH | A processor core count or thread count does not match the expected L10 BOM. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Hardware | L10.CPU.HOST_CPU | CheckCpu | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-CPU-0010-S4Q9 | 357105674 | v1 | 0.3 | Ulysses Kao | CPU_CLOCK_MISMATCH | A processor base or maximum clock reported by inventory does not match the expected L10 BOM. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.CPU.HOST_CPU | CheckCpu | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-DISK-0001-S4Q9 | 390660097 | v1 | 0.3 | Ulysses Kao | DISK_INVENTORY_QUERY_FAILED | The block-device or NVMe inventory command failed; no storage verdict is available. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.DISK.STORAGE | CheckDiskDiskFormatDiskSmartDiskStress | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-DISK-0002-S4Q9 | 390660098 | v1 | 0.3 | Ulysses Kao | DISK_INVENTORY_OUTPUT_INVALID | Block-device or NVMe inventory output was malformed or could not be parsed. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.DISK.STORAGE | CheckDiskDiskFormatDiskStress | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-DISK-0003-S2Q9 | 304676867 | v1 | 0.3 | Ulysses Kao | NO_DISKS_DETECTED | No storage devices were detected although the L10 BOM requires disks. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Storage | L10.DISK.STORAGE | CheckDiskDiskSmart | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-DISK-0004-S2Q9 | 304676868 | v1 | 0.3 | Ulysses Kao | DISK_COUNT_MISMATCH | The discovered disk count does not match the expected L10 BOM. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Storage | L10.DISK.STORAGE | CheckDisk | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-DISK-0005-S2Q9 | 455671813 | v1 | 0.3 | Ulysses Kao | DISK_IDENTITY_OR_CAPACITY_MISMATCH | A disk model, serial number, namespace, or capacity does not match the expected L10 BOM. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Data | L10.DISK.STORAGE | CheckDisk | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-DISK-0006-S4Q9 | 357105670 | v1 | 0.3 | Ulysses Kao | MIXED_DRIVE_CONFIGURATION | Installed drives do not satisfy the required homogeneous model/capacity configuration. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.DISK.STORAGE | CheckDisk | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-DISK-0007-S5Q9 | 358154247 | v1 | 0.3 | Ulysses Kao | BOOT_DEVICE_UNDETERMINED | The test could not reliably identify the host boot device and therefore could not safely select data drives. | S5 CONFIG | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Config | L10.DISK.STORAGE | DiskFormatDiskStress | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-DISK-0008-S5Q9 | 358154248 | v1 | 0.3 | Ulysses Kao | NO_TESTABLE_NVME_DEVICES | No eligible non-boot NVMe device remained after discovery and configured exclusions. | S5 CONFIG | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Config | L10.DISK.STORAGE | DiskFormatDiskSmartDiskStress | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-DISK-0009-S2Q9 | 304676873 | v1 | 0.3 | Ulysses Kao | NVME_NAMESPACE_MISSING | An expected NVMe controller has no usable namespace and automatic formatting is disabled. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Storage | L10.DISK.NVME | DiskFormat | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-DISK-0010-S4Q9 | 306774026 | v1 | 0.3 | Ulysses Kao | NVME_FORMAT_FAILED | The NVMe format or namespace creation command failed. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Storage | L10.DISK.NVME | DiskFormat | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-DISK-0011-S2Q9 | 304676875 | v1 | 0.3 | Ulysses Kao | NVME_NAMESPACE_NOT_CREATED | The expected NVMe namespace was still absent after the format/namespace operation completed. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Storage | L10.DISK.NVME | DiskFormat | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-DISK-0012-S2Q9 | 304676876 | v1 | 0.3 | Ulysses Kao | NVME_SMART_SELF_TEST_FAILED | An NVMe device self-test completed with a failure or did not complete successfully. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Storage | L10.DISK.NVME | DiskSmart | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-DISK-0013-S4Q9 | 390660109 | v1 | 0.3 | Ulysses Kao | NVME_SMART_DATA_UNAVAILABLE | SMART/health data could not be read for an eligible NVMe device. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.DISK.NVME | DiskSmart | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-DISK-0014-S2Q9 | 304676878 | v1 | 0.3 | Ulysses Kao | NVME_SMART_CRITICAL_WARNING | The NVMe SMART critical-warning field is non-zero. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Storage | L10.DISK.NVME | DiskSmart | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-DISK-0015-S2Q9 | 304676879 | v1 | 0.3 | Ulysses Kao | NVME_AVAILABLE_SPARE_BELOW_THRESHOLD | NVMe available spare is below the device or configured threshold. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Storage | L10.DISK.NVME | DiskSmart | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-DISK-0016-S2Q9 | 304676880 | v1 | 0.3 | Ulysses Kao | NVME_MEDIA_ERRORS_DETECTED | NVMe SMART reports one or more media/data-integrity errors. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Storage | L10.DISK.NVME | DiskSmart | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-DISK-0017-S4Q9 | 373882897 | v1 | 0.3 | Ulysses Kao | NVME_TEMPERATURE_OUT_OF_RANGE | NVMe temperature is outside the configured safe operating range. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Hardware | L10.DISK.NVME | DiskSmart | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-DISK-0018-S2Q9 | 304676882 | v1 | 0.3 | Ulysses Kao | NVME_WEAR_LIMIT_EXCEEDED | NVMe percentage-used or wear indicator exceeds the configured acceptance limit. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Storage | L10.DISK.NVME | DiskSmart | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-DISK-0019-S2Q6 | 304480275 | v1 | 0.3 | Ulysses Kao | DISK_STRESS_FIO_FAILED | The fio disk stress workload returned a workload-failure status for an eligible drive. | S2 ISOLATE | Q6 RETRY | Retry the affected L10 subset once and capture logs from the failing run. If persistent, isolate the unit; attach logs; route to FA; open Jira. | Storage | L10.DISK.STORAGE | DiskStress | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-DISK-0020-S4Q9 | 390660116 | v1 | 0.3 | Ulysses Kao | DISK_STRESS_OUTPUT_INVALID | fio output was missing or could not be parsed into valid per-drive results. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.DISK.STORAGE | DiskStress | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-DMA-0002-S2Q6 | 371589122 | v1 | 0.4 | Ulysses Kao | DMA_DATA_PATH_TEST_FAILED | Host-chip DMA data-path test failure | S2 ISOLATE | Q6 RETRY | Rerun the DMA subset up to 3 times; reseat the module if failures persist. Preserve unit; attach dma_test_app output and seed; isolate the module and route to triage. | Hardware | L10.DMA.HOST_CHIP_PATH | SohuDmaMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-ENV-0005-S4Q9 | 289996805 | v1 | 0.3 | Ulysses Kao | MODEL_REGISTRY_MOUNT_FAILED | The model-registry NFS share could not be mounted or validated on the L10 host. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Network | L10.ENV.MODEL_REGISTRY | ModelRegistryNfsMountTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-ENV-0006-S5Q9 | 458817542 | v1 | 0.3 | Ulysses Kao | MODEL_ARTIFACT_UNAVAILABLE | A required model artifact, Hugging Face cache, or model path is missing or unreadable. | S5 CONFIG | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Data | L10.ENV.MODEL_REGISTRY | StageModelRegistryToShmTestCaseSohuMlpTp8TestCaseSohuLlama70bForwardIteratedTestCaseSohuLlama70bHp8ForwardIteratedTestCaseSohuLlamaHp2Up4CachedInferenceMaxTestCaseSohuLlamaHp2Up4NospecInferenceMaxTestCaseSohuLlamaHp8CachedInferenceMaxTestCaseLlama8bFp8Tp1SmokeTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-ENV-0007-S4Q9 | 424214535 | v1 | 0.3 | Ulysses Kao | MODEL_STAGE_TO_SHM_FAILED | The required model artifact could not be copied or staged into shared memory. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Resource | L10.ENV.SHM | StageModelRegistryToShmTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-ENV-0008-S4Q9 | 424214536 | v1 | 0.3 | Ulysses Kao | MODEL_SHM_CLEANUP_FAILED | The staged model could not be removed from shared memory during cleanup. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Resource | L10.ENV.SHM | FreeModelRegistryFromShmTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-ENV-0009-S4Q9 | 289996809 | v1 | 0.4 | Ulysses Kao | HOST_OR_BMC_UNREACHABLE | The station host or its BMC did not answer ping or SSH after boot, so no module test could start. | S4 TRIAGE | Q9 ESCALATE | Check station power, BMC network and management cabling; confirm the host answers ping and ssh. Route to test-infra with the ping/ssh transcript. Do NOT route any module to rework or FA on this code. | Network | L10.ENV.STATION | SohuHostRebootTestCaseBmcPowerCycleBmcConnectivityCheck | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-ENV-0010-S4Q9 | 357105674 | v1 | 0.4 | Ulysses Kao | PLX_SWITCH_DISCOVERY_FAILED | PLX daemon reported no PEX switches | S4 TRIAGE | Q9 ESCALATE | Check and update the VBB MCU firmware, then rerun the affected test. Escalate to Etched if no PEX switches are reported after the firmware check. | Config | L10.ENV.STATION | CheckPlxDaemonFirmwareVersionUpdatePlxDaemonFirmwareVersion | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-ENV-0011-S4Q9 | 357105675 | v1 | 0.4 | Ulysses Kao | HOST_DRIVER_SETUP_FAILED | A host driver/setup utility step failed: kernel module reload, PCIe teardown or bus rescan, sysfs parameter write, or log-reader activation returned non-zero. | S4 TRIAGE | Q9 ESCALATE | Fix station host environment (reload the driver, restore permissions) and rerun the setup subset. Route to test-infra with the command output and lsmod/dmesg excerpt. Do NOT route the unit to rework or FA. | Config | L10.ENV.STATION | SoftwareProvisioningBroadcomUpdateFwAndConfigure | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-ENV-0012-S4Q9 | 424214540 | v1 | 0.4 | Ulysses Kao | WEIGHTS_STAGE_TO_DISK_FAILED | The requested model weights could not be mirrored onto the server's local disk, or the destination lacked the free space to hold them. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Resource | L10.ENV.WEIGHTS_CACHE | StageWeightsTestCase | L10 Test | — | supercomputing-sw | harness 2026.253 |
TH-ENV-0013-S4Q9 | 424214541 | v1 | 0.4 | Ulysses Kao | WEIGHTS_DISK_CLEANUP_FAILED | The staged model weights could not be removed from the server's local disk during cleanup. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Resource | L10.ENV.WEIGHTS_CACHE | FreeWeightsTestCase | L10 Test | — | supercomputing-sw | harness 2026.253 |
TH-FW-0001-S2Q9 | 355008513 | v1 | 0.3 | Ulysses Kao | SOHU_FIRMWARE_UPDATE_FAILED | Sohu app firmware update failed or ASIC not pingable after update | S2 ISOLATE | Q9 ESCALATE | Capture the UART log for the update window; verify the staged image package matches the release. Escalate to firmware with the UART log and the image package release id; do not retry blind. | CATEGORY_CONFIG | L10.FW.FIRMWARE | MultiChipSohuHotReloadFirmwareUpdateTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0005-S5Q2 | 357695493 | v1 | 0.3 | Ulysses Kao | FIXTURE_FIRMWARE_UPDATE_FAILED | Fixture firmware update failed (golden VBB / retimer): OpenOCD flash, ST-Link adapter, post-flash UART/RPC ready check, or application firmware-hash verification did not succeed. | S5 CONFIG | Q2 WARM_REBOOT | Halt testing and escalate to Etched. | CATEGORY_CONFIG | L10.FW.FIRMWARE | L10VBBUpdate | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0010-S5Q9 | 358154250 | v1 | 0.3 | Ulysses Kao | EXPECTED_VERSION_CONFIG_UNAVAILABLE | Expected host or component version data could not be loaded; no version verdict is available. | S5 CONFIG | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Config | L10.FW.FIRMWARE | CheckOsCheckKernelVersionCheckSohuFwVersionCheckPsuFirmwareVersionCheckCpldBpFirmwareVersion | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0011-S4Q9 | 390660107 | v1 | 0.3 | Ulysses Kao | HOST_OS_VERSION_QUERY_FAILED | The host operating-system release could not be queried. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.FW.HOST_OS | CheckOs | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0012-S4Q9 | 390660108 | v1 | 0.3 | Ulysses Kao | HOST_OS_VERSION_OUTPUT_INVALID | Operating-system release output was incomplete or could not be parsed. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.FW.HOST_OS | CheckOs | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0013-S5Q9 | 358154253 | v1 | 0.3 | Ulysses Kao | HOST_OS_VERSION_MISMATCH | The host operating-system release does not match the required L10 image. | S5 CONFIG | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.FW.HOST_OS | CheckOs | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0014-S4Q9 | 390660110 | v1 | 0.3 | Ulysses Kao | HOST_KERNEL_VERSION_QUERY_FAILED | The running host kernel version could not be queried. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.FW.HOST_KERNEL | CheckKernelVersion | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0015-S4Q9 | 390660111 | v1 | 0.3 | Ulysses Kao | HOST_KERNEL_VERSION_OUTPUT_INVALID | Kernel version output was empty or could not be parsed. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.FW.HOST_KERNEL | CheckKernelVersion | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0016-S5Q9 | 358154256 | v1 | 0.3 | Ulysses Kao | HOST_KERNEL_VERSION_MISMATCH | The running host kernel does not match the required L10 release. | S5 CONFIG | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.FW.HOST_KERNEL | CheckKernelVersion | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0017-S4Q9 | 390660113 | v1 | 0.3 | Ulysses Kao | SOHU_FIRMWARE_VERSION_QUERY_FAILED | The running Sohu firmware version could not be queried. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.FW.SOHU | CheckSohuFwVersion | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0018-S5Q9 | 358154258 | v1 | 0.3 | Ulysses Kao | SOHU_FIRMWARE_VERSION_MISMATCH | The running Sohu firmware version does not match the required L10 release. | S5 CONFIG | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | CATEGORY_CONFIG | L10.FW.SOHU | CheckSohuFwVersion | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0019-S4Q9 | 390660115 | v1 | 0.3 | Ulysses Kao | PSU_FIRMWARE_VERSION_QUERY_FAILED | One or more PSU firmware versions could not be queried. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.FW.PSU | CheckPsuFirmwareVersion | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0020-S5Q9 | 358154260 | v1 | 0.3 | Ulysses Kao | PSU_FIRMWARE_VERSION_MISMATCH | A PSU firmware version does not match the required L10 release. | S5 CONFIG | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.FW.PSU | CheckPsuFirmwareVersion | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0021-S4Q9 | 390660117 | v1 | 0.3 | Ulysses Kao | CPLD_BP_FIRMWARE_VERSION_QUERY_FAILED | The backplane/CPLD firmware version could not be queried. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.FW.CPLD | CheckCpldBpFirmwareVersion | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0022-S5Q9 | 358154262 | v1 | 0.3 | Ulysses Kao | CPLD_BP_FIRMWARE_VERSION_MISMATCH | The backplane/CPLD firmware version does not match the required L10 release. | S5 CONFIG | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.FW.CPLD | CheckCpldBpFirmwareVersion | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0023-S4Q9 | 357105687 | v1 | 0.3 | Ulysses Kao | PSU_FIRMWARE_UPDATE_FAILED | The PSU firmware update command failed or did not complete successfully. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.FW.PSU | PsuFirmwareUpdate | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0024-S4Q9 | 357105688 | v1 | 0.3 | Ulysses Kao | PSU_POST_UPDATE_VERIFY_FAILED | A PSU did not report the requested firmware version after update. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.FW.PSU | PsuFirmwareUpdate | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0025-S4Q9 | 357105689 | v1 | 0.3 | Ulysses Kao | CPLD_FIRMWARE_UPDATE_FAILED | The CPLD/backplane firmware update command failed or did not complete successfully. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.FW.CPLD | CpldFirmwareUpdate | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0026-S4Q9 | 357105690 | v1 | 0.3 | Ulysses Kao | CPLD_POST_UPDATE_VERIFY_FAILED | The CPLD/backplane did not report the requested firmware version after update. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.FW.CPLD | CpldFirmwareUpdate | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0027-S4Q9 | 357105691 | v1 | 0.3 | Ulysses Kao | SOFTWARE_PROVISIONING_FAILED | The L10 host software provisioning workflow failed to apply the required software state. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.FW.HOST_SW | SoftwareProvisioning | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0028-S4Q9 | 390660124 | v1 | 0.3 | Ulysses Kao | LINUX_PACKAGE_INSTALL_FAILED | A required Linux package could not be installed or configured. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Software | L10.FW.HOST_SW | InstallLinuxPackagesTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0029-S4Q9 | 357105693 | v1 | 0.3 | Ulysses Kao | SOHU_BOOTLOADER_UPDATE_FAILED | The Sohu bootloader update command failed or the device was not usable after programming. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.FW.SOHU_BOOTLOADER | UpdateSohuBootloader | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0030-S4Q9 | 357105694 | v1 | 0.3 | Ulysses Kao | SOHU_BOOTLOADER_POST_UPDATE_VERIFY_FAILED | The running Sohu bootloader does not match the requested image after update. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.FW.SOHU_BOOTLOADER | UpdateSohuBootloader | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0031-S4Q9 | 390660127 | v1 | 0.3 | Ulysses Kao | HOST_SYSTEM_LOG_CRITICAL_EVENT_DETECTED | A new critical host kernel, service, or system event was detected after the L10 workload. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Software | L10.FW.HOST_SW | CheckSystemLogStates | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0032-S4Q9 | 357105696 | v1 | 0.4 | Ulysses Kao | SOHU_FIRMWARE_HASH_MISMATCH | The ASIC booted application firmware after the update, but the running image hash does not match the hash in the image package that was just staged. | S4 TRIAGE | Q9 ESCALATE | Compare the staged image package against the intended release package; confirm the station is serving the right build. Route to firmware/test-infra triage with both hashes and the package release id. | Config | L10.FW.FIRMWARE | MultiChipSohuHotReloadFirmwareUpdateTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0033-S5Q9 | 358154273 | v1 | 0.4 | Ulysses Kao | DVFS_PROFILE_APPLY_FAILED | Board DVFS profile detection or application failed | S5 CONFIG | Q9 ESCALATE | Verify the board ID and profile tables. Escalate if the profile is missing for this board revision. | Config | L10.FW.FIRMWARE | SohuSetClockProfileMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0034-S5Q9 | 358154274 | v1 | 0.4 | Ulysses Kao | SOHU_HOT_RELOAD_INELIGIBLE | Hot reload could not start, so no firmware was written | S5 CONFIG | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Config | L10.FW.FIRMWARE | MultiChipSohuHotReloadFirmwareUpdateTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-FW-0035-S4Q9 | 390660131 | v1 | 0.4 | Ulysses Kao | SOHU_SECURITY_STATE_QUERY_FAILED | The running Sohu firmware did not report its security mode or TPM state, so the required provisioning could not be judged. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.FW.SOHU | CheckSohuSecurityState | L10 Test | — | supercomputing-sw | harness 2026.260 |
TH-FW-0036-S2Q9 | 438894628 | v1 | 0.4 | Ulysses Kao | SOHU_SECURITY_MODE_MISMATCH | The running Sohu firmware does not enforce the required security mode, so the unit is not running production-locked firmware. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Security | L10.FW.SOHU | CheckSohuSecurityState | L10 Test | — | supercomputing-sw | harness 2026.260 |
TH-FW-0037-S2Q9 | 438894629 | v1 | 0.4 | Ulysses Kao | SOHU_TPM_PROVISIONING_MISMATCH | The Sohu TPM does not carry the required provisioning: its anti-rollback state is not permanent, or debug signing keys are not revoked. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Security | L10.FW.SOHU | CheckSohuSecurityState | L10 Test | — | supercomputing-sw | harness 2026.260 |
TH-FW-0038-S2Q9 | 438894630 | v1 | 0.4 | Ulysses Kao | SOHU_SPDM_ATTESTATION_FAILED | A Sohu did not attest over SPDM: its certificate chain does not validate against the required trust anchor, its measurements do not match the policy, it did not answer, or its attestation report is unusable. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-chip transcript, report and suite URL; route to FA; open Jira. | Security | L10.FW.SOHU | SpdmAttestMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.260 |
TH-FW-0039-S4Q9 | 390660135 | v1 | 0.4 | Ulysses Kao | SOHU_SPDM_ATTESTATION_UNAVAILABLE | The station could not run SPDM attestation for one or more chips: the PCIe MCTP tunnel could not be served or reached, or the verifier could not be launched or its output written. The affected chips were not judged. | S4 TRIAGE | Q9 ESCALATE | Fix the station, proxy deployment, or suite configuration; do NOT route the unit to rework. | Software | L10.HAR.HARNESS | SpdmAttestMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.260 |
TH-FW-0040-S2Q9 | 438894632 | v1 | 0.4 | Ulysses Kao | SOHU_DEBUG_KEY_REVOCATION_FAILED | The root of trust refused or did not apply the debug signing-key revocation, or the revoked mask read back after reset does not cover every debug slot. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Security | L10.FW.SOHU | RevokeSohuDebugKeys | L10 Test | — | supercomputing-sw | harness 2026.265 |
TH-HAR-0001-S4Q0 | 390070273 | v1 | 0.3 | Ulysses Kao | TEST_TIMEOUT | The test exceeded its configured timeout and was terminated before producing a verdict (ABORTED disposition). | S4 TRIAGE | Q0 NO_ACT | Rerun the affected subset once to check reproducibility. Route to engineering triage with the per-test log and timeout configuration. | CATEGORY_SOFTWARE | L10.HAR.HARNESS | — | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HAR-0002-S5Q9 | 358154242 | v1 | 0.3 | Ulysses Kao | TEST_ARGS_INVALID | A test case was invoked with invalid or unsupported arguments and rejected the run before touching the DUT. | S5 CONFIG | Q9 ESCALATE | Fix the suite configuration for this test; do NOT route the unit to rework. | CATEGORY_CONFIG | L10.HAR.HARNESS | CpuStressMemoryStressNgMemoryStressIoStressSohuSnapTestCaseExecBinaryTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HAR-0003-S5Q9 | 358154243 | v1 | 0.3 | Ulysses Kao | TEST_DEPENDENCY_UNAVAILABLE | Tool/binary/data file the test depends on is missing on the test system (test-system ERROR — excluded from module yield) | S5 CONFIG | Q9 ESCALATE | Fix the station deployment / harness package; do NOT route the unit to rework. | CATEGORY_CONFIG | L10.HAR.HARNESS | CpuStressMemoryStressNgMemoryStressIoStressLoopbackTopologyTestNetworkBwRdmaTestModelRegistryNfsMountTestCaseStageModelRegistryToShmTestCaseSohuServerBmcFirmwareUpdateSohuServerBiosUpdatePsuFirmwareUpdateCpldFirmwareUpdateL10VBBUpdateInstallLinuxPackagesTestCaseUpdatePlxDaemonFirmwareVersionBroadcomUpdateFwAndConfigureUpdateSohuBootloaderMultiChipSohuHotReloadFirmwareUpdateTestCaseExecBinaryTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HAR-0004-S4Q9 | 357105668 | v1 | 0.3 | Ulysses Kao | NESTED_SUITE_ORCHESTRATION_FAILED | Nested suite failed to launch or complete orchestration (child test failures carry their own codes) | S4 TRIAGE | Q9 ESCALATE | Check the suite config and resources for the nested wrapper. Route to test-infra; do NOT route the unit to rework or FA. | CATEGORY_CONFIG | L10.HAR.HARNESS | RunInStress | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HAR-0005-S4Q6 | 390463493 | v1 | 0.3 | Ulysses Kao | TEST_TOOL_EXECUTION_FAILED | A test-owned command or binary failed to execute and did not produce a domain-specific DUT verdict. | S4 TRIAGE | Q6 RETRY | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.HAR.HARNESS | IoStressExecBinaryTestCaseCpldDiagnosticsRemoteTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HAR-0006-S4Q9 | 390660102 | v1 | 0.3 | Ulysses Kao | TEST_METRICS_PARSE_FAILED | A test tool completed without complete, parseable metrics needed to determine the result. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.HAR.HARNESS | IoStressExecBinaryTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HAR-0007-S4Q9 | 390660103 | v1 | 0.3 | Ulysses Kao | TEST_OUTPUT_UNAVAILABLE | Expected test output, state, or result artifacts were not produced. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.HAR.HARNESS | StoreSystemLogStatesCheckSystemLogStatesExecBinaryTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HAR-0008-S5Q9 | 358154248 | v1 | 0.3 | Ulysses Kao | TEST_PRECONDITION_NOT_MET | A required test precondition was not satisfied, so the test did not exercise the DUT as intended. | S5 CONFIG | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Config | L10.HAR.HARNESS | RunInStressIoStressNetworkBwRdmaTest | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HAR-0009-S4Q9 | 390660105 | v1 | 0.3 | Ulysses Kao | TEST_CLEANUP_FAILED | Test cleanup failed and may have left processes, mounts, namespaces, files, or device state behind. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.HAR.HARNESS | LoopbackTopologyTestNetworkBwRdmaTestFreeModelRegistryFromShmTestCaseFreeWeightsTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HAR-0010-S4Q9 | 390660106 | v1 | 0.3 | Ulysses Kao | SYSTEM_LOG_SNAPSHOT_FAILED | The pre-test system-log baseline could not be captured. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.HAR.HARNESS | StoreSystemLogStates | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HAR-0011-S4Q9 | 390660107 | v1 | 0.3 | Ulysses Kao | SYSTEM_LOG_COMPARE_FAILED | System-log state could not be queried, parsed, or compared with the stored baseline. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.HAR.HARNESS | CheckSystemLogStates | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HAR-9998-S3Q0 | 271591182 | v1 | 0.3 | Ulysses Kao | TEST_FAILED_UNCLASSIFIED | A test returned TestFailedError without a specific TH identity; this catch-all is a taxonomy-health failure and must be replaced by a specific code. | S3 UNKNOWN | Q0 NO_ACT | Follow the failing test case's own documentation/SOP. Open a taxonomy-gap ticket to allocate a specific code for this failure mode. | CATEGORY_NONE | L10.HAR.HARNESS | — | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HAR-9999-S3Q9 | 389621519 | v1 | 0.3 | Ulysses Kao | UNHANDLED_EXCEPTION | An unhandled exception escaped the test and was wrapped by the harness; the traceback is retained in the per-test log (ERROR disposition). | S3 UNKNOWN | Q9 ESCALATE | Route to the owning test/SW engineer with the traceback; do NOT route the unit to rework or FA. | CATEGORY_SOFTWARE | L10.HAR.HARNESS | — | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HBM-0006-S2Q9 | 321454086 | v1 | 0.3 | Ulysses Kao | HBM_LANE_REPAIR_FAILED | HBM lane repair failed - no passing repair found | S2 ISOLATE | Q9 ESCALATE | Escalate to Etched with the lane-repair map; the stack is unrepairable, so screen the module out rather than reworking it. | CATEGORY_MEMORY | L10.HBM.HBM | SohuLaneRepairMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HBM-0009-S2Q9 | 321454089 | v1 | 0.4 | Ulysses Kao | HBM_PMBIST_FAILED | HBM PMBIST failure | S2 ISOLATE | Q9 ESCALATE | Rerun up to 3 times (stack in affected_instances); isolate. Isolate the module after retry exhaustion; escalate to Etched with the captured evidence. | Memory | L10.HBM.HBM | SohuPmbistMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HBM-0010-S2Q6 | 321257482 | v1 | 0.4 | Ulysses Kao | HBM_MTC_STRESS_FAILED | HBM MTC stress test failure | S2 ISOLATE | Q6 RETRY | Rerun up to 3 times (stack+pseudo-channel in affected_instances); isolate. Isolate the module after retry exhaustion; escalate to Etched with the captured evidence. | Memory | L10.HBM.HBM | SohuMtcStressMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HBM-0011-S2Q9 | 321454091 | v1 | 0.4 | Ulysses Kao | HBM_DMA_BANDWIDTH_BELOW_THRESHOLD | PCIe-to-HBM read or write throughput on the side under test did not reach the configured min_bandwidth_gb_s floor. | S2 ISOLATE | Q9 ESCALATE | Escalate to Etched with the measured bandwidth, the side tested, and the configured threshold. | Memory | L10.HBM.HBM | SohuDmaMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HBM-0012-S2Q9 | 455671820 | v1 | 0.4 | Ulysses Kao | HBM_DEVICE_ID_MISMATCH | HBM device ID read failure or mismatch | S2 ISOLATE | Q9 ESCALATE | Escalate to Etched with the failing stack id and the reported HBM system info. | Data | L10.HBM.HBM | SohuHbmDeviceIdMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HBM-0013-S4Q9 | 323551245 | v1 | 0.4 | Ulysses Kao | HBM_LANE_REPAIR_INCONCLUSIVE | HBM lane repair sweep did not complete, so repairability was never established | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Memory | L10.HBM.HBM | SohuLaneRepairMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-INFRA-0001-S4Q9 | 424214529 | v1 | 0.3 | Ulysses Kao | RACKSIM_UNAVAILABLE | The configured racksim service or simulated server endpoint is unavailable. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Resource | L10.INFRA.STATION | RunInStressExecBinaryTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-INFRA-0002-S4Q9 | 289996802 | v1 | 0.3 | Ulysses Kao | REMOTE_EXECUTION_TRANSPORT_FAILED | The infrastructure transport used to execute a remote L10 operation failed before a DUT verdict was obtained. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Network | L10.INFRA.STATION | CpldDiagnosticsRemoteTestCaseExecBinaryTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-INFRA-0003-S4Q9 | 289996803 | v1 | 0.3 | Ulysses Kao | INFRA_NETWORK_PATH_UNAVAILABLE | A required station-side network path, route, or service endpoint is unavailable. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Network | L10.INFRA.STATION | ModelRegistryNfsMountTestCaseBmcConnectivityCheck | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-INFRA-0004-S4Q9 | 424214532 | v1 | 0.3 | Ulysses Kao | TEST_RESOURCE_EXHAUSTED | A station-side resource limit prevented the L10 test from running or completing. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Resource | L10.INFRA.STATION | StageModelRegistryToShmTestCaseStageWeightsTestCaseRunInStress | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-MEM-0001-S4Q9 | 390660097 | v1 | 0.3 | Ulysses Kao | MEMORY_INVENTORY_QUERY_FAILED | The host memory inventory command failed; no DIMM verdict is available. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.MEM.HOST_DIMM | CheckMemory | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-MEM-0002-S4Q9 | 390660098 | v1 | 0.3 | Ulysses Kao | MEMORY_INVENTORY_OUTPUT_INVALID | The host memory inventory output was malformed or could not be parsed. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.MEM.HOST_DIMM | CheckMemory | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-MEM-0003-S2Q9 | 321454083 | v1 | 0.3 | Ulysses Kao | NO_DIMMS_DETECTED | No populated DIMMs were detected although the L10 BOM requires host memory. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Memory | L10.MEM.HOST_DIMM | CheckMemory | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-MEM-0004-S2Q9 | 321454084 | v1 | 0.3 | Ulysses Kao | DIMM_COUNT_MISMATCH | The number of populated DIMMs does not match the expected L10 BOM. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Memory | L10.MEM.HOST_DIMM | CheckMemory | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-MEM-0005-S2Q9 | 455671813 | v1 | 0.3 | Ulysses Kao | DIMM_IDENTITY_OR_CAPACITY_MISMATCH | A DIMM manufacturer, part number, serial number, location, or capacity does not match the expected BOM. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Data | L10.MEM.HOST_DIMM | CheckMemory | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-MEM-0006-S4Q9 | 357105670 | v1 | 0.3 | Ulysses Kao | MIXED_DIMM_CONFIGURATION | The installed DIMMs are not homogeneous where the platform configuration requires matching parts and capacities. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.MEM.HOST_DIMM | CheckMemory | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-MEM-0007-S2Q6 | 321257479 | v1 | 0.3 | Ulysses Kao | MEMORY_STRESS_FAILED | The host memory stress workload reported data errors, verification failures, or a workload-failure exit status. | S2 ISOLATE | Q6 RETRY | Retry the affected L10 subset once and capture logs from the failing run. If persistent, isolate the unit; attach logs; route to FA; open Jira. | Memory | L10.MEM.HOST_DIMM | MemoryStressNgMemoryStress | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-MEM-0008-S4Q9 | 390660104 | v1 | 0.3 | Ulysses Kao | MEMORY_STRESS_METRICS_INVALID | The memory stress tool completed without usable result statistics or produced malformed metrics, so the verdict is inconclusive. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.MEM.HOST_DIMM | MemoryStressNgMemoryStress | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-MEM-0009-S5Q9 | 358154249 | v1 | 0.3 | Ulysses Kao | NUMA_TOPOLOGY_PRECONDITION_NOT_MET | The host NUMA/SNC topology does not provide the nodes required by the configured cross-NUMA I/O stress. | S5 CONFIG | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Config | L10.MEM.HOST_DIMM | IoStress | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-MEM-0010-S2Q6 | 321257482 | v1 | 0.3 | Ulysses Kao | CROSS_NUMA_MEMORY_STRESS_FAILED | The cross-NUMA stream or memory-transfer workload failed its execution or acceptance criteria. | S2 ISOLATE | Q6 RETRY | Retry the affected L10 subset once and capture logs from the failing run. If persistent, isolate the unit; attach logs; route to FA; open Jira. | Memory | L10.MEM.HOST_DIMM | IoStress | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-MEZZ-0004-S2Q6 | 371589124 | v1 | 0.4 | Ulysses Kao | MEZZ_CPLD_ACCESS_FAILED | Mezz/eval CPLD register access or write failure | S2 ISOLATE | Q6 RETRY | Retest the affected test once. Escalate to Etched if the failure repeats. | Hardware | L10.MEZZ.CPLD | CpldDiagnosticsRemoteTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0001-S5Q9 | 358154241 | v1 | 0.3 | Ulysses Kao | NIC_EXPECTED_CONFIG_UNAVAILABLE | The expected NIC/BOM/network configuration could not be loaded; no NIC verdict is available. | S5 CONFIG | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Config | L10.NIC.NIC | CheckNicCheckLomNicMacLoopbackTopologyTestNetworkBwRdmaTestCheckNicAsicRails | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0002-S4Q9 | 390660098 | v1 | 0.3 | Ulysses Kao | NIC_INVENTORY_QUERY_FAILED | NIC inventory or interface discovery failed. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.NIC.NIC | CheckNicCheckLomNicMacLoopbackTopologyTestNetworkBwRdmaTest | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0003-S4Q9 | 390660099 | v1 | 0.3 | Ulysses Kao | NIC_DISCOVERY_SERVICE_UNAVAILABLE | The NIC discovery/netd service or its Unix-domain socket was not ready or reachable. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.NIC.NIC | CheckNicCheckInterfaceLinkStatusCheckNetdReadiness | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0004-S2Q9 | 287899652 | v1 | 0.3 | Ulysses Kao | NIC_COUNT_MISMATCH | The discovered NIC count does not match the expected L10 BOM. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Network | L10.NIC.NIC | CheckNic | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0005-S2Q9 | 455671813 | v1 | 0.3 | Ulysses Kao | NIC_IDENTITY_MISMATCH | A NIC vendor, device, name, speed capability, or expected grouping does not match configuration. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Data | L10.NIC.NIC | CheckNic | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0006-S4Q9 | 457768966 | v1 | 0.3 | Ulysses Kao | NIC_DUPLICATE_PCI_ADDRESS | Multiple NIC records resolve to the same PCI address. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Data | L10.NIC.NIC | CheckNic | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0007-S2Q9 | 287899655 | v1 | 0.3 | Ulysses Kao | UNEXPECTED_NIC_DETECTED | A NIC that is not present in the expected L10 configuration was discovered. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Network | L10.NIC.NIC | CheckNic | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0008-S2Q9 | 287899656 | v1 | 0.3 | Ulysses Kao | LOM_NIC_NOT_DETECTED | The configured LAN-on-motherboard interface was not found. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Network | L10.NIC.LOM | CheckLomNicMac | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0009-S2Q9 | 455671817 | v1 | 0.3 | Ulysses Kao | LOM_MAC_ADDRESS_MISMATCH | The LAN-on-motherboard MAC address does not match the expected service/BOM value. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Data | L10.NIC.LOM | CheckLomNicMac | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0010-S4Q9 | 390660106 | v1 | 0.3 | Ulysses Kao | NIC_DRIVER_QUERY_FAILED | The installed NIC driver or firmware version could not be queried. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.NIC.NIC | CheckBroadcomDrivers | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0011-S5Q9 | 358154251 | v1 | 0.3 | Ulysses Kao | NIC_DRIVER_VERSION_MISMATCH | The installed NIC kernel driver does not match the required L10 release. | S5 CONFIG | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.NIC.NIC | CheckBroadcomDrivers | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0012-S5Q9 | 358154252 | v1 | 0.3 | Ulysses Kao | NIC_FIRMWARE_VERSION_MISMATCH | The installed NIC firmware does not match the required L10 release. | S5 CONFIG | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.NIC.NIC | CheckBroadcomDrivers | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0013-S4Q9 | 390660109 | v1 | 0.3 | Ulysses Kao | NIC_VERSION_OUTPUT_INVALID | NIC driver/firmware version output was incomplete or could not be parsed. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.NIC.NIC | CheckBroadcomDrivers | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0014-S2Q6 | 287703054 | v1 | 0.3 | Ulysses Kao | NIC_INTERFACE_LINK_DOWN | A required NIC interface has no carrier or is not operationally up. | S2 ISOLATE | Q6 RETRY | Retry the affected L10 subset once and capture logs from the failing run. If persistent, isolate the unit; attach logs; route to FA; open Jira. | Network | L10.NIC.NIC | CheckInterfaceLinkStatusLoopbackTopologyTest | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0015-S4Q9 | 390660111 | v1 | 0.3 | Ulysses Kao | NETD_NOT_READY | netd did not reach its ready state within the configured wait interval. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.NIC.NIC | CheckNetdReadiness | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0016-S5Q9 | 358154256 | v1 | 0.3 | Ulysses Kao | NETD_ACTIVE_SLAVE_MISMATCH | The active bond slave reported by netd is not the configured expected interface. | S5 CONFIG | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.NIC.NIC | CheckNetdReadiness | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0017-S2Q9 | 287899665 | v1 | 0.3 | Ulysses Kao | LOOPBACK_GROUP_COUNT_MISMATCH | A configured loopback NIC group does not contain the expected number of interfaces. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Network | L10.NIC.NIC | LoopbackTopologyTestNetworkBwRdmaTest | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0018-S5Q9 | 358154258 | v1 | 0.3 | Ulysses Kao | LOOPBACK_TOPOLOGY_INDEX_INVALID | A configured loopback client/server index is outside the discovered NIC group. | S5 CONFIG | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Config | L10.NIC.NIC | LoopbackTopologyTestNetworkBwRdmaTest | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0019-S2Q9 | 287899667 | v1 | 0.3 | Ulysses Kao | RDMA_DEVICE_UNAVAILABLE | A required interface has no RDMA device binding or the configured group has no RDMA-capable interface. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Network | L10.NIC.NIC | LoopbackTopologyTestNetworkBwRdmaTest | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0020-S4Q9 | 390660116 | v1 | 0.3 | Ulysses Kao | LOOPBACK_PAIR_CONFIGURATION_FAILED | The test could not configure the loopback pair IP, MTU, namespace, neighbor, or interface state. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.NIC.NIC | LoopbackTopologyTestNetworkBwRdmaTest | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0021-S4Q9 | 289996821 | v1 | 0.3 | Ulysses Kao | ROCE_GID_UNAVAILABLE | A required IPv4 RoCE v2 GID could not be found for one or both interfaces in the pair. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Network | L10.NIC.NIC | LoopbackTopologyTestNetworkBwRdmaTest | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0022-S2Q6 | 287703062 | v1 | 0.3 | Ulysses Kao | LOOPBACK_PROBE_FAILED | The short RDMA loopback probe failed to establish traffic or returned a workload failure. | S2 ISOLATE | Q6 RETRY | Retry the affected L10 subset once and capture logs from the failing run. If persistent, isolate the unit; attach logs; route to FA; open Jira. | Network | L10.NIC.NIC | LoopbackTopologyTest | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0023-S5Q9 | 358154263 | v1 | 0.3 | Ulysses Kao | NO_RDMA_PAIRS_TO_TEST | No valid RDMA-capable client/server pair remained after discovery and topology validation. | S5 CONFIG | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Config | L10.NIC.NIC | NetworkBwRdmaTest | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0024-S4Q9 | 390660120 | v1 | 0.3 | Ulysses Kao | RDMA_SERVER_START_FAILED | The ib_write_bw server process could not be started for an RDMA pair. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.NIC.NIC | NetworkBwRdmaTest | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0025-S2Q6 | 287703065 | v1 | 0.3 | Ulysses Kao | RDMA_CLIENT_EXECUTION_FAILED | The ib_write_bw client timed out or returned a non-zero status for an RDMA pair. | S2 ISOLATE | Q6 RETRY | Retry the affected L10 subset once and capture logs from the failing run. If persistent, isolate the unit; attach logs; route to FA; open Jira. | Network | L10.NIC.NIC | NetworkBwRdmaTest | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0026-S4Q9 | 390660122 | v1 | 0.3 | Ulysses Kao | RDMA_BANDWIDTH_OUTPUT_INVALID | The RDMA bandwidth tool did not emit a parseable average-bandwidth result. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.NIC.NIC | NetworkBwRdmaTest | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0027-S2Q9 | 287899675 | v1 | 0.3 | Ulysses Kao | RDMA_BANDWIDTH_BELOW_THRESHOLD | Measured RDMA bandwidth for a configured pair is below the configured percentage of expected line rate. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Network | L10.NIC.NIC | NetworkBwRdmaTest | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0028-S5Q9 | 358154268 | v1 | 0.3 | Ulysses Kao | NIC_RAIL_CONFIG_UNAVAILABLE | The expected NIC ASIC rail limits could not be loaded from product configuration. | S5 CONFIG | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Config | L10.NIC.ASIC_RAILS | CheckNicAsicRails | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0029-S4Q9 | 373882909 | v1 | 0.3 | Ulysses Kao | NIC_RAIL_READING_UNAVAILABLE | A required NIC ASIC voltage-rail reading is unavailable or reported as not applicable. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Hardware | L10.NIC.ASIC_RAILS | CheckNicAsicRails | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0030-S2Q9 | 371785758 | v1 | 0.3 | Ulysses Kao | NIC_RAIL_OUT_OF_RANGE | A NIC ASIC voltage rail is below its configured minimum or above its configured maximum. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Hardware | L10.NIC.ASIC_RAILS | CheckNicAsicRails | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0031-S4Q9 | 357105695 | v1 | 0.3 | Ulysses Kao | NIC_FIRMWARE_UPDATE_FAILED | The Broadcom NIC firmware update command failed. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.NIC.NIC | BroadcomUpdateFwAndConfigure | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0032-S4Q9 | 357105696 | v1 | 0.3 | Ulysses Kao | NIC_CONFIGURATION_APPLY_FAILED | Required NIC port, driver, or firmware configuration could not be applied. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.NIC.NIC | BroadcomUpdateFwAndConfigure | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0033-S4Q9 | 357105697 | v1 | 0.3 | Ulysses Kao | NIC_POST_UPDATE_VERIFY_FAILED | NIC firmware or configuration did not match the requested state after update. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.NIC.NIC | BroadcomUpdateFwAndConfigure | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0034-S4Q9 | 457768994 | v1 | 0.3 | Ulysses Kao | NIC_MAC_ADDRESS_UNAVAILABLE | A required interface MAC address could not be read while constructing a loopback/RDMA pair. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Data | L10.NIC.NIC | LoopbackTopologyTestNetworkBwRdmaTest | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0035-S2Q6 | 287703075 | v1 | 0.3 | Ulysses Kao | NIC_LINK_RECOVERY_FAILED | A carrier-down interface did not recover after the configured link-speed/FEC recovery sequence. | S2 ISOLATE | Q6 RETRY | Retry the affected L10 subset once and capture logs from the failing run. If persistent, isolate the unit; attach logs; route to FA; open Jira. | Network | L10.NIC.NIC | LoopbackTopologyTest | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-NIC-0036-S4Q9 | 289996836 | v1 | 0.4 | Ulysses Kao | NIC_PORT_INTERFACE_MISSING | netd reported a NIC port with no kernel interface name, so ethtool cannot read that port's module EEPROM and the cable there cannot be verified. | S4 TRIAGE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Network | L10.NIC.PORT | CheckCable | L10 Test | — | supercomputing-sw | harness 2026.224 |
TH-NIC-0037-S4Q9 | 457768997 | v1 | 0.4 | Ulysses Kao | CABLE_MODULE_INFO_INCOMPLETE | A pluggable module was read but cannot be checked: netd gave its port no position identifier or one outside the known layout, or the module did not report every identity field the check requires. | S4 TRIAGE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Data | L10.NIC.CABLE | CheckCable | L10 Test | — | supercomputing-sw | harness 2026.224 |
TH-NIC-0038-S2Q9 | 287899686 | v1 | 0.4 | Ulysses Kao | CABLE_MISSING_AT_POSITION | The BOM declares a cable at a port position where none was found: no NIC reported that position, the port was pruned earlier, or the module read failed. | S2 ISOLATE | Q9 ESCALATE | Verify the cable is installed and fully seated at the reported port position per the cabling SOP, then retest. Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Network | L10.NIC.CABLE | CheckCable | L10 Test | — | supercomputing-sw | harness 2026.224 |
TH-NIC-0039-S4Q9 | 357105703 | v1 | 0.4 | Ulysses Kao | NIC_PORT_POSITION_AMBIGUOUS | More than one module reported the same port position, so the check cannot tell which one is physically there and prunes all of them rather than guessing. | S4 TRIAGE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Config | L10.NIC.PORT | CheckCable | L10 Test | — | supercomputing-sw | harness 2026.224 |
TH-NIC-0040-S2Q9 | 371785768 | v1 | 0.4 | Ulysses Kao | CABLE_IDENTITY_MISMATCH | A module's identity fields do not match the BOM entry for its port position: Vendor PN (including the entry's alternate part numbers), Vendor name, Identifier, Transmitter technology, or a shopfloor-declared Vendor SN. | S2 ISOLATE | Q9 ESCALATE | Compare the installed cable against the BOM part for this position and install the correct part, then retest. Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | L10.NIC.CABLE | CheckCable | L10 Test | — | supercomputing-sw | harness 2026.224 |
TH-NIC-0041-S2Q9 | 371785769 | v1 | 0.4 | Ulysses Kao | CABLE_UNEXPECTED_AT_POSITION | A module was found at a port position where the BOM declares no cable. | S2 ISOLATE | Q9 ESCALATE | Remove or relocate the cable at the reported position per the cabling SOP, then retest. Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Hardware | L10.NIC.CABLE | CheckCable | L10 Test | — | supercomputing-sw | harness 2026.224 |
TH-PCIE-0001-S2Q6 | 371589121 | v1 | 0.3 | Ulysses Kao | PCIE_LINK_STATE_MISMATCH | PCIe link state mismatch (not L0 / Gen5 / x16) | S2 ISOLATE | Q6 RETRY | Rerun the PCIe subset up to 3 times; capture lspci and AER state before retry. Isolate the module after retry exhaustion; escalate to Etched with the captured evidence. | CATEGORY_HARDWARE | L10.PCIE.HOST_PCIE | CheckPcieTopology | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-PCIE-0003-S2Q9 | 371785731 | v1 | 0.3 | Ulysses Kao | PCIE_CSRAM_DMA_THROUGHPUT_BELOW_THRESHOLD | PCIe->CSRAM DMA throughput below threshold (50GB/s) | S2 ISOLATE | Q9 ESCALATE | Rerun the PCIe/DMA subset up to 3 times; reseat the module if failures persist. Capture lspci + AER state; isolate the module and route to triage. | CATEGORY_HARDWARE | L10.PCIE.HOST_PCIE | SohuDmaMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-PCIE-0005-S2Q9 | 371785733 | v1 | 0.3 | Ulysses Kao | PCIE_HBM_DMA_THROUGHPUT_BELOW_THRESHOLD | PCIe->HBM DMA throughput below threshold (16GB/s) | S2 ISOLATE | Q9 ESCALATE | Rerun the DMA subset up to 3 times; reseat the module if failures persist. Capture lspci link state and throughput numbers; isolate the module and route to triage. | CATEGORY_HARDWARE | L10.PCIE.HOST_PCIE | SohuDmaMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-PCIE-0008-S2Q6 | 371589128 | v1 | 0.3 | Ulysses Kao | SOHU_ENDPOINT_NOT_ENUMERATED | Sohu endpoint not enumerated (absent from PCIe tree / RPC proxy) | S2 ISOLATE | Q6 RETRY | Escalate to Etched. | CATEGORY_HARDWARE | L10.PCIE.HOST_PCIE | CheckPcieTopologySohuVfioPingMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-PCIE-0009-S5Q9 | 358154249 | v1 | 0.3 | Ulysses Kao | PCIE_EXPECTED_TOPOLOGY_UNAVAILABLE | The expected PCIe topology or device specification could not be loaded; no PCIe DUT verdict is available. | S5 CONFIG | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Config | L10.PCIE.HOST_PCIE | CheckPcieTopology | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-PCIE-0010-S4Q9 | 390660106 | v1 | 0.3 | Ulysses Kao | PCIE_INVENTORY_QUERY_FAILED | The PCIe inventory command failed, so the test could not enumerate devices or links. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.PCIE.HOST_PCIE | CheckPcieTopology | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-PCIE-0011-S4Q9 | 390660107 | v1 | 0.3 | Ulysses Kao | PCIE_INVENTORY_OUTPUT_INVALID | The PCIe inventory output was empty, incomplete, or could not be parsed into a reliable topology. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.PCIE.HOST_PCIE | CheckPcieTopology | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-PCIE-0012-S5Q9 | 358154252 | v1 | 0.3 | Ulysses Kao | PCIE_TOPOLOGY_SPEC_INVALID | The configured PCIe topology contains invalid or conflicting device identifiers, counts, or topology indices. | S5 CONFIG | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Config | L10.PCIE.HOST_PCIE | CheckPcieTopology | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-PCIE-0013-S2Q6 | 371589133 | v1 | 0.3 | Ulysses Kao | PCIE_TOPOLOGY_MISMATCH | The discovered PCIe device count or parent/child topology does not match the expected server topology. | S2 ISOLATE | Q6 RETRY | Retry the affected L10 subset once and capture logs from the failing run. If persistent, isolate the unit; attach logs; route to FA; open Jira. | Hardware | L10.PCIE.HOST_PCIE | CheckPcieTopology | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-PCIE-0014-S2Q9 | 455671822 | v1 | 0.3 | Ulysses Kao | PCIE_DEVICE_IDENTITY_MISMATCH | A discovered PCIe device has an unexpected vendor ID, device ID, subsystem ID, or configured device name. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Data | L10.PCIE.HOST_PCIE | CheckPcieTopology | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-PCIE-0015-S4Q9 | 457768975 | v1 | 0.3 | Ulysses Kao | PCIE_DUPLICATE_BDF | Two expected or discovered PCIe devices resolve to the same bus/device/function address. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Data | L10.PCIE.HOST_PCIE | CheckPcieTopology | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-PCIE-0016-S4Q9 | 390660112 | v1 | 0.3 | Ulysses Kao | PLX_DAEMON_VERSION_QUERY_FAILED | The PLX daemon firmware/version query failed or returned no usable switch information. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.PCIE.PLX_SWITCH | CheckPlxDaemonFirmwareVersion | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-PCIE-0017-S5Q9 | 358154257 | v1 | 0.3 | Ulysses Kao | PLX_DAEMON_FIRMWARE_VERSION_MISMATCH | The PLX daemon or switch firmware version does not match the expected L10 release. | S5 CONFIG | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.PCIE.PLX_SWITCH | CheckPlxDaemonFirmwareVersion | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-PCIE-0018-S4Q9 | 357105682 | v1 | 0.3 | Ulysses Kao | PLX_DAEMON_FIRMWARE_UPDATE_FAILED | The PLX daemon firmware update or activation command failed. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.PCIE.PLX_SWITCH | UpdatePlxDaemonFirmwareVersion | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-PCIE-0019-S4Q9 | 357105683 | v1 | 0.3 | Ulysses Kao | PLX_DAEMON_POST_UPDATE_VERIFY_FAILED | PLX daemon firmware did not report the requested version after the update completed. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.PCIE.PLX_SWITCH | UpdatePlxDaemonFirmwareVersion | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-PCIE-0020-S2Q9 | 455671828 | v1 | 0.4 | Ulysses Kao | PCIE_HBM_DMA_DATA_MISMATCH | PCIe->HBM DMA data integrity mismatch | S2 ISOLATE | Q9 ESCALATE | Rerun the DMA subset up to 3 times with the reported seed. Preserve unit; attach output and failing block map; isolate the module and route to triage. | Data | L10.PCIE.HOST_PCIE | SohuDmaMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-PWR-0007-S2Q9 | 371785735 | v1 | 0.3 | Ulysses Kao | UNEXPECTED_DVFS_RAMPDOWN | Unexpected DVFS rampdown event during workload | S2 ISOLATE | Q9 ESCALATE | Capture DVFS/power telemetry; rerun up to 3 times, then isolate. Isolate the module after retry exhaustion; escalate to Etched with the captured evidence. | CATEGORY_HARDWARE | L10.PWR.POWER | SohuPowerVirusMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-PWR-0009-S4Q9 | 390660105 | v1 | 0.3 | Ulysses Kao | BMC_POWER_CYCLE_COMMAND_FAILED | The BMC power-cycle command returned a failure. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.PWR.POWER | BmcPowerCycle | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-PWR-0010-S2Q9 | 371785738 | v1 | 0.3 | Ulysses Kao | HOST_DID_NOT_POWER_OFF | The host remained reachable after the BMC power-off step and did not enter the expected off state. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Hardware | L10.PWR.POWER | BmcPowerCycle | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-PWR-0011-S2Q9 | 371785739 | v1 | 0.3 | Ulysses Kao | HOST_DID_NOT_RECOVER_AFTER_POWER_CYCLE | The host did not become reachable within the configured interval after BMC power-on. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Hardware | L10.PWR.POWER | BmcPowerCycle | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-PWR-0012-S4Q9 | 390660108 | v1 | 0.3 | Ulysses Kao | HOST_REBOOT_COMMAND_FAILED | The host reboot request failed before the reboot transition could be observed. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.PWR.POWER | SohuHostRebootTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-PWR-0013-S4Q9 | 373882893 | v1 | 0.3 | Ulysses Kao | HOST_REBOOT_NOT_OBSERVED | The host never became unreachable after the reboot request, so the expected reboot transition was not observed. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Hardware | L10.PWR.POWER | SohuHostRebootTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-PWR-0014-S2Q9 | 371785742 | v1 | 0.3 | Ulysses Kao | HOST_DID_NOT_RECOVER_AFTER_REBOOT | The host did not return to a reachable and usable state within the configured reboot recovery interval. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Hardware | L10.PWR.POWER | SohuHostRebootTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-SOHU-0002-S2Q9 | 371785730 | v1 | 0.3 | Ulysses Kao | SOHU_NOT_PINGABLE | Sohu not pingable / no response from firmware | S2 ISOLATE | Q9 ESCALATE | Escalate to Etched with the unresponsive device index and the UART log. | CATEGORY_HARDWARE | L10.SOHU.SOHU_CHIP | SohuPingTestCaseSohuVfioPingMultiChipTestCaseSohuFlrMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-SOHU-0028-S4Q9 | 390660124 | v1 | 0.3 | Ulysses Kao | SOHU_WORKLOAD_BINARY_FAILED | MTT binary exited non-zero for a reason the harness cannot determine | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Software | L10.SOHU.SOHU_CHIP | SohuMlpTp8TestCaseSohuLlama70bForwardIteratedTestCaseSohuLlama70bHp8ForwardIteratedTestCaseSohuLlamaHp2Up4CachedInferenceMaxTestCaseSohuLlamaHp2Up4NospecInferenceMaxTestCaseSohuLlamaHp8CachedInferenceMaxTestCaseLlama8bFp8Tp1SmokeTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-SOHU-0029-S2Q9 | 321454109 | v1 | 0.4 | Ulysses Kao | SOHU_SRAM_TEST_FAILED | On-chip SRAM test failure (stuck bits / readback mismatch) | S2 ISOLATE | Q9 ESCALATE | Rerun the SRAM subset once (SRAM names in affected_instances). Isolate module; attach failing SRAM list and March C+ log; route to FA; open Jira. | Memory | L10.SOHU.SOHU_CHIP | SohuSramMemoryMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-SOHU-0030-S2Q9 | 371785758 | v1 | 0.4 | Ulysses Kao | SOHU_WORKLOAD_TIMEOUT | Workload hang / operation timeout | S2 ISOLATE | Q9 ESCALATE | Capture state before recovery and retry up to three times. Isolate the module after retry exhaustion. | Hardware | L10.SOHU.SOHU_CHIP | SohuMlpTp8TestCaseSohuLlama70bForwardIteratedTestCaseSohuLlama70bHp8ForwardIteratedTestCaseSohuLlamaHp2Up4CachedInferenceMaxTestCaseSohuLlamaHp2Up4NospecInferenceMaxTestCaseSohuLlamaHp8CachedInferenceMaxTestCaseLlama8bFp8Tp1SmokeTestCaseSohuPowerVirusMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-SOHU-0031-S2Q9 | 455671839 | v1 | 0.4 | Ulysses Kao | SOHU_WEIGHT_LOAD_MISMATCH | Weight load spot-check mismatch | S2 ISOLATE | Q9 ESCALATE | Rerun up to 3 times; isolate (subset offsets in raw_data). Isolate the module after retry exhaustion; escalate to Etched with the captured evidence. | Data | L10.SOHU.SOHU_CHIP | SohuLlama70bForwardIteratedTestCaseSohuLlama70bHp8ForwardIteratedTestCaseSohuLlamaHp2Up4CachedInferenceMaxTestCaseSohuLlamaHp2Up4NospecInferenceMaxTestCaseSohuLlamaHp8CachedInferenceMaxTestCaseLlama8bFp8Tp1SmokeTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-SOHU-0032-S2Q9 | 371785760 | v1 | 0.4 | Ulysses Kao | SOHU_INFERENCE_THROUGHPUT_BELOW_THRESHOLD | Sustained inference throughput on the million-token test did not reach 95% of the suite's expected_chip_tokens_per_second. | S2 ISOLATE | Q9 ESCALATE | Escalate to Etched with the measured tokens/s, the configured expectation, and the DVFS/thermal telemetry. | Hardware | L10.SOHU.SOHU_CHIP | SohuLlamaHp2Up4CachedInferenceMaxTestCaseSohuLlamaHp2Up4NospecInferenceMaxTestCaseSohuLlamaHp8CachedInferenceMaxTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-SOHU-0033-S2Q9 | 371785761 | v1 | 0.4 | Ulysses Kao | SOHU_INFERENCE_NUMERIC_MISMATCH | End-to-end inference numeric mismatch vs reference | S2 ISOLATE | Q9 ESCALATE | Retry up to three times. Isolate the module after retry exhaustion. | Hardware | L10.SOHU.SOHU_CHIP | SohuMlpTp8TestCaseSohuLlama70bForwardIteratedTestCaseSohuLlama70bHp8ForwardIteratedTestCaseSohuLlamaHp2Up4CachedInferenceMaxTestCaseSohuLlamaHp2Up4NospecInferenceMaxTestCaseSohuLlamaHp8CachedInferenceMaxTestCaseLlama8bFp8Tp1SmokeTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-SOHU-0034-S2Q9 | 371785762 | v1 | 0.4 | Ulysses Kao | SOHU_FLR_FAILED | FLR failed / flr_timeout_s 5.0s / recovery_timeout_s 10.0s / not pingable between FLRs | S2 ISOLATE | Q9 ESCALATE | Escalate to Etched with the failing iteration index, measured FLR durations, and post-reset link state. | Hardware | L10.SOHU.SOHU_CHIP | SohuFlrMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-SOHU-0035-S4Q9 | 373882915 | v1 | 0.4 | Ulysses Kao | SOHU_SNAPSHOT_CAPTURE_FAILED | Chip state snapshot capture failed | S4 TRIAGE | Q9 ESCALATE | Rerun up to 3 times. Escalate to Etched with the snapshot log. | Hardware | L10.SOHU.SOHU_CHIP | SohuSnapTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-SOHU-0036-S2Q9 | 371785764 | v1 | 0.4 | Ulysses Kao | SOHU_VFIO_PING_LATENCY_EXCEEDED | VFIO ping responses received but none within the allowed latency (device pingable, latency out of spec) | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Hardware | L10.SOHU.SOHU_CHIP | SohuVfioPingMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-THM-0008-S4Q9 | 357105672 | v1 | 0.3 | Ulysses Kao | FAN_LOCATION_MISMATCH | Fan location identifiers read back from the BMC do not match the configured physical fan positions. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.THM.FAN | BmcCheckFanLocation | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-THM-0009-S4Q9 | 373882889 | v1 | 0.3 | Ulysses Kao | FAN_PWM_SET_FAILED | The requested fan PWM value could not be applied through the BMC control interface. | S4 TRIAGE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Hardware | L10.THM.FAN | BmcCheckFanControl | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-THM-0010-S2Q9 | 371785738 | v1 | 0.3 | Ulysses Kao | FAN_RPM_RESPONSE_OUT_OF_RANGE | A fan RPM reading is outside the expected range or does not increase as required after a PWM change. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Hardware | L10.THM.FAN | BmcCheckFanControl | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-THM-0011-S4Q9 | 373882891 | v1 | 0.3 | Ulysses Kao | FAN_AUTOMATIC_MODE_RESTORE_FAILED | Fan control could not be returned to automatic mode after the manual fan test. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Hardware | L10.THM.FAN | BmcCheckFanControl | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-THM-0012-S2Q6 | 371589132 | v1 | 0.4 | Ulysses Kao | THERMAL_DIODE_READING_INVALID | Thermal diode temperature reading out of the qualified range, reported invalid by the sensor, or an uncalibrated mezzanine diode reading below 0 C. | S2 ISOLATE | Q6 RETRY | Verify thermal head contact and station ambient; rerun the thermal subset once (channel in affected_instances). Inspect mezzanine diode solder joints if an uncalibrated channel reads below 0 C. Isolate module; attach per-channel readings; route to FA; open Jira. | Hardware | L10.THM.THERMAL | SohuDtsMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-THM-0013-S2Q6 | 371589133 | v1 | 0.4 | Ulysses Kao | DTS_SPREAD_EXCEEDED | The peak-to-peak spread across usable Sohu DTS readings exceeded the configured deviation limit. | S2 ISOLATE | Q6 RETRY | Verify thermal forcing head seating, rerun the DTS test up to 3 times. Reseat the thermal forcing head and reapply TIM per assembly SOP. Isolate the module after retry exhaustion; escalate to Etched with the per-sensor spread. | Hardware | L10.THM.THERMAL | SohuDtsMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-THM-0014-S2Q6 | 371589134 | v1 | 0.4 | Ulysses Kao | CATASTROPHIC_THERMAL_TRIP | Catastrophic trip event during stress workload | S2 ISOLATE | Q6 RETRY | Capture thermal telemetry before retry; isolate if repeated. Isolate the module after retry exhaustion; escalate to Etched with the captured evidence. | Hardware | L10.THM.THERMAL | SohuPowerVirusMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-THM-0015-S4Q9 | 373882895 | v1 | 0.4 | Ulysses Kao | DTS_DEVIATION_UNDER_STRESS | DTS peak-to-peak deviation under power virus out of bounds | S4 TRIAGE | Q9 ESCALATE | Check TIM / thermal head seating; rerun; triage against the tentative bounds. Isolate the module after retry exhaustion; escalate to Etched with the captured evidence. | Hardware | L10.THM.THERMAL | SohuPowerVirusMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-THM-0016-S2Q9 | 371785744 | v1 | 0.4 | Ulysses Kao | PEAK_DTS_TEMPERATURE_EXCEEDED | Peak DTS temperature under power virus exceeds expected maximum | S2 ISOLATE | Q9 ESCALATE | Verify the unit completed the required 45-minute TIM baking process (updated from 15 minutes). If baked and still out of spec, escalate to Etched for FA. | Hardware | L10.THM.THERMAL | SohuPowerVirusMultiChipTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-VBB-0001-S4Q9 | 390660097 | v1 | 0.3 | Ulysses Kao | BASEBOARD_INVENTORY_QUERY_FAILED | Baseboard/VBB inventory or version information could not be queried. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.VBB.BASEBOARD | CheckBaseboardCheckBaseboardVersion | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-VBB-0002-S2Q9 | 455671810 | v1 | 0.3 | Ulysses Kao | BASEBOARD_IDENTITY_MISMATCH | Baseboard manufacturer, product, serial, or board identity does not match the expected BOM. | S2 ISOLATE | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Data | L10.VBB.BASEBOARD | CheckBaseboard | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-VBB-0003-S5Q9 | 358154243 | v1 | 0.3 | Ulysses Kao | BASEBOARD_VERSION_MISMATCH | The baseboard/VBB hardware or firmware version does not match the expected release. | S5 CONFIG | Q9 ESCALATE | Capture logs and isolate the unit; do not continue the L10 suite on this code. Attach the per-test log and suite URL; route to FA; open Jira. | Config | L10.VBB.BASEBOARD | CheckBaseboardVersion | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-VBB-0004-S5Q9 | 358154244 | v1 | 0.3 | Ulysses Kao | BASEBOARD_EXPECTED_CONFIG_UNAVAILABLE | Expected baseboard/VBB BOM or version data could not be loaded; no DUT verdict is available. | S5 CONFIG | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Config | L10.VBB.BASEBOARD | CheckBaseboardCheckBaseboardVersion | L10 Test | — | supercomputing-sw | harness 2026.226 |
Cross-station codes from common_th_registry.yaml. 19 codes.
| Error Code ID | Packed | Ver | Doc Rev | Original Author | Name | Message | Severity | Quick Action | Recover / Troubleshooting Procedure | Error Type | Component | Test Case | Source | Bugs | Owner | Since |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
TH-C2C-0010-S4Q9 | 390660106 | v1 | 0.4 | Ulysses Kao | C2C_CONFIG_REQUEST_FAILED | Firmware rejected the Ethernet configuration request or the configuration helper received a non-OK firmware status. No FLR is attempted; earlier chips may already have stored the override. | S4 TRIAGE | Q9 ESCALATE | Inspect the firmware status and host/firmware compatibility; common-reset the tray before retrying to clear partially stored overrides. | Software | C2C.SOHU_SERDES | C2cConfigureMultiChipTestCase | Common | — | supercomputing-sw | unreleased |
TH-C2C-0011-S4Q9 | 390660107 | v1 | 0.4 | Ulysses Kao | C2C_CONFIG_FLR_FAILED | All Ethernet override requests were stored, but the FLR helper returned an error while applying them. FEC readback was not attempted, so the requested configuration is unverified and this code alone does not diagnose hardware. | S4 TRIAGE | Q9 ESCALATE | Inspect the FLR error, proxy health, and firmware recovery; common-reset the tray before rerunning configuration and link validation. | Software | C2C.SOHU_SERDES | C2cConfigureMultiChipTestCase | Common | — | supercomputing-sw | unreleased |
TH-C2C-0012-S4Q9 | 357105676 | v1 | 0.4 | Ulysses Kao | C2C_CONFIG_READBACK_MISMATCH | After storing the override and completing FLR, one or more datalink pairs reported an unset FEC mode or a mode different from the requested value. All readback mismatches are included in the test message. This proves a configuration mismatch, not a link-quality or hardware failure. | S4 TRIAGE | Q9 ESCALATE | Compare expected and actual modes, inspect firmware configuration application and competing resets, then rerun configuration and link validation. | Config | C2C.SOHU_SERDES | C2cConfigureMultiChipTestCase | Common | — | supercomputing-sw | unreleased |
TH-ENV-0001-S4Q9 | 289996801 | v1 | 0.2 | Ulysses Kao | HOST_NOT_REACHABLE_AFTER_BOOT | The station host or its BMC did not answer ping or SSH after boot, so no module test could start. This is a test-system condition, not a module verdict. | S4 TRIAGE | Q9 ESCALATE | Connect a monitor to the Devkit and verify that the system boots successfully, then try ping with host / BMC. Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. | Network | 1X.ENV.STATION_HOST | — | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-ENV-0002-S4Q9 | 357105666 | v1 | 0.2 | Aidan Holm | STATION_FIXTURE_SETUP_CHECK_FAILED | Station fixture/setup check failed (Chroma telemetry / eval board power / vfio-proxy version query or mismatch / PEX version). | S4 TRIAGE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. DO NOT retest. | Config | 1X.ENV.STATION_FIXTURE | ChromaTelemetryCheckTestCaseEvalBoardPowerTestCaseVfioProxyVersionTestCaseCheckPexFirmwareVersionTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-ENV-0003-S4Q9 | 357105667 | v1 | 0.2 | Mykola Servetnyk | PLX_DAEMON_NO_PEX_SWITCHES | PLX daemon reported no PEX switches. | S4 TRIAGE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. DO NOT retest. | Config | 1X.ENV.PEX_SWITCH | CheckPexFirmwareVersionTestCase | 1X Module Test | ETCH-35552: CheckPexFirmwareVersionTestCase failure | supercomputing-sw | harness 2026.216 |
TH-ENV-0004-S4Q9 | 357105668 | v1 | 0.2 | Aidan Holm | HOST_SETUP_STEP_FAILED | A host driver/setup utility step failed: a kernel module reload, PCIe teardown or bus rescan, Kayak RAID remount, sohu_mdev sysfs parameter write, or log-reader activation returned non-zero on the station host. | S4 TRIAGE | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs. DO NOT retest. | Config | 1X.ENV.HOST_SETUP | ReloadKernelModuleTestCasePcieTeardownTestCasePcieRescanTestCaseSetSkipFlrOnOpenCloseTestCaseSohuLogReaderActivateTestCaseLogReaderCaptureTestCasePublishBootloaderTypeTestCaseKayakRaidRemountTestCase | 1X Module Test | 0 | supercomputing-sw | harness 2026.216 |
TH-HAR-0001-S4Q0 | 390070273 | v1 | 0.3 | Ulysses Kao | TEST_TIMEOUT | A test case exceeded its configured timeout and was terminated by the harness before producing a verdict. | S4 TRIAGE | Q0 NO_ACT | Rerun the affected subset once to check reproducibility. Route to engineering triage with the per-test log and timeout configuration. | CATEGORY_SOFTWARE | L10.HAR.HARNESS | — | L10 Test | — | supercomputing-sw | harness 2026.216 |
TH-HAR-0002-S5Q9 | 358154242 | v1 | 0.2 | Ulysses Kao | TEST_ARGS_INVALID | A test case was invoked with invalid or unsupported arguments (e.g. unexpected kwargs, unsupported mode) and rejected the run before touching the DUT. | S5 CONFIG | Q9 ESCALATE | Fix the suite YAML for this test; do not route the unit to rework | Config | L10.HAR.HARNESS | SohuSecureBootRecoveryUpdateTestCaseServerNestedTestCaseSohuSnapTestCaseVfioProxyVersionTestCaseSohuWeightLoadTestCaseSltModuleNestedTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-HAR-0003-S5Q9 | 358154243 | v1 | 0.2 | Ulysses Kao | TEST_DEPENDENCY_UNAVAILABLE | A tool, binary, or data file the test depends on (e.g. the pldm_ua binary) is missing on the test system, so the test could not perform its function. Distinct from TH-CPU-0003, which covers a tool missing on the DUT. Also covers missing VBB flash runfiles / nm toolchain / version_info artifacts for fixture firmware update and chipsim /mnt/weights 9p mount failure for Kayak workloads that never start. | S5 CONFIG | Q9 ESCALATE | Fix the station deployment / harness package; do not route the unit to rework | Config | L10.HAR.HARNESS | SohuSecureBootRecoveryUpdateTestCaseSohuWeightLoadTestCaseVbbFirmwareUpdateTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-HAR-0004-S4Q9 | 357105668 | v1 | 0.2 | Ulysses Kao | NESTED_SUITE_ORCHESTRATION_FAILED | A nested suite failed to launch or complete orchestration (container image load, API ready/config, abort/shutdown, missing kv_store inputs, or resource wiring). Child test failures carry their own codes; the wrapper never mints this code for an inner suite that ran and failed. Unpopulated chip slots on SltModuleNestedTestCase remain SKIP (not this code). | S4 TRIAGE | Q9 ESCALATE | Check suite config/resources; wrapper must propagate the first/most-severe child error code - never mint duplicate codes for child failures (test-system ERROR - excluded from yield) | Config | 1X.HAR.NESTED_SUITE | ServerNestedTestCaseSltModuleNestedTestCase | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
TH-HAR-0005-S4Q6 | 390463493 | v1 | 0.3 | Ulysses Kao | TEST_TOOL_EXECUTION_FAILED | A test-owned command or binary failed to execute and did not produce a domain-specific DUT verdict. | S4 TRIAGE | Q6 RETRY | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.HAR.HARNESS | IoStressExecBinaryTestCaseCpldDiagnosticsRemoteTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HAR-0006-S4Q9 | 390660102 | v1 | 0.3 | Ulysses Kao | TEST_METRICS_PARSE_FAILED | A test tool completed without complete, parseable metrics needed to determine the result. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.HAR.HARNESS | IoStressExecBinaryTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HAR-0007-S4Q9 | 390660103 | v1 | 0.3 | Ulysses Kao | TEST_OUTPUT_UNAVAILABLE | Expected test output, state, or result artifacts were not produced. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.HAR.HARNESS | StoreSystemLogStatesCheckSystemLogStatesExecBinaryTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HAR-0008-S5Q9 | 358154248 | v1 | 0.3 | Ulysses Kao | TEST_PRECONDITION_NOT_MET | A required test precondition was not satisfied, so the test did not exercise the DUT as intended. | S5 CONFIG | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Config | L10.HAR.HARNESS | RunInStressIoStressNetworkBwRdmaTest | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HAR-0009-S4Q9 | 390660105 | v1 | 0.3 | Ulysses Kao | TEST_CLEANUP_FAILED | Test cleanup failed and may have left processes, mounts, namespaces, files, or device state behind. | S4 TRIAGE | Q9 ESCALATE | Fix the station, image, or suite configuration; do NOT route the unit to rework. | Software | L10.HAR.HARNESS | LoopbackTopologyTestNetworkBwRdmaTestFreeModelRegistryFromShmTestCaseFreeWeightsTestCase | L10 Test | — | supercomputing-sw | harness 2026.226 |
TH-HAR-9998-S3Q0 | 271591182 | v1 | 0.3 | Ulysses Kao | TEST_FAILED_UNCLASSIFIED | Catch-all - a test case failed (raised TestFailedError) without carrying a specific TH error code. Frequent sources should be migrated to dedicated codes (spec §9 taxonomy-health metric). | S3 UNKNOWN | Q0 NO_ACT | Follow the failing test case's own documentation/SOP. Open a taxonomy-gap ticket to allocate a specific code for this failure mode. | CATEGORY_NONE | L10.HAR.HARNESS | — | L10 Test | — | supercomputing-sw | harness 2026.216 |
TH-HAR-9999-S3Q9 | 389621519 | v1 | 0.3 | Ulysses Kao | UNHANDLED_EXCEPTION | Unhandled exception during test execution, wrapped by the harness (spec §11.2.2). The traceback is preserved in the per-test log. | S3 UNKNOWN | Q9 ESCALATE | Collect the suite/test logs and artifacts for this failure. Escalate to Etched engineer with the collected logs and traceback; do NOT route the unit to rework or FA. | CATEGORY_SOFTWARE | L10.HAR.HARNESS | — | L10 Test | — | supercomputing-sw | harness 2026.216 |
TH-PWR-0001-S4Q2 | 373424129 | v1 | 0.2 | Mykola Servetnyk | STATION_PSU_LOAD_IMBALANCE | Station PSU load imbalance above 5% or PGOOD failure (fixture problem; excluded from module yield). | S4 TRIAGE | Q2 WARM_REBOOT | Known Devkit intermittent issue, retest once. | Hardware | 1X.PWR.STATION_PSU | CheckPsuHealth | 1X Module Test | — | supercomputing-sw | harness 2026.216 |
61 codes.
| Error Code ID | Ver | Doc Rev | Original Author | Name | Message | Severity | Quick Action | Recover / Troubleshooting Procedure | Error Type | Component | Test Case | Source | Possible Root Cause | Bugs | Owner | Since |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
EC-16060a0a003f sheet only | v1 | 0.3 | Ulysses Kao | — | ToR redundancy failed (aggregate) | — | Undefined | See enumerated sub-failures | Function | ToR Switch | TorSwitchRedundancyTestCase | L11 Test | — | — | — | — |
EC-16060b0b003a sheet only | v1 | 0.3 | Ulysses Kao | — | Mgmt switch redundancy failed (aggregate) | — | Undefined | See enumerated sub-failures | Function | Mgmt switch | MgmtSwitchRedundancyTestCase | L11 Test | — | — | — | — |
EC-160401000009 sheet only | v1 | 0.3 | Ulysses Kao | — | Resource not SSH | — | Undefined | Fix resource config | Initial | Undefined | PingAllP2pPairsTestCase | L11 Test | — | — | — | — |
EC-16040106000d sheet only | v1 | 0.3 | Ulysses Kao | — | Primary RMS provision: no ssh_target | — | Undefined | Set ssh_target in suite | Initial | RMS | ProvisionPrimaryRms | L11 Test | — | — | — | — |
EC-160401060015 sheet only | v1 | 0.3 | Ulysses Kao | — | Provision CDU/PDU/compute/RMS2: no ssh | — | Undefined | Set ssh_target | Initial | RMS | ProvisionCduProvisionPduProvisionComputeServerProvisionRedundantRms | L11 Test | — | — | — | — |
EC-16040106001f sheet only | v1 | 0.3 | Ulysses Kao | — | CDU/PDU health: no ssh_target | — | Undefined | Set suite ssh_target | Initial | RMS | CduHealthCheckTestCasePduHealthCheckTestCase | L11 Test | — | — | — | — |
EC-160401070030 sheet only | v1 | 0.3 | Ulysses Kao | — | BMC power-cycle wrapper: no ssh_target | — | Undefined | Point ssh at RMS | Initial | BMC | SohuServerBmcPowerCycle | L11 Test | — | — | — | — |
EC-160401070033 sheet only | v1 | 0.3 | Ulysses Kao | — | Firmware check: no ssh_target | — | Undefined | Point ssh at RMS | Initial | BMC | SohuServerFirmwareInventoryCheck | L11 Test | — | — | — | — |
EC-160402000001 sheet only | v1 | 0.3 | Ulysses Kao | — | Runner date failed | — | Undefined | Fix runner shell / harness | Connection | Undefined | CheckSystemClockTestCase | L11 Test | — | — | — | — |
EC-16040200000a sheet only | v1 | 0.3 | Ulysses Kao | — | SSH resource missing | — | Undefined | Fix keys / inventory | Connection | Undefined | PingAllP2pPairsTestCase | L11 Test | — | — | — | — |
EC-16040200000b sheet only | v1 | 0.3 | Ulysses Kao | — | SSH resource open error | — | Undefined | Undefined connection / fix RM | Connection | Undefined | PingAllP2pPairsTestCase | L11 Test | — | — | — | — |
EC-16040200000c sheet only | v1 | 0.3 | Ulysses Kao | — | Ping mesh failure | — | Undefined | Fix L3 / bonds / switches | Connection | Undefined | PingAllP2pPairsTestCase | L11 Test | — | — | — | — |
EC-160402020032 sheet only | v1 | 0.3 | Ulysses Kao | — | Host SSH not up after power-on | — | Undefined | OS / network / sshd | Connection | CPU | SohuServerBmcPowerCycle | L11 Test | — | — | — | — |
EC-160402040019 sheet only | v1 | 0.3 | Ulysses Kao | — | PDU unreachable (ping) | — | Undefined | PDU network / power | Connection | PDU | ProvisionPdu | L11 Test | — | — | — | — |
EC-160402050018 sheet only | v1 | 0.3 | Ulysses Kao | — | CDU unreachable (ping) | — | Undefined | CDU network / power | Connection | CDU | ProvisionCdu | L11 Test | — | — | — | — |
EC-160402060003 sheet only | v1 | 0.3 | Ulysses Kao | — | RMS date over SSH failed | — | Undefined | Check RMS SSH / routing | Connection | RMS | CheckSystemClockTestCase | L11 Test | — | — | — | — |
EC-160402060013 sheet only | v1 | 0.3 | Ulysses Kao | — | MTS apt proxy unreachable locally | — | Undefined | Start / fix apt proxy | Connection | RMS | ProvisionPrimaryRms | L11 Test | — | — | — | — |
EC-160402060014 sheet only | v1 | 0.3 | Ulysses Kao | — | RMS cannot reach apt proxy | — | Undefined | Routing / firewall to proxy | Connection | RMS | ProvisionPrimaryRms | L11 Test | — | — | — | — |
EC-1604020a002a sheet only | v1 | 0.3 | Ulysses Kao | — | Switch eAPI curl failed | — | Undefined | Fix switch mgmt / creds | Connection | ToR Switch | SwitchInventoryCheckTestCaseMgmtSwitchRedundancyTestCase | L11 Test | — | — | — | — |
EC-16040302001d sheet only | v1 | 0.3 | Ulysses Kao | — | Compute server provisioning failed | — | Undefined | RC provision logs | Provision | CPU | ProvisionComputeServer | L11 Test | — | — | — | — |
EC-160403060010 sheet only | v1 | 0.3 | Ulysses Kao | — | Ansible playbook failed | — | Undefined | Read ansible artifacts / logs | Provision | RMS | ProvisionPrimaryRms | L11 Test | — | — | — | — |
EC-160403060011 sheet only | v1 | 0.3 | Ulysses Kao | — | Image import to RMS failed | — | Undefined | Check microk8s / disk on RMS | Provision | RMS | ProvisionPrimaryRms | L11 Test | — | — | — | — |
EC-160403060012 sheet only | v1 | 0.3 | Ulysses Kao | — | Cleanup of images failed | — | Undefined | SSH / permissions on RMS | Provision | RMS | ProvisionPrimaryRms | L11 Test | — | — | — | — |
EC-16040306001a sheet only | v1 | 0.3 | Ulysses Kao | — | Rack-controller provisioning task failed | — | Undefined | RC logs / rack controller | Provision | RMS | ProvisionCduProvisionPduProvisionRedundantRms | L11 Test | — | — | — | — |
EC-1604030b0006 sheet only | v1 | 0.3 | Ulysses Kao | — | ZTP container start failed | — | Undefined | Check ZTP service on RMS | Provision | Mgmt switch | SwitchZtpTriggerTestCase | L11 Test | — | — | — | — |
EC-1604030b0007 sheet only | v1 | 0.3 | Ulysses Kao | — | ZTP container stop failed | — | Undefined | Inspect ZTP logs / daemon | Provision | Mgmt switch | SwitchZtpTriggerTestCase | L11 Test | — | — | — | — |
EC-160404020035 sheet only | v1 | 0.3 | Ulysses Kao | — | Per-server BOM scope: missing server_id | — | Undefined | Pass server_id in suite | Config | CPU | _SohuServerScopedBomMixin | L11 Test | — | — | — | — |
EC-160404040023 sheet only | v1 | 0.3 | Ulysses Kao | — | PDU resource not PduConnection | — | Undefined | Fix PDU resource entry | Config | PDU | PduHealthCheckTestCasePduInventoryCheckTestCase | L11 Test | — | — | — | — |
EC-16040404002e sheet only | v1 | 0.3 | Ulysses Kao | — | No PDU in BOM | — | Undefined | Fix BOM | Config | PDU | PduInventoryCheckTestCase | L11 Test | — | — | — | — |
EC-160404040039 sheet only | v1 | 0.3 | Ulysses Kao | — | PDU resource invalid for redundancy test | — | Undefined | Fix pdu_target resource | Config | PDU | MgmtSwitchRedundancyTestCase | L11 Test | — | — | — | — |
EC-16040404003e sheet only | v1 | 0.3 | Ulysses Kao | — | PDU resource invalid for ToR test | — | Undefined | Fix pdu_target resource | Config | PDU | TorSwitchRedundancyTestCase | L11 Test | — | — | — | — |
EC-160404050017 sheet only | v1 | 0.3 | Ulysses Kao | — | CDU interface / entity mismatch | — | Undefined | Fix CDU hostname in registry | Config | CDU | ProvisionCdu | L11 Test | — | — | — | — |
EC-160404050022 sheet only | v1 | 0.3 | Ulysses Kao | — | CDU resource not CduConnection | — | Undefined | Fix CDU resource entry | Config | CDU | CduHealthCheckTestCaseCduInventoryCheckTestCase | L11 Test | — | — | — | — |
EC-16040405002c sheet only | v1 | 0.3 | Ulysses Kao | — | No CDU in BOM | — | Undefined | Fix BOM | Config | CDU | CduInventoryCheckTestCase | L11 Test | — | — | — | — |
EC-160404060005 sheet only | v1 | 0.3 | Ulysses Kao | — | Clock skew over tolerance | — | Undefined | Fix NTP / time sync | Config | RMS | CheckSystemClockTestCase | L11 Test | — | — | — | — |
EC-16040406000e sheet only | v1 | 0.3 | Ulysses Kao | — | Vault / inputs validation failed | — | Undefined | Fix l11_inputs / vault file | Config | RMS | ProvisionPrimaryRms | L11 Test | — | — | — | — |
EC-16040406000f sheet only | v1 | 0.3 | Ulysses Kao | — | No provisioning images found | — | Undefined | Restore deploy image tarball | Config | RMS | ProvisionPrimaryRms | L11 Test | — | — | — | — |
EC-160404060016 sheet only | v1 | 0.3 | Ulysses Kao | — | Unknown rack entity | — | Undefined | Fix rack_entities / IDs | Config | RMS | ProvisionCduProvisionPdu | L11 Test | — | — | — | — |
EC-16040406001e sheet only | v1 | 0.3 | Ulysses Kao | — | Redundant RMS vault/data error | — | Undefined | Fix vault inputs | Config | RMS | ProvisionRedundantRms | L11 Test | — | — | — | — |
EC-160404060020 sheet only | v1 | 0.3 | Ulysses Kao | — | RMS SSH resource wrong type | — | Undefined | Fix resource YAML | Config | RMS | CduHealthCheckTestCasePduHealthCheckTestCase | L11 Test | — | — | — | — |
EC-160404060021 sheet only | v1 | 0.3 | Ulysses Kao | — | Rack controller port missing | — | Undefined | Add RC port to resources | Config | RMS | CduHealthCheckTestCasePduHealthCheckTestCase | L11 Test | — | — | — | — |
EC-160404060028 sheet only | v1 | 0.3 | Ulysses Kao | — | RMS BOM missing | — | Undefined | Fix BOM YAML | Config | RMS | RmsInventoryCheckTestCase | L11 Test | — | — | — | — |
EC-160404060029 sheet only | v1 | 0.3 | Ulysses Kao | — | RMS inventory mismatch | — | Undefined | Replace HW / fix BOM | Config | RMS | RmsInventoryCheckTestCase | L11 Test | — | — | — | — |
EC-160404070034 sheet only | v1 | 0.3 | Ulysses Kao | — | Firmware version mismatch | — | Undefined | Update BMC/BIOS/PSU FW | Config | BMC | SohuServerFirmwareInventoryCheck | L11 Test | — | — | — | — |
EC-1604040a002b sheet only | v1 | 0.3 | Ulysses Kao | — | Switch BOM mismatch | — | Undefined | Replace switch / BOM | Config | ToR Switch | SwitchInventoryCheckTestCase | L11 Test | — | — | — | — |
EC-1604040a003b sheet only | v1 | 0.3 | Ulysses Kao | — | No ToR switches in registry | — | Undefined | Fix registry accessors | Config | ToR Switch | TorSwitchRedundancyTestCase | L11 Test | — | — | — | — |
EC-1604040a003c sheet only | v1 | 0.3 | Ulysses Kao | — | ToR switch ID not in registry | — | Undefined | Fix rack_entities.txtpb | Config | ToR Switch | TorSwitchRedundancyTestCase | L11 Test | — | — | — | — |
EC-1604040a003d sheet only | v1 | 0.3 | Ulysses Kao | — | ToR switch lacks pdu_connections | — | Undefined | Fix PDU outlet mapping | Config | ToR Switch | TorSwitchRedundancyTestCase | L11 Test | — | — | — | — |
EC-1604040b0036 sheet only | v1 | 0.3 | Ulysses Kao | — | Mgmt switch ID not in registry | — | Undefined | Fix rack_entities.txtpb | Config | Mgmt switch | MgmtSwitchRedundancyTestCase | L11 Test | — | — | — | — |
EC-1604040b0037 sheet only | v1 | 0.3 | Ulysses Kao | — | Mgmt switch lacks pdu_connections | — | Undefined | Fix PDU outlet mapping | Config | Mgmt switch | MgmtSwitchRedundancyTestCase | L11 Test | — | — | — | — |
EC-1604040b0038 sheet only | v1 | 0.3 | Ulysses Kao | — | Cannot resolve mgmt switch peer | — | Undefined | Expect two mgmt switches | Config | Mgmt switch | MgmtSwitchRedundancyTestCase | L11 Test | — | — | — | — |
EC-160406000002 sheet only | v1 | 0.3 | Ulysses Kao | — | Runner epoch parse failed | — | Undefined | Fix runner clock output | Function | Undefined | CheckSystemClockTestCase | L11 Test | — | — | — | — |
EC-160406000008 sheet only | v1 | 0.3 | Ulysses Kao | — | Need ≥2 ping targets | — | Undefined | Fix suite ping_targets | Function | Undefined | PingAllP2pPairsTestCase | L11 Test | — | — | — | — |
EC-160406060004 sheet only | v1 | 0.3 | Ulysses Kao | — | RMS epoch parse failed | — | Undefined | Fix RMS clock output | Function | RMS | CheckSystemClockTestCase | L11 Test | — | — | — | — |
EC-16060504001c sheet only | v1 | 0.3 | Ulysses Kao | — | PDU post-provision health bad | — | Undefined | PDU telemetry / RC state | Function | PDU | ProvisionPdu | L11 Test | — | — | — | — |
EC-160605040025 sheet only | v1 | 0.3 | Ulysses Kao | — | PDU health aggregate failure | — | Undefined | See per-line daemon logs | Function | PDU | PduHealthCheckTestCase | L11 Test | — | — | — | — |
EC-16060504002f sheet only | v1 | 0.3 | Ulysses Kao | — | PDU inventory mismatch | — | Undefined | Fix PDU vs BOM | Function | PDU | PduInventoryCheckTestCase | L11 Test | — | — | — | — |
EC-16060505001b sheet only | v1 | 0.3 | Ulysses Kao | — | CDU post-provision health bad | — | Undefined | CDU telemetry / RC state | Function | CDU | ProvisionCdu | L11 Test | — | — | — | — |
EC-160605050024 sheet only | v1 | 0.3 | Ulysses Kao | — | CDU health aggregate failure | — | Undefined | See per-line daemon logs | Function | CDU | CduHealthCheckTestCase | L11 Test | — | — | — | — |
EC-16060505002d sheet only | v1 | 0.3 | Ulysses Kao | — | CDU inventory mismatch | — | Undefined | Fix CDU vs BOM | Function | CDU | CduInventoryCheckTestCase | L11 Test | — | — | — | — |
EC-160607070031 sheet only | v1 | 0.3 | Ulysses Kao | — | BMC power-cycle sequence failed | — | Undefined | BMC / bmc-daemon logs | Function | BMC | SohuServerBmcPowerCycle | L11 Test | — | — | — | — |
DRI 1 of a code’s primary test case is what the Original Author column above resolves to; codes with no DRI listed fall back to the spec author (Ulysses Kao). Hover an author cell to see which basis was used.
| Test case | DRI 1 | DRI 2 | Status | PR |
|---|---|---|---|---|
BootloaderResultTestCase | Mykola Servetnyk | Ulysses Kao | merged | #23002 |
C2cIntegrityTestCase | Mykola Servetnyk | Ulysses Kao | merged | #22810 |
C2cLaneMarginTestCase | Mykola Servetnyk | Ulysses Kao | merged | #22813 |
C2cLinkupTestCase | Mykola Servetnyk | Ulysses Kao | merged | #22808 |
C2cPrbsLoopbackTestCase | Mykola Servetnyk | Ulysses Kao | merged | #23117 |
C2cThroughputTestCase | Mykola Servetnyk | Ulysses Kao | merged | #22812 |
CheckFirmwareVersionTestCase | Mykola Servetnyk | Ulysses Kao | — | #24591 |
CheckPexFirmwareVersionTestCase | Mykola Servetnyk | Ulysses Kao | merged | #23116 |
CheckPsuHealth | Mykola Servetnyk | Ulysses Kao | merged | #23963 |
ChromaTelemetryCheckTestCase | Aidan Holm | Ulysses Kao | — | — |
CpldDiagnosticsTestCase | Mykola Servetnyk | Ulysses Kao | — | #24594 |
DetectBoardDvfsProfileTestCase | Mykola Servetnyk | Ulysses Kao | merged | #23003 |
EvalBoardPowerTestCase | Aidan Holm | Jack Chou | — | — |
KayakTestCase | Aidan Holm | Jack Chou | — | — |
KvFlushAndAttentionInterleavedSingleChipTestCase | Aidan Holm | Jack Chou | — | — |
KvFlushAndAttentionSingleChipTestCase | Aidan Holm | Jack Chou | — | — |
LogReaderCaptureTestCase | Aidan Holm | Jack Chou | — | — |
MezzanineI2cAlertsTestCase | Aidan Holm | Jack Chou | — | — |
PcieRescanTestCase | Ulysses Kao | Mykola Servetnyk | — | #22868 |
PcieSetupTestCase | Aidan Holm | Jack Chou | — | — |
PcieTeardownTestCase | Aidan Holm | Jack Chou | — | — |
ProgramSecureBootloaderTestCase | Aidan Holm | Jack Chou | — | — |
PublishBootloaderTypeTestCase | Aidan Holm | Jack Chou | — | — |
ReloadKernelModuleTestCase | Aidan Holm | Jack Chou | — | — |
RetimerFirmwareUpdateTestCase | Aidan Holm | Jack Chou | — | — |
RunAllAttnHbmBypassTestsSingleChipTestCase | Sam Jiang | James Chou | In review | #22847 |
RunAllAttnTestsSingleChipTestCase | Sam Jiang | James Chou | In review | #22838 |
RunKvFlushTestsSingleChipTestCase | Sam Jiang | James Chou | In review | #22839 |
SaSortRampupHbmSingleChipTestCase | Sam Jiang | James Chou | In review | #22840 |
ServerNestedTestCase | Ulysses Kao | Mykola Servetnyk | — | #22859 |
SetSkipFlrOnOpenCloseTestCase | Sam Jiang | James Chou | In review | #22846 |
SltModuleNestedTestCase | Ulysses Kao | Mykola Servetnyk | — | #22863 |
SohuBusConnectivityTestCase | Sam Jiang | James Chou | In review | #22841 |
SohuChipBinTestCase | Ulysses Kao | Mykola Servetnyk | — | #22865 |
SohuChipIdTestCase | Sam Jiang | James Chou | In review | #22842 |
SohuCtsTestCase | Sam Jiang | James Chou | In review | #22843 |
SohuDmaTestCase | Sam Jiang | James Chou | In review | #22811 |
SohuDtsTestCase | Sam Jiang | James Chou | In review | #22844 |
SohuEfuseTestCase | Sam Jiang | James Chou | In review | #22845 |
SohuFirmwareUpdateTestCase | Mykola Servetnyk | James Chou | merged | #22816 #22817 |
SohuFlrTestCase | James Chou | Aidan Holm | In review | #22807 |
SohuGpioTestCase | James Chou | Aidan Holm | In review | #22801 |
SohuHbmBandwidthTestCase | James Chou | Aidan Holm | In review | #22819 |
SohuHbmDeviceIdTestCase | James Chou | Aidan Holm | In review | #22805 |
SohuHotReloadFirmwareUpdateTestCase | James Chou | Aidan Holm | — | #22820 |
SohuI2cTestCase | James Chou | Aidan Holm | In review | #22800 |
SohuJtagTestCase | James Chou | Aidan Holm | In review | #22802 |
SohuLaneRepairTestCase | James Chou | Aidan Holm | — | #22806 |
SohuLlama8bFp8ForwardTestCase | Aidan Holm | James Chou | — | — |
SohuLlama8bFp8PrefillTestCase | Aidan Holm | James Chou | — | — |
SohuLlamaMttTestCase | James Chou | Aidan Holm | — | #22827 |
SohuLogReaderActivateTestCase | James Chou | Aidan Holm | In review | #22804 |
SohuMtcStressTestCase | Jack Chou | Sam Jiang | — | #22899 |
SohuPcieAerCheckTestCase | Jack Chou | Sam Jiang | — | #22892 |
SohuPcieCsramThroughputTestCase | Jack Chou | Sam Jiang | — | #22890 |
SohuPcieLaneMarginTestCase | Jack Chou | Sam Jiang | — | #22889 |
SohuPcieRateChangeTestCase | Jack Chou | Sam Jiang | — | #22893 |
SohuPdTestCase | Jack Chou | Sam Jiang | — | #22898 |
SohuPmbistTestCase | Jack Chou | Sam Jiang | — | #22891 |
SohuPowerTelemetryTestCase | Jack Chou | Sam Jiang | — | #22897 |
SohuPowerVirusTestCase | Jack Chou | Sam Jiang | — | #22895 |
SohuPowerVirusThermalTestCase | Jack Chou | Sam Jiang | — | #22894 |
SohuRdqsSweepTrainingTestCase | Jack Chou | Sam Jiang | — | #22888 |
SohuSaScreeningTestCase | Jack Chou | Sam Jiang | — | #22896 |
SohuSauPllPhaseAlignmentTestCase | Ulysses Kao | Mykola Servetnyk | — | #22803 |
SohuSecureBootRecoveryUpdateTestCase | Ulysses Kao | Mykola Servetnyk | — | #22809 |
SohuSnapTestCase | Ulysses Kao | Mykola Servetnyk | — | #22866 |
SohuSramMemoryTestCase | Ulysses Kao | Mykola Servetnyk | — | #22828 |
SohuThermalDiodeTestCase | Ulysses Kao | Mykola Servetnyk | — | #22830 |
SohuUartTestCase | Ulysses Kao | Mykola Servetnyk | — | #22851 |
SohuVfioPingTestCase | Ulysses Kao | Mykola Servetnyk | — | #22853 |
SohuVmTestCase | Ulysses Kao | Mykola Servetnyk | — | #22854 |
SohuVrmTestCase | Ulysses Kao | Mykola Servetnyk | — | #22855 |
SohuWeightLoadTestCase | Ulysses Kao | Mykola Servetnyk | — | #22856 |
VbbFirmwareUpdateTestCase | Ulysses Kao | Mykola Servetnyk | — | #22858 |
VfioProxyVersionTestCase | Ulysses Kao | Mykola Servetnyk | — | #22860 |
Points where the spreadsheet needed a judgment call to normalize.
These Error Code IDs each appear twice in one stage block of the sheet (an older short-form row plus a later expanded row). Identity segments are immutable, so the repeat is the same code: the row with more detail is kept and the other is listed here for reconciliation in the sheet.
| Error Code ID | Stage | Quick action kept | Quick action dropped | Dropped row message | Dropped row test cases |
|---|---|---|---|---|---|
TH-SOHU-0024-S5Q9 | MLT / 1X Module Test | ESCALATE | HALT_ESCALATE | Required screening data missing or failed to load (SA mask / bin inputs) | SohuChipBinTestCase, SohuSaScreeningTestCase, SaSortRampupHbmSingleChipTestCase |
TH-SOHU-0026-S2Q9 | MLT / 1X Module Test | ESCALATE | HALT_ESCALATE | PLL glitch frequency did not read back within 5% of target before timeout | SetPllGlitchFreqTestCase |
TH-SOHU-0027-S2Q9 | MLT / 1X Module Test | ESCALATE | HALT_ESCALATE | VFIO ping responses received but none within the allowed latency (device pingable, latency out of spec) | SohuVfioPingTestCase |
etched-ai/sw → host/system_test/error_codes (325 codes at sw@65cf3520a606; snapshots committed alongside this page).gen.py from that
registry plus the spreadsheet — regenerate rather than hand-edit.