AMC13 Debugging Hints

My AMC13 is plugged in but I can't contact it!

Check sensor info (NAT MCH)

If you have an NAT MCH, you can learn a lot about the state of your AMC13 (or other MicroTCA module) with the show_sensorinfo command. First, connect to your MCH using telnet:

[cms2] /home/hazen/amc13_python/src_amc13 > telnet
Connected to
Escape character is '^]'.

Welcome to NAT-MCH

nat> show_fru

FRU Information:
 FRU  Device  State  Name
  0   MCH      M4    NMCH-CM
  3   mcmc1    M4    NAT-MCH-MCMC
  8   AMC4     M4    AMC13
 13   AMC9     M4    AMC13
 30   AMC13    M4    AMC13
 40   CU1      M4    VT VT095
 41   CU2      M4    VT VT095
 50   PM1      M4    VT UTC010
 60   Clk1     M4    MCH-DTC

The show_fru command lists the "field replaceable units" (i.e. cards) plugged in to the crate. It knows about AMC13's, so they appear by name (in fact it currently mis-identifies MiniCTR2 as AMC13!). We have a MiniCTR2 in slot 4, an AMC13 in slot 9 and another AMC13 in the MCH2 sigte (FRU 30).

nat> show_sensorinfo 30
Sensor Information for AMC 13
  #   SDRType  Sensor Entity Inst  Value   State  Name
  0   MDevLoc          0xc1  0x7a                 AMC13
  0   Full     0xf2    0xc1  0x7a  0x01        Hotswap
  1   Full     Temp    0xc1  0x7a  33.6    ok     Amb. Temp
  3   Full     Voltage 0xc1  0x7a  12.544  ok     +12V
  4   Full     Voltage 0xc1  0x7a  3.2688  ok     +3.3V BkEnd
  5   Full     Voltage 0xc1  0x7a  1.1932  ok     T2 1.2V
  9   Full     0xc0    0xc1  0x7a   0x83     0x00  GPIO 7:0

The Hotswap sensor indicates the state of the switch connected to the handle (actually, the MMC state machine driven by it). When the module is correctly plugged in and the handle pushed in, it should read 0x01 as above. The values gleaned from the MMC firmware listing are as follows:

(bit values given as an 'or' in the readout):

  01 - handle closed
  02 - handle open
  04 - quiesced (??)
  08 - backend power failure
  10 - backend power shutdown

Also, the +12V should read something like 12V.

IP Address set incorrectly or unknown

What IP address is your AMC13 using? You can find out as follows.

$ cd ...../dev_tools/amc13Config
$ #--- edit to set your MCH IP address ---
$ ./
 1: MMC: -none-
 2: MMC: -none-
 3: MMC: -none-
AMC13Tool2 threw an exception
Address table path "/home/hazen/work/new/amc13/amc13/etc/amc13" set from AMC13_ADDRESS_TABLE_PATH
use_ch false
Created URI from IP address:
  T2: ipbusudp-2.0://
  T1: ipbusudp-2.0://
Caught microHAL exception.
 4: MMC: 2.2    IP: vv: 0x0000 sv: 0x0000 sn: 0
 5: MMC: -none-
 6: MMC: -none-
 7: MMC: -none-
 8: MMC: -none-
 9: MMC: 2.1    IP: vv: 0x4037 sv: 0x0025 sn: 43
10: MMC: -none-
11: MMC: 2.2    IP: vv: 0x4037 sv: 0xfff9 sn: 227
12: MMC: 2.2    IP: vv: 0x100a sv: 0x0021 sn: 106
13: MMC: 2.2    IP: vv: 0x4037 sv: 0x002d sn: 161

You can see that the board in slot 4 has crazy IP addresses (20.200 and 10.100). You can reset them as follows:

./ --slot=4 -i
./ --slot=4 -i

This will reset the AMC13 to two successive IP addresses beginning with The applyConfig sets the address immediately. The storeConfig sets it in the EEPROM for next power-up.

If the above fails for some reason you may need to erase the EEPROM attached to the MMC. Connect a mini USB cable (not micro USB) to the front panel, which is the console for the MMC microcontroller. It should enumerate as a usb emulated serial port, maybe /dev/ttyUSB0. (look in dmesg to see).

Then connect using a terminal program (minicom is what I use) set to 19200 baud, 8 data bits, no parity, no hardware or software handshake.

You can type "help" to get a list of commands, but what you want is "eeperase" followed by "yes". After this, cycle the crate or module power and the IP address should be reset to the S/N default (see IPaddressAssignment). Then you can use applyConfig and storeConfig to reset it.

Point 5 Test Crate

Some helpful hints from Jim.

from hcaldaq12:

(this gets you a list of commands)

(this gets you list of modules)

pwr_off 30
(power off module no. 30)

pwr_on 30
(power on module no. 30)

Remote Hard reset of AMC13

It is rumored that the following will reset an AMC moule if you have an NAT MCH:

I have a good news. After discussing this issue with several people and also with colleagues from HCAL, I discovered that NAT-MCH has a hidden set of commands, which you can simply run from telnet. So there is a working solution for the FPGA<->MMC communication getting stuck. The way I do so:

  1. "telnet" - connect to the MCH using telnet
  2. "show_pm" - print the list of the units, we will need the FruId from there. So if your FC7 is in slot AMC3, the FruId is 7
  3. "hidden" - print hidden commands:))
  4. "hard_reset 7" - reset the FC7
Description is for FC7 but should apply to AMC13 as well

Talking to the MMC over IPMI

The IPMI command for graceful reboot is as follows:

  ipmitool -H -U '' -P '' -T 0x82 -B 0 -b 7 -t 0xa4 raw 0x2c 0x04 0 26 0x02 

-- EricHazen - 20 Mar 2012

This topic: BUCMSPublic > WebHome > HcalDTC > AMC13DebuggingHints
Topic revision: r9 - 25 Apr 2019 - EricHazen
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2023 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback