2010-05-20 (Mike)

An error is generated everytime we tried to program firmware to the t2's flash. As a result, a new one is being made. In the mean time, I reconnected the old dtc test board to the v3.3 mch to test the write function that I wrote. Now that both the low-level read and write functions (only deal with one register at a time) are operational, I can move on to writing the multi-register versions as well as 32-bit versions.

2010-05-20 (Mike)

Rolled back to the old firmware -- could not write registers with the new firmware. Safe to write to 0x08, as long as bit 0 is unchanged. Also, successfully removed the front plate from the v3.3 MCH card -- use a screwdriver to pry the plastic handle apart and then pry off the front plate.

2010-05-19 (Mike)

Updated the MCH with firmware sent from NAT. Can no longer access registers on the clock board with

ipmitool -H 192.168.1.10 -P "" -t 0x14 raw 0x30 0x01 0 0 < register >

Hence, the read function that I have written cannot work.

Update: Natview identifies the DTC properly. However, the option for enabling backwards compatibility has been removed from this firmware release. In order to read/write registers, we must use double-bridged messages:

ipmitool -H 192.168.1.149 -P "" -T 0x82 -B 0 -t 0x14 -b 7 raw 0x30 0x01 0 0 < register >

I now have to modify the code that I have written to include these extra tags -- this just involves giving values to some variables that are NULL at this point, so there should not be any trouble.

2010-05-07 (Mike) New MCH arrived. Everything seems to be all right--it doesn't have the firmware problems that the first card came with. The IP address of this new card is 192.168.1.41

2010-05-03 (Mike) Can connect to the board again--had to restart cms1. The code that I have written doesn't want to send messages, however. The issue lies in the sendrecv function. It looks like the ipmitool library sets up a placeholder for this function (see ipmi_intf.h line 184), but the actual function is defined in a series of plugins. I am going through the ipmitool code to figure out how the function allocated by the sendrecv callback is selected. It is not clear to me how this is achieved.

2010-04-26 (Mike) I have written a test program that tests the dtc_read function written with the ipmitool library. If the function works, then it will be simple enough to write the dtc_write function. However, after plugging in the MCH with the DTC addition, I can't even ping the board 192.168.1.10 through cms1. Tried removing the DTC--same problem. Everything seems normal in communication with the board via usb. Need to fix this.

2010-04-05 (Mike) Gave up on OpenIPMI -- even with single bridging, it responded with error messages. Furthermore, the code is more or less opaque to me and the documentation is incomplete. Now working with the ipmitool code, which is open source, simpler, and much easier to understand. Most importantly, it works. There is now a watered-down version of ipmitool called "dtc_ipmi":

~msdim/ipmi/ipmitool-1.8.11/src/dtc_ipmi

The makefile in that location has been edited to make this rather than ipmitool. The ipmitool makefile has been renamed "Makefile_orig".

The next step sould be to start building the library. These are the options that I see: 1. Create a library that utilizes the ipmitool library directly and further modify the Makefile to include the new code. (This is more difficult but potentially offers more funtionality) 2. Simply write it to pass arguments directly to dtc_ipmi (or ipmitool), which will take care of sending the ipmi messages via raw. (Easier to accomplish but not ideal)

I will attempt the first option, which is feasible; the second one is available if time gets short.

2010-03-29 (Mike)

Used mchcfg to enable Backward Compatibility on the MCH. This allows for single bridging--direct access to the DTC. To read registers on the DTC, use:

ipmitool -H 192.168.1.10 -P "" -t 0x14 raw 0x30 0x01 0 0 < register >

2010-03-26 (Mike)

Tried talking to the DTC with IPMI commands listed for ipmitool using the "openipmicmd" program in the "sample" folder of the OpenIPMI directory--while it connected and a response was generated, it was the c1h error message--"invalid command."

The difference is that ipmitool allows for "double bridging," whereas the openipmicmd program doesn't seem to.

When looking through the OpenIPMI source, there is a function called "ipmi_format_msg" (lib/ipmi_payload.c) that controls how messagesare formatted when they are sent out. Perhaps editing this will allow for double bridging.

2010-03-25 (Eric)

Can talk to the module using "raw" IPMI commands with "ipmitool". Using the recipe from NAT's e-mail:

> Reading/writing of registers is realized with a vendor specific 
> command via bridged messages:
>
> ipmitool -H 192.168.1.149 -P "" -T 0x82 -B 0 -t 0x14 -b 7 raw 0x30 
> 0x01 0 0 1
>
> explanation:
> 0x82 is the carrier number (0x80 + (carrier number * 2)
> 0 is the external ipmi bus for communication between shelf manager an 
> carrier manager
> 0x14 is the i2c address of the clock module
> 7 is the internal ipmi bus for communication between mch and clkmod
> 0x30 is the IPMI_NETFN_CONTR0_REQ for vendor control commands
> 0x01 is the command 'read reg' (implemented by NAT, since it isn't 
> documented in the spec, same for write reg).
> parameter 1 and 2 should be 0, parameter 3 is the register number, 
> return value is the content of the register
>
> for write register, the command is 0x02
> parameter 1 and 2 are 0, parameter 3 register number, parameter 4 is 
> the value to be written to the register

If we run the command:

   $ ./ipmitool -H 192.168.1.10 -P "" -T 0x82 -B 0 -t 0x14 -b 7 raw 0x30 0x01 0 0 0

This will read from the address specified as the last parameter (0) and we get the value we expect from one of the addresses documented below!

Up to now only following registers are implemented:

0x01    pcb version (set by DIP switches, read only. You may need to 
send a reset to clock board to get pcb version into CPLD)
0x03   FPGA version( set by vhdl code, 0x34 now, read only)
0x07   FPGA status (read only)
             bits 7-3 all read zero
             bit 2 is SPI mux select signal from the uC
             bit 1 is FPGA INIT_B pin
             bit 0 is FPGA DONE pin
0x08   read/write register.
             bit 7-1 are implemeted but have no function
             bit 0 set to 1 and then back to 0 reprograms FPGA

Shouxiang

2010-03-24 (Mike)

The sample OpenIPMI program detects entities which it then tries to label as FRU's or sensors. It seems to handle the sensors okay (it determines that they measure temperature, voltage, ...), but it cannot recognize FRU's. The following is the output of the program while the code is initializing the domain:

   INFO: 0 ipmi_lan.c(connection_up): Connection 0 to the BMC is up
   INFO: 0 ipmi_lan.c(connection_up): Connection to the BMC restored
   SEVR: oem_atca.c(atca_fru_254_get_timestamp_done): Error fetching the FRU timestamp: 0xd3
   SEVR: oem_atca.c(shelf_fru_fetched): Error getting FRU information: 0x10000d3
   EINF: .1.10.1.0.0.0 fru.c(fru_inventory_area_handler): IPMI error getting FRU inventory area: c9
   SEVR: oem_atca.c(shelf_fru_fetched): Error getting FRU information: 0x10000c9
   SEVR: oem_atca.c(alt_shelf_fru_cb): Error getting alternate FRU information: 0xc9
   WARN: entity.c(decode_frudlr): Error getting device ID string from SDR record 8: 22, this entity will be named **INVALID**

The program might not be able to access the SDR (or misreads it). Natview doesn't have a problem reading the SDR and it displays correct names, etc-- so the information is correct. An attempt to access the SDR directly with the addition of

ipmi_domain_get_main_sdrs(...)

just resulted in a segmentation fault (maybe unrelated).

*2010-03-18 (hazen) *

Added HowtoProgramDTCFirmware link to main page with detailed flash programming instructions.

2010-03-16 (Mike)

I was able to get a simple OpenIPMI program to interface with the board. The program is a trivially (for now) modified version of

  ~msdim/ipmi/OpenIPMI-2.0.14/sample/ipmisample.c

and is run:

   $ ./ipmisample lan 192.168.1.10

The program is very basic--it just loops through the sensors and outputs their readings (temperature, voltage, etc) and waits for any changes.

As for getting/setting registry values, there seems to be some relevant functions in the "ipmi_fru.h" file, namely:

  int ipmi_fru_node_get_field(...)
  int ipmi_fru_node_set_field(...)

The ipmi header files are well commented, so trying to implement the above functions could be a good next step.

2010-03-12 (hazen)

Updated CPLD firmware, hardware mods. Now, temperature is 25C as expected. Will try to program FPGA flash. Need a binary file... assuming this would be a raw binary image of the MCS hex file. Install the "srecord" Debian package on cms1, convert as follows:

  $ srec_cat DTC_test.mcs  -Guess -Output DTC_test.bin -Binary

This produces a binary file which by inspection (od -Ax -v -t x1 file.bin) is identical to the hex file. Let's give it a try!

Doesn't work. Use "diag" menu on MCH, option 9 (CLK board), option 2 (FPGA), get this error:

  ERR unsupported PCB V0.0 or MC version V1.0 

2010-03-11 (hazen)

Can access with ipmitool as follows:

[cms1] ~/NatView/ipmitool-1.8.11/src > ./ipmitool -H 192.168.1.10 -A NONE sdr list all
NAT-MCH-MCMC     | Log FRU @03h c2.61 | ok
Temp CPU         | 28 degrees C      | ok
Temp I/O         | 26 degrees C      | ok
HotSwap          | Not Readable      | ns
Version Change   | Not Readable      | ns
Volt 1.2V        | 1.20 Volts        | ok
Volt 1.5V        | 1.50 Volts        | ok
Volt 1.8V        | 1.80 Volts        | ok
Volt 2.5V        | 2.51 Volts        | ok
Volt 3.3V        | 2.66 Volts        | ok
Volt 12V         | 7.80 Volts        | ok
Base Current     | disabled          | ns
BP-FRU-253       | Phy FRU @A4h d0.61 | ok
BP-FRU-254       | Phy FRU @A4h f2.61 | ok
MCH-DTC          | Log FRU @3Ch c2.63 | ok
Temp Clock       | -1 degrees C      | nc
NMCH-CM          | Log FRU @00h c2.61 | ok
CEN Power Modul  | Log FRU @32h 0a.61 | ok
Hot Swap         | disabled          | ns
Temp Sensor 1    | 206 degrees C     | ok
Temp Sensor 2    | 255 degrees C     | ok
NMCH-ShM         | Dynamic MC @ 20h  | ok
[cms1] ~/NatView/ipmitool-1.8.11/src > 

Now trying to figure out to R/W arbitrary registers. Reading paper from openipmi web page.

2010-03-10 - Spring break, no notes from Mike, on my own

Connect micro-USB cable to eeePC laptop. Install minicom. Run "sudo minicom -s" and set serial port to /dev/ttyACM0 as gleaned from dmesg output. Works fine, can talk to console.

Instructions to program clock module firmware:

  Unzip the file (clkmod.zip) and store it to a tftp-server. Use the diag menu of the MCH firmware:
  diag -> 10 ->2 ->2 -> IP:/folder/clkmod.hex 

Instructions to program FPGA flash via SPI:

  diag-> 9 (CLOCK PCB menu) -> 1 (FPGA update) -> 7 (program file to FLASH) -> enter Loops: 1 -> IP:/folder/file
  Only binary formats are supported (e.g. for Altera *.rpd files or *.bin files for Lattice). 

Try to program AVR. Doesn't work:

DIAG (RET=0/0x0): 10
  [ 0] : no action (unsupported)
  [ 1] : (submenu) AVR PROG menu 
  [ 2] : program firmware to AVR flash 
  [ 3] : verify firmware in AVR flash 
  [ 4] : show board information 
  [ 5] : set board information 
  [ 6] : reset AVR device 
  [ ?] : ?: help 
  [ h] : h: help 
  [ q] : q: quit submenu 
AVR (RET=0/0x0): 2
Choose AVR device:
   LED module AVR      : 0
   HS AVR              : 1
   CLOCK PCB AVR       : 2
   HUB PCB AVR         : 3
AVR device (RET=2/0x2): 2
Enter host and file name of AVR firmware hex-image
[IP:FILENAME]:>192.168.1.1:clkmod.hex
TFTP: getting BIN file:
  192.168.1.1:clkmod.hex
.....
TFTP: getting file done (28916 bytes)
avr_prog_enable: ERR retry timeout
AvrProg_Init(2): ERR avr_prog_enable failed
avr_prog_init(2): ERR AvrProg_Init failure -6
Nmch_AvrProgBuf(2): ERR avr_prog_init failure -3
Nmch_AvrProgFile(2): ERR Nmch_AvrProgBuf failure -3

Update MCH firmware to latest using update_firmware with nmch_mch.bin file just received. Power-cycle after. MCH comes up ok, but still can't program AVR.

OK, found the problem. The nRESET pin on the AVR is connected to the wrong pin (HUB card reset instead of CLK card reset) on the 28-pin mezzanine connector. If we select '3' (HUB card) then we can program the AVR.

2010-01-25 - Working with Mike. Tried to connect to NAT-MCH using USB console. Worked once on Mike's vista laptop, then not again.

Try again on L3 XP machine. First time USB device is unrecognized, but observe that Mike has downloaded the nat-mch.inf file. Use the device manager to load this driver file, then it works using teraterm. Only problem seen is that one must exit teraterm before power cycling the uTCA crate, otherwise it won't connect.

The recover procedure which seems to always work is to exit Teraterm, then (with uTCA power on) unplug the USB cable, count to 5, then replug and count to 5. Then start Teraterm and select 'com4'. This has now worked many times.

2010-01-26 - Still can't connect via ethernet. Using eth1 with x-over cable on cms1. Capture packets with nmap... see arp request with no replay. Manually add to arp tables with:

sudo /sbin/arp -s 192.168.1.41 00:e0:81:32:12:b3

Still fails. Try cabling to regular switch on 43 subnet, assign unused I/P address 'cms3.bu.edu'. Same problem. Have to take a break for now.

10am - still no luck. Tried tracking with wireshark... see a 'Ping' sent to port 623, reply is 'destination unreachable' Try C client/server pair on port 623 and it works but requires sudo. Try running natview with sudo... no luck. Also try ipmitool (normal and sudo) with same results.

Back to private network. Set I/P = 192.168.1.10. Change cables. Switch from long "Cat 5" (cross-over?) cable to 15' "Cat 6" new yellow cable. Ping works, so no need for crossover. Same problem.

-- EricHazen - 26 Jan 2010