AMC13ProductionTesting < BUCMSPublic

Tags: view all tags

New 2020 Production Test

1. Test Fixture

Install the module in the green v2 testing board connected to the Lambda power supplies
Turn on the power supply connected to the 3.3V pin, check current (should be a few milliamps)
Leave the 3.3V power supply on. Power up 12V, check current (Expect ~0.2A, but may be as high as 1.8A if FLASH memory is programmed as firmware will be running on the FPGAs)
Turn off power (12V then 3.3.V), move the module to the white bench test fixture, and power on the 3.3V
Connect AVR JTAGICE3 cable (connector bends inwards on the front plate when correctly oriented), and program MMC and erase EEPROM (follow these instructions)
Place TTC loop back SFP module into bottom SFP cage
Place three 10G(8G) SFP loop back module into other three cages
Power on 12V on the white test fixture (Make sure fan is underneath is spinning)

2. Memory Test

Connect JTAG cable into FPGA port on AMC13 (as shown here)
Program test stand bit files into AMC13 (instructions for Vivado)
Connect to AMC13 using AMC13Tool2.exe -c serial_no. Firmware should read as 0x1fff and 0xffff if using "new" test bitfiles
Check the following regesters
1. in T2
  1. Check that reg 0x7 (ErrCnt) is 0, or at least not changing
  2. Check that reg 0x4 (TxFB) and reg 0x5 (RxFB) are changing
  3. Check that bit 0 of reg 0x6 (T1_Ready) is 1
2. in T1
  1. Check that reg 0x1f (TTCcntr) is changing
  2. Set bit 13 of reg 0xf to 1 to enable memory test with the following command: wv 0xf 020000
  3. Check that reg 0xa (mem_sta) is changing. Otherwise, there is a memory errer
  4. Register 0xf controls settings of GTX link tests
  5. Change bits[2:0] (AMC_RxSel) and bits[5:3] AMC_TxSel of reg 0xf. Do this with the following command: wv 0xf 0200ab, where legal values of a and b are 1, 2, 3, or 4.
  6. When AMC_RxSel = AMC_TxSel, all AMC error counters should not change (reg 0x10 to reg0x1b) Otherwise, they should change.
  7. Do the same for bits[8:6] (SFP_RxSel) and bits[11:9] (SFP_TxSel) with the following command: wv 0xf 02ab00, where legal values are 1, 2, 3, or 4.
  8. Check SFP error counters (0x1c, 0x1d, and 0x1e)
  9. Set bit 12 of 0xf to 1 to reset all counters: wv 0xf 010000
Do rg and rc to do a general reset and reset counters
Use a scope to look at signals at the terminals of resistors on the test stand. You should see 40MHz LVDS level signals on them.

3. Programming

Using AMC13ToolFlash.exe, program the flash memory: Header, Golden, Spartan, Kintex ( Flash Programming instructions)
1. Header: AMC13T2Header_6slx45t.mcs
2. Golden: T2Goldenv0x33.mcs (need to rename to AMC13T2Golden_6slx45t.mcs)
3. Spartan: AMC13T2v0x0034_6slx45t.mcs
4. Kintex Files: AMC13T1v0x2266_7k325t.mcs
Power cycle the AMC13 to load firmware from flash
Connect to board with AMC13Tool2.exe and check the firmware with fv

4. Local Event Builder Test

Follow instructions here for sending localL1A's and building fake events
1. NOTE: On step 3, run the command >en 1-12 f t instead of >en 1-4 f t

5. AMC Link Test

Move the AMC13 into the center MCH slot of a populated uTCA crate (all AMC slots should be filled with AMC13s that have completed at least all of the steps up to this point. They must be programmed with this T1 firmware)
Run "wget http://ohm.bu.edu/~amc13/allConnectCrate.txt" to download an example of a script that will connect to and configure all the AMC13s in the crate to act as fake AMC devices.
Modify the script to use the serial numbers of the modules you put into AMC slots 1-12. It is most convenient to leave these in their slots for all future modules you test (though if these modules themselves are under test, you will need to do some shifting around when you are down to 13 remaining modules).
Connect to the AMC13 you are testing: $ AMC13Tool2.exe -c serial_no
Run the script to connect to all devices in the crate: >load allConnectCrate.txt
Verify that all devices connected, by looking at the output of the "fv" command a few lines up.
Send 1000 local triggers: >lt 1000
View the event counters for each AMC_Link to ensure that they are all 0x3e8 (or at least the same number): >st 1 AMC_Links

6. Check and Record amc13 Device DNA

Read T1 registers 0x1e and 0x1f and record the values into the board database (if not working, use this as a placeholder)
Read T2 registers 0xe and 0xf and record the values into the board database

7. Pre-Shipping Configuration

Be sure all stickers are in place
Program with the agreed firmware version
Add red dot with owner on it
Fill out and include Shipping Sheet ( found here. You can use this tool to determine the IP and MAC addresses; $ python addressCalc.py -s serial_no. More info on how these addresses are assigned here)
Update board database

AMC13 Bench Test Instructions

1. Test Fixture

Power up Test and MMC Programming
Install the module in the bench test fixture with power off
Power up 3.3V, check current ( ~ 0A )
Connect AVR JTAGICE3 cable and program MMC ( MMC Programming)
Power up 12V, check current (0.2-1.8 A)

2. Run Test

cd into directory containing .mcs firmware files
1. Files can be downloaded from the following links:
2. Header and Golden Files: http://bucms.bu.edu/twiki/bin/view/BUCMSPublic/AMC13FlashProgramming
3. Spartan and Kintex Files: http://ohm.bu.edu/~dzou/firmware.cgi
Set AMC13_ADDRESS_TABLE_PATH to the location of the address tables
1. in bash: export AMC13_ADDRESS_TABLE_PATH=/path/to/address/table
2. in csh: setenv AMC13_ADDRESS_TABLE_PATH /path/to/address/table
Run AMC13BenchTest.exe -s serial_no
Follow the on screen instructions
1. Note, test requires use of AVR studio and impact
2. impact instructions: ( impact instructions)
3. Test stand bit files: ( test bit files)
4. Initial bit file: http://physics.bu.edu/~wusx/download/AMC13/?C=M;O=D

Update the board database with DNA numbers

3. Pre-Shipping Configuration

Program with the agreed firmware version
Install TTC SFP (currently FTLF1318P3BTL)
Install DAQ0 SFP (currently AFBR-709SMZ)
Be sure all stickers are in place
Add red dot with owner on it
Fill out and include Shipping Sheet
Update board database
1. pdf: http://ohm.bu.edu/~hazen/CMS/AMC13/ShippingNotes2014.pdf
2. IP/MAC Address algorithm: http://bucms.bu.edu/twiki/bin/view/BUCMSPublic/IPaddressAssignment

New 2015 Production Test

This is the production testing recipe for testing without using AMC13BenchTest.exe:

1. Test Fixture

Power up Test and MMC Programming
Install the module in the bench test fixture with power off
Power up 3.3V, check current ( ~ 0A )
Connect AVR JTAGICE3 cable and program MMC ( MMC Programming)
Power up 12V, check current (0.2-1.8 A)

2. Memory Test (Replaces Chipscope Test)

Make sure fan is in place next to the AMC13 to prevent overheating
Connect JTAG cable into FPGA port on AMC13
Place TTC loop back SFP module into bottom SFP cage
Place three 10G(8G) SFP loop back module into other three cages
Program test stand bit files into amc13 using iMPACT
1. impact instructions: ( impact instructions)
2. Test stand bit files: ( test bit files)
Connect to amc13 using AMC13Tool2.exe -c serial_no
Check the following regesters
1. in T2
  1. Check that reg 0x7 (ErrCnt) is 0, or at least not changing
  2. Check that reg 0x4 (TxFB) and reg 0x5 (RxFB) are changing
  3. Check that bit 0 of reg 0x6 (T1_Ready) is 1
2. in T1
  1. Check that reg 0x1f (TTCcntr) is changing
  2. Set bit 13 of reg 0xf to 1 to enable memory test with the following command: wv 0xf 020000
  3. Check that reg 0xa (mem_sta) is changing. Otherwise, there is a memory errer
  4. Register 0xf controls settings of GTX link tests
  5. Change bits[2:0] (AMC_RxSel) and bits[5:3] AMC_TxSel of reg 0xf. Do this with the following command: wv 0xf 0200ab, where legal values of a and b are 1, 2, 3, or 4.
  6. When AMC_RxSel = AMC_TxSel, all AMC error counters should not change (reg 0x10 to reg0x1b) Otherwise, they should change.
  7. Do the same for bits[8:6] (SFP_RxSel) and bits[11:9] (SFP_TxSel) with the following command: wv 0xf 02ab00, where legal values are 1, 2, 3, or 4.
  8. Check SFP error counters (0x1c, 0x1d, and 0x1e)
  9. Set bit 12 of 0xf to 1 to reset all counters: wv 0xf 010000
Do rg and rc to do a general reset and reset counters
use a scope to look at signals at the terminals of resistors on the test stand. you should see 40MHz LVDS level signals on them.

3. Erase EEPROM ( instructions link)

Connect Mini USB to amc13, start minicom or putty
Enter command sensread all, check the voltages are ok
Enter command mreset to reboot MMC
Enter command eeperase, ENTER, yes, ENTER, ENTER to erase the EEPROM

4. Programming

Connect JTAB cable to FPGA port
Program initial bit file into Spartan FPGA using iMPACT Initial bit file: http://physics.bu.edu/~wusx/download/AMC13/?C=M;O=D
Calculate the IP Adress of the module
Using AMC13ToolFlash.exe, program the flash memory: Header, Golden, Spartan, Kintex ( Flash Programming instructions)
1. Header and Golden Files: Header and Golden
2. Spartan and Kintex Files: firmware files
Power cycle the amc13 to load firmware from flash
Connect to board with AMC13Tool2.exe and check the firmware with fv

5. Event Builder Test

Follow instructions for sending localL1A's and building fake events
http://bucms.bu.edu/twiki/bin/view/BUCMSPublic/AMC13Tool2Recipes#FakeDataReadout

6. Check and Record amc13 Device DNA

Read T1 registers 0x1e and 0x1f and record the values into the board database
Read T2 registers 0xe and 0xf and record the values into the board database

7. Pre-Shipping Configuration

Program with the agreed firmware version
Install TTC SFP (currently FTLF1318P3BTL)
Install DAQ0 SFP (currently AFBR-709SMZ)
Be sure all stickers are in place
Add red dot with owner on it
Fill out and include Shipping Sheet
Update board database
1. pdf: http://ohm.bu.edu/~hazen/CMS/AMC13/ShippingNotes2014.pdf
2. IP/MAC Address algorithm: http://bucms.bu.edu/twiki/bin/view/BUCMSPublic/IPaddressAssignment

New 2014 Production Test

We're re-working the production test for the 2014 production run. Here is a tentative recipe:

Test Fixture
1. Power up Test and MMC Programming
2. Install the module in the bench test fixture with power off
3. Power up 3.3V, check current ( ~ 0A )
4. Connect AVR JTAGICE3 cable and program MMC ( instructions link)
5. Power up 12V, check current (0.2-1.8 A)
Chipscope Test (in Test Fixture) ( Wu's README instructions)
1. At this point if it doesn't take too long it would make sense to download special test firmware into both T1 and T2 to do a loop-back test on all the backplane connections and test the clock outputs. Otherwise we need to move the board back to this fixture later.
2. Chipscope test files: ( teststand.cpj) ( amc13_t2.bit) ( AMC13_T1.bit)
3. During the Chipscope Test, make sure to position a fan to cool the board. The board can overheat during the test, and is designed to shut off if it gets too hot.
4. Memory Test Run the on-board SDRAM test (part of special Chipscope firmware test)
5. Fiber Test Perform fiber loop-back test (part of special Chipscope firmware test)
MicroTCA Crate
1. Power down and move the board to any slot in the MicroTCA crate
2. The blue LED should illuminate with the handle out.
3. Push the handle in. Check that the blue LED goes out
4. Erase EEPROM ( instructions link)
  1. Connect MiniUSB cable to console connector, start minicom or putty
  2. Enter command mreset to reboot the MMC. Capture all the output log and save it for permanent record
  3. Enter commands eeperase, ENTER, yes, ENTER, ENTER to erase the EEPROM
5. Communications Check and Programming
  1. Connect the JTAG cable and program the initial bit file into the Spartan FPGA ( instructions using iMPACT)
  2. Log on to CMS2, calculate the IP address and ensure you can ping the module
  3. Using AMC13ToolFlash, program the flash memory: Header, Golden, Spartan, Kintex
  4. Power AMC13 to load firmware from flash
  5. Connect to AMC13 using AMC13Tool2 and check firmware with 'fv' command
6. IP address setting check ( MMC software documentations)
  1. Use applyConfig.py to set the IP addresses to a new value and check access
  2. Use storeConfig.py to set the IP addresses to a new value, cycle power and check access
  3. Use storeConfig.py to set the IP addresses to 0.x.x.x, edit the rarp table, and check RARP access
7. Event Builder Test
  1. Follow instructions for sending local L1As w/ loopback and building fake events
  2. http://bucms.bu.edu/twiki/bin/view/BUCMSPublic/AMC13ToolGettingStarted/#EventTestProcedure
8. Check and Record AMC13 Device DNA
  1. Each AMC13 board has an unique Device DNA
  2. Read T1 registers (0x1e) and (0x1f) and record the values listed into the board database (edf sign-in)
```
            rv 0x1e
            rv 0x1f
            
```
Chipscope Test (in Test Fixture) (if not already done in previous step) ( Wu's instructions)
1. (See above)
2. Chipscope test files: ( teststand.cpj) ( amc13_t2.bit) ( AMC13_T1.bit)
Update Board Database w/ test results ( edit) ( read only)
Pre-Shipping Configuration
1. Program with agreed firmware version
2. Install TTC SFP (currently FTLF1318P3BTL)
3. Install DAQ0 SFP (currently AFBR-709SMZ)
4. Be sure stickers are in place (including front panel and T3)
5. If appropriate add a red dot with owner written on it
6. Fill out and include Shipment Sheet (also scan it to keep for records) (links to PDF) ( Notes for IP/MAC address algorithm) ( Example page w/ MAC address for SN57 change last digits in URL for different SN's)

Charlie's Old Production Test

This page documents an initial pre-shipping test procedure for AMC13 modules. It is assumed (for now) that Mr Wu or someone sufficiently qualified has performed an initial programming and test of the module.

Initialize TTC system (only needs to be done once if no one has messed it up)

Press the reset button marked "PROG" on the Xilinx board next to the test stand
Log on to daq@cms1 (password M.....e). Initialize the Xilinx TTC timing generator
1. cd src/TTS_ctrl
2. ./periodic_12hz
3. cd ~/ttc
4. DCCdiagnose.exe -x setup_ttc.dcc

Prepare the AMC13

Install SFP:
1. TTC in bottom site. Connect orange TTC fiber
2. DAQ in top two sites. Connect blue jumper fiber
Install AMC13 in MCH2 (top MCH site in middle of crate)
Power up crate.
Program firmware versions V=0x1f S=0xd, or any more recent version which contains the DAQ link receiver.

Run tests

Log on and go to directory with test scripts (currently ~hazen/AMC13DaqTest but check for updates)
1. It may be best to just make a copy of this directory somewhere writable and work there:
  1. cp -r ~hazen/AMC13DaqTest /tmp/myTests
  2. cd /tmp/myTests
Edit runme.sh if needed to set serial number correctly
Type ./runme.sh AMC13DaqTest.exe with correct serial number (details rapidly changing)
1. Execute do init.amc. This initializes the AMC13 and two uHTR boards
2. Execute rc (reset counters)
3. Execute st (status) several times. Should look "Sample status display" below.
4. Check that under "Ctrl 0" it does not say "TTC not ready". This confirms that the TTC input is working
5. Check that the "AMC Link status" is 04100410. This confirms that the uHTR backplane links are working
Type command rv 4 (read Virtex register 4)
1. Check that low 4 digits are "0111". This confirms that the 3 SFP are correctly installed, and that the optical fibers are connected.
Type do test_with_daq.amc. This runs a test which should initialize the AMC13 and two uHTR and send data through the DAQ fiber.
Type rv 0x10 0x10. This displays "secret" counters which check the DAQ link (see AMC13SecretLDCRegisters). A working link will show a display like "sample LDC registers" below. Note that offset 0x10 and 0x12 have arbitrary counts, while 0x13 and 0x14 should have the values below. All other counters should be zero.

Sample LDC registers

> rv 0x10 0x10
00000010: 000efa26 00000000 000efa3b 0000000a 00000154 00000000 00000000 00000000
00000018: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

Sample status display:

> st
Ctrl 0: 01000010
  memory input fifo for monitored events overflow
Ctrl 1: 00100001
  run mode
AMC Link status: 04100410
Mon buffer page: 00000000  Evts: 0000000f  words: 0000100e
                                               --- AMC13 Event Builder status ---
                          uHTR CRC errors [000f] 00025889 00000000
                                Run time [0048] 00000000 0a1c5092
                                RDY time [004a] 00000000 0a1c8796
                                        --- AMC13 input link status ---
                         AMC Input link:     Link        4    Link       10
                 AMC Counter ACKs [000e] 00000000 000033a1 00000000 000033a1
       AMC ev buf near full times [0016] 00000000 00000005 00000000 00000005
                  Counter Accepts [004e] 00000000 00001435 00000000 00001435

-- EricHazen - 31 Aug 2012

Topic revision: r34 - 14 Sep 2022 - ChristopherCosby