Difference: AMC13DebugLog (1 vs. 165)

2022-09-06, Chris Cosby

Monitoring script running currently to detect when bad states show up overnight, with corresponing 8b-10b error info. Will update tomorrow.

2022-09-02, Chris Cosby

Tests done following TTTTtestInstructions

Test of "old" firmware (0x2264, 0x2e) running for several minutes outputting only TTS test pattern shows no anomolous registers counting

edf@raspberrypi:~ $ python3 ttttControl.py ttsReport
Iterating through TTS counter registers (100-1FF), searching for non-zero entries...
non-zero counter found on TTS reg 10f: 0x00000000000000FF
Done!

With the same firmware, we switched to running with data from the fake AMC devices in slots 1-4. TTTT shows expected counters climbing (1=Overflow_warning, 8=Ready), and a slowly counting entry on an additional register (9=Private_Request_1)

edf@raspberrypi:~ $ python3 ttttControl.py resetCount
TCDS counters reset
edf@raspberrypi:~ $ python3 ttttControl.py ttsReport
Iterating through TTS counter registers (100-1FF), searching for non-zero entries...
non-zero counter found on TTS reg 101: 0x00000000000000FF
non-zero counter found on TTS reg 108: 0x00000000000000FF
Done!
edf@raspberrypi:~ $ python3 ttttControl.py ttsReport
Iterating through TTS counter registers (100-1FF), searching for non-zero entries...
non-zero counter found on TTS reg 101: 0x00000000000000FF
non-zero counter found on TTS reg 108: 0x00000000000000FF
non-zero counter found on TTS reg 109: 0x0000000000000006
Done!
edf@raspberrypi:~ $ python3 ttttControl.py ttsReport
Iterating through TTS counter registers (100-1FF), searching for non-zero entries...
non-zero counter found on TTS reg 101: 0x00000000000000FF
non-zero counter found on TTS reg 108: 0x00000000000000FF
non-zero counter found on TTS reg 109: 0x0000000000000011
Done!
edf@raspberrypi:~ $ python3 ttttControl.py ttsReport
Iterating through TTS counter registers (100-1FF), searching for non-zero entries...
non-zero counter found on TTS reg 101: 0x00000000000000FF
non-zero counter found on TTS reg 108: 0x00000000000000FF
non-zero counter found on TTS reg 109: 0x0000000000000017
Done!
edf@raspberrypi:~ $ python3 ttttControl.py ttsReport
Iterating through TTS counter registers (100-1FF), searching for non-zero entries...
non-zero counter found on TTS reg 101: 0x00000000000000FF
non-zero counter found on TTS reg 108: 0x00000000000000FF
non-zero counter found on TTS reg 109: 0x000000000000003C
Done!
edf@raspberrypi:~ $ python3 ttttControl.py ttsReport
Iterating through TTS counter registers (100-1FF), searching for non-zero entries...
non-zero counter found on TTS reg 101: 0x00000000000000FF
non-zero counter found on TTS reg 108: 0x00000000000000FF
non-zero counter found on TTS reg 109: 0x0000000000000042
Done!
edf@raspberrypi:~ $ python3 ttttControl.py ttsReport
Iterating through TTS counter registers (100-1FF), searching for non-zero entries...
non-zero counter found on TTS reg 101: 0x00000000000000FF
non-zero counter found on TTS reg 108: 0x00000000000000FF
non-zero counter found on TTS reg 109: 0x0000000000000050
Done!

These were read out over the course of a couple minutes, about 10-30 seconds apart.

Repeating these tests with current firmware (0x2266, 0x34) shows the same results when sending the test pattern:

edf@raspberrypi:~ $ python3 ttttControl.py ttsReport
Iterating through TTS counter registers (100-1FF), searching for non-zero entries...
non-zero counter found on TTS reg 10f: 0x00000000000000FF
Done!

and the same counters climbing when using real AMC devices for data

edf@raspberrypi:~ $ python3 ttttControl.py resetCount
TCDS counters reset
edf@raspberrypi:~ $ python3 ttttControl.py ttsReport
Iterating through TTS counter registers (100-1FF), searching for non-zero entries...
non-zero counter found on TTS reg 101: 0x00000000000000FF
non-zero counter found on TTS reg 108: 0x00000000000000FF
non-zero counter found on TTS reg 109: 0x0000000000000001
Done!
edf@raspberrypi:~ $ python3 ttttControl.py ttsReport
Iterating through TTS counter registers (100-1FF), searching for non-zero entries...
non-zero counter found on TTS reg 101: 0x00000000000000FF
non-zero counter found on TTS reg 108: 0x00000000000000FF
non-zero counter found on TTS reg 109: 0x0000000000000004
Done!
edf@raspberrypi:~ $ python3 ttttControl.py ttsReport
Iterating through TTS counter registers (100-1FF), searching for non-zero entries...
non-zero counter found on TTS reg 101: 0x00000000000000FF
non-zero counter found on TTS reg 108: 0x00000000000000FF
non-zero counter found on TTS reg 109: 0x0000000000000005
Done!
edf@raspberrypi:~ $ python3 ttttControl.py ttsReport
Iterating through TTS counter registers (100-1FF), searching for non-zero entries...
non-zero counter found on TTS reg 101: 0x00000000000000FF
non-zero counter found on TTS reg 108: 0x00000000000000FF
non-zero counter found on TTS reg 109: 0x000000000000000D
Done!
edf@raspberrypi:~ $ python3 ttttControl.py ttsReport
Iterating through TTS counter registers (100-1FF), searching for non-zero entries...
non-zero counter found on TTS reg 101: 0x00000000000000FF
non-zero counter found on TTS reg 108: 0x00000000000000FF
non-zero counter found on TTS reg 109: 0x0000000000000023
Done!
edf@raspberrypi:~ $ python3 ttttControl.py ttsReport
Iterating through TTS counter registers (100-1FF), searching for non-zero entries...
non-zero counter found on TTS reg 101: 0x00000000000000FF
non-zero counter found on TTS reg 108: 0x00000000000000FF
non-zero counter found on TTS reg 109: 0x0000000000000025
Done!
edf@raspberrypi:~ $ python3 ttttControl.py ttsReport
Iterating through TTS counter registers (100-1FF), searching for non-zero entries...
non-zero counter found on TTS reg 101: 0x00000000000000FF
non-zero counter found on TTS reg 108: 0x00000000000000FF
non-zero counter found on TTS reg 109: 0x000000000000002F
Done!

  # /sbin/ifdown eth1

ok so I’ve tried various methods now to configure and send BGO commands from AMC13

#

File 1:run000039_ls0001_index000001.raw

EvN 000001 (000001) BcN 19c ID 0064 Evt_ty=1   [1 blocks, 0x9 words] 

  Slink_Express|        SFP0|

  daq 2

evb::BU_0

>lspci -v

% sudo yum erase “daq-*”

% sudo yum groupinstall extern_coretools

Using AMC13 software ver:41528
>rg

rg

21 Jan 2016 17:48:11.625 [139692899636992] FATAL edu.bu.cms1.p: 33001.pt::frl::Application.instance(1) <> - Caught exception: pt::frl::exception::Exception 'Overflow of i2o: 32696' raised at processCopy(/usr/local/src/xdaq/baseline12/trunk/daq/pt/frl/src/common/CopyWork er.cc:301)
21 Jan 2016 17:48:11.626 [139693435836160] ERROR edu.bu.cms1.p: 33001.executive::Application.lid(0) <toolbox/exception/Processor> - Unhandled exception occured

Caught exception: exception::DiskWriting 'Failed to get value from EVM at http://cms1.bu.edu: 33001/urn:xdaq-application:lid=12/eventCountForLumiSection?ls=3: Couldn't resolve host name' raised at getValueFromEVM(/usr/local/src/xdaq/baseline12/trunk/daq/evb/src/common/bu/RUproxy.cc:539)

21 Jan 2016 15:35:54.755 [139942756808448] FATAL edu.bu.cms1.p: 33001.evb::BU.instance(0) <> - Failed: Caught exception: exception::DiskWriting 'Failed to get value from EVM at http://cms1.bu.edu: 33001/urn:xdaq-application:lid=12/eventCountForLumiSection?ls=3: Couldn't resolve host name' raised at getValueFromEVM(/usr/local/src/xdaq/baseline12/trunk/daq/evb/src/common/bu/RUproxy.cc:539)
21 Jan 2016 15:35:54.755 [139942756808448] WARN  edu.bu.cms1.p: 33001.evb::BU.instance(0).RcmsStateNotifier <> - Unable to notify state change. RCMS state listener has not been found. Has findRcmsStateListener() been called at least once? State:Failed
21 Jan 2016 15:35:54.760 [139943433701120] ERROR edu.bu.cms1.p: 33001.executive::Application.lid(0) <toolbox/exception/Processor> - Unhandled exception occured

>lt 1

19 Jan 2016 18:00:32.963 [140717694822144] ERROR edu.bu.cms1.p: 33001.executive::Application.lid(0) <toolbox/exception/Processor> - Unhandled exception occured
Caught exception: pt::frl::exception::Exception 'failed to process copy for stream on index: 0 for copy worker id:0 - current frl event queue elements: 2 last processed input event dump: [[00000000]   47 5A 00 00 80 00 01 FE   00 00 00 64 00 0 0 00 06   GZ...... ...d....

19 Jan 2016 16:15:37.684 [140287254984448] FATAL edu.bu.cms1.p: 33001.pt::frl::Application.instance(1) <> - Caught exception: pt::frl::exception::Exception 'Overflow of i2o: 32696' raised at processCopy(/usr/local/src/xdaq/baseline12/trunk/daq/pt/frl/src/common/CopyWork er.cc:301)
19 Jan 2016 16:15:37.685 [140287800329984] ERROR edu.bu.cms1.p: 33001.executive::Application.lid(0) <toolbox/exception/Processor> - Unhandled exception occured

>rg

[dan@cms1 FEROL]$ /opt/xdaq/bin/fedKit.py 

>lt c

reconfigureFPGAs

rg

fedKit.py 

  lt 5

>rg

[daq@cms1 FEROL]$ fedKit.py 

  > wv CONF.AMC.FAKE_DATA_SIZE 0x100       # small size for single blocks
  > lt 0x3fe                               # generate 1k - 2 triggers
  > st 2 Monitor_Buffer                    # two empty buffers (0x400 - 2)

  06:04.0 Memory controller: Device ecd6:fea0 (rev 01)

$ source /home/daqowner/dist/etc/env.sh
$ perl installDAQ_12_4_9.perl --mode=teststand   --ownsource=${HOME}/src/12_4_9   --svnuser=ehazen   --packages=all

    // Enable calibration events in orbit gap if config doc says so                                                                                                              

Copy ISO to both CDROM and USB stick

  $ AMC13Tool2.exe -c 192.168.1.82

Nmap scan report for rfc1918.address.not.used.bu.edu (192.168.2.56)

  bits 0-5     complement of serial number bits 0-5

Nmap scan report for rfc1918.address.not.used.bu.edu (192.168.2.56)

1. Reload AMC13 firmware
2. Disable uHTR DAQ Path
3. Power cycle modules

1. Reload firmware for module using software (in principle this should have the effect as powercycle)
2. Rerun initialization

   $ sudo yum install perl-Tree-Simple

  if ( SLOT == 2 ) {

  localTtcSignal=true

  sudo /etc/init.d/controlhub start

1. singleBit_OnOff is a method that checks bits and populates a vector of strings indicating with AMC ports have their relevant bit set (in 0-based enumation). All Status display items from AMC Link Status through Bc0 Status use this to enumerate. (Actions.cc, line 2625)

1. 1. 1. (ss bit shift by i+1? and the if statement to i<=10) or

1. ctr64_status_amc is the method use in AMC Counters. (Actions.cc, line 2657, ena_amcs[i] )

ping 192.168.2.2

*****AMC13 Status*****

uHTR status and uHTR DTC status:

Unable to enter programming mode. The read device ID does not match the slected device or any other supported devices. 

Pick an action (h for menu): st

amc13> i 3 0x20000         initialize two inputs for fakes length 0x20000
amc13> t                   generate one trigger
amc13> r 0xd               read size

amc13>rd 0x20000 0x20

000004: 010000011f420003    AMC1 header length=0x20003 (upper byte incorrect)

022002: 3ff73ff63ff53ff4

amc13>rd 0x24004 0x20

amc13>w 0xc 0
amc13>rd 0x20000 0x20

amc13>rd 0x22000 0x20

amc13>rd 0x24000 0x20

amc13>rd 0x20000 0x20

amc13>w 0xc 0
amc13>r 0xd

1. Check out a test copy somewhere cgi scripts can be run (David can to this in his public_html on ohm)

1. Point browser at the copy you have checked out to be sure it works, e.g.:

1. Edit the file ...boards/tools/CMSConf.pm

1. Modify the database itself

1. 1. enter command
      $ sqlite3 test.db
   2. enter commands
      sqlite> alter table amc13 add column dna varchar(30)
      =sqlite3> .quit=
      (e.g.) to add column "dna"

 > ./storeConfig.py --slot=1 --ip=192.168.1.148

[cms2] /home/hazen > telnet 192.168.1.41

nat> show_fru

FRU Information:

# net interface 0

1. Install the VadaTech module in the MCH2
2. Initial attempt at IPbus-based communication with the uTCA crate results in successful pinging and control of the AMC13 in MCH1 and also the AMC13 in a non-MCH slot, but unsuccessful pinging of mCTR2s
3. Relevant Documents are...

1. 10/100 Ethernet has the default IP address 192.168.1.252, which conflicts with our IP assignment scheme for the AMC13s.
2. According to the 'Getting Started Guide', the default IP address of the GbE is 192.168.40.250. Unsuccessful ping to this address.
3. Run Eric's pinger.py to see what I can talk to. Within the range 192.168.1.255.......1, I can only see the AMC13s. This was to check and make sure the VadaTech MCH card didn't magically take on the same IP address as the NAT MCH card
4. Despite my not being able to ping the GbE, I am going to try and connect anyway, as the documentation suggests.

1. Next try connecting via the 10/100 Ethernet Port at 192.168.1.252. This pings successfully!
2. Eric has now take over.
3. TelNet into the card successfully.
   • > telnet 192.168.1.252

1. The HUB card will be running Linux. The command-line interface is based on the IPMI v2.0. The procedure to reassign the IP addresses of the ethernet ports is as follows
   1. Open the file /etc/rc.d/rc.conf for editing
   2. net interface 0 is the 10/100 Ethernet port. Edit IPADDR0 and/or NETMASK0 and/or BROADCAST0 as desired
   3. net interface 1 is the GbE port. Edit IPADDR1 and/or NETMASK1 and/or BROADCAST1 as desired
   4. Only one of either GATEWAY0 or GATEWAY1 should be set!! The MCH will use the device with the set 'GATEWAY' value to send traffic to other subnets and networks
   5. Power cycle the MCH for the changes to take effect

export IPMITOOLARGS="-H 192.168.1.11 -P \"\" -T 0x82 -B 0 -b 7"

# set spartan address low byte to 128
> ipmitool $IPMITOOLARGS -t 0xa4 raw 0x32 0x33 0 0 7 1 0x80

> spy

Starting to read file...

AMC13 EvN = 0x00000012  uHTR(10) EvN = 0x00451112 !!

000770 deadbeef 0000001a 1d500008 51000012

0x10 LDC_accept_cntr

  NET "*/DAQ_Link_wu/UsrClk" TNM_NET = "DAQ_UsrClk";

  1) Program the 4.02 firmware bitfile from JTAG.

[cms2] /home/hazen/work/uHTR_Firmware > telnet 192.168.1.11

1. b10 0000000000000a00 a00003891aca0000

AMC13_AddressTable_S6/V6.txt

endRun()

AMCInputEnable()

nextEventSize()

readNextEvent()

To install on a teststand (as daqowner) :

  pwr_on [fru #]

  unzip the e-mail attachement somewhere

2016-11-01, Gastler

Packets sent to them from outside the 192.168.1.* network are received, but can't be routed back since the netmask is 24.

Modified the netmask to 16 on the vadatech MCH and things are working again.

The modification was in /etc/rc.d/rc.conf.

The NAT MCH seems to have a bug in it that keeps it from correctly using the 255.255.0.0 (16) netmask correctly.

Due to this, you can only talk to it via a 192.168.1.* addressed machine (cms2, cms3, cms4)

2016-10-31, hazen

Can't access any microTCA stuff. Pings to MCH go unanswered. Disconnect local Ethernet cable on cmssun1 from local switch, move to direct connection to Vadatech MCH. No luck. Maybe because IP address is 192.168.10.19?

Edit /etc/sysconfig/network-scripts/ifcfg-eth1 to change IP temporarily to 192.168.1.5.

  # /sbin/ifdown eth1
  # /sbin/ifup eth1

Now I can access the Vadatech MCH. I think the problem is maybe the netmask or something like that. Probably Dan's DHCP experiments

Now Johnni says:

ok so I’ve tried various methods now to configure and send BGO commands from AMC13
but I fail completely.

I’m using version: 0x2255 and 0x02e.

Have you tested these versions in the lab recently to make sure you are still able
to configure BGO commands such as Resynch (0x4) and see them in the TTC history?

Reprogram S/N 161 with 0x225b since it's the latest. Create a little script for Jonni:

#
# demo BGO
#
# enable AMC13 (needed to start TTC output)
en 1 f t
# channel 0, short command 0x1c
# every 1000 orbits, repeating
#
bgo 0 short 0xc0 125 1000 repeat
#
# now set up the history filter to exclude BC0
#
ttc f s 0 0x01 0xfe
ttc f on
#
# clear the history and enable
#
ttc h clr
ttc h on
#
sleep 0.5
#
# display the history, last 10 entries
#
ttc h d

It works on 0x225d.

2016-05-05, hazen

Trying to do tests. Many cards in "new" crate do not respond. Cycle power and re-scan. Two boards are dead. Slot 6 (S/N 309) and Slot 8 (S/N 285).

Can't set IP addresses with applyConfig and storeConfig. Take 309 to white test stand. Reprogram T2 with bit file using JTAG. Use AMC13ToolFlash to program Spartan and Kintex. Still doesnt' work.

Removed 309 and 285 to be dealt with later.

2016-04-19, djarcaro

edit: Updated the unpacker class in amc13 to version 0x2.

2016-04-15, djarcaro

Able to send events from SN85 to the FEROL with fake data enabled and a rate of ~10kHz with the local o 1 1 setting. All the payloads have size 1. Next trying the same thing but with 258 producing fake events.

Used 258 to generate fake events for 86 and the FEROL successfully built the events. Dumps below.

From the FEROL:

File 1:run000039_ls0001_index000001.raw
EvN 000191 (000401) BcN 5ea ID 0064 Evt_ty=1   [1 blocks, 0x9 words] 
EvN 000192 (000402) BcN 2b6 ID 0064 Evt_ty=1   [1 blocks, 0x9 words] 
EvN 000193 (000403) BcN 313 ID 0064 Evt_ty=1   [1 blocks, 0x9 words] 
EvN 000194 (000404) BcN 455 ID 0064 Evt_ty=1   [1 blocks, 0x9 words] 
EvN 000195 (000405) BcN 588 ID 0064 Evt_ty=1   [1 blocks, 0x9 words] 
EvN 000196 (000406) BcN 5f6 ID 0064 Evt_ty=1   [1 blocks, 0x9 words]

From the AMC13:

EvN 000001 (000001) BcN 19c ID 0064 Evt_ty=1   [1 blocks, 0x9 words] 
EvN 00c351 (050001) BcN a5f ID 0064 Evt_ty=1   [1 blocks, 0x9 words] 
EvN 0186a1 (100001) BcN 5e2 ID 0064 Evt_ty=1   [1 blocks, 0x9 words] 
EvN 0249f1 (150001) BcN be8 ID 0064 Evt_ty=1   [1 blocks, 0x9 words] 
EvN 030d41 (200001) BcN 55c ID 0064 Evt_ty=1   [1 blocks, 0x9 words] 
EvN 03d091 (250001) BcN 5da ID 0064 Evt_ty=1   [1 blocks, 0x9 words]

  Slink_Express|        SFP0|
                --|------------|
 BACKPRESSURE_TIME| 0x14D8BA436|
            BLOCKS| 0x    55BC8|
       BLOCKS_SENT| 0x    55BC8|
       EVENTS_SENT| 0x    55BC8|
       INITIALIZED|           1|
       LINKNOTFULL|           0|
           LINK_UP|           1|
   NO_BACKPRESSURE|           1|
      PACKETS_RCVD| 0x    55BE2|
      PACKETS_SENT| 0x   8131E2|
          REVISION| 0x 5E100001|
      SFP_LSC_DOWN|           0|
         TEST_MODE|           0|
             WORDS| 0x   303A08|
        WORDS_SENT| 0x   303A08|

     State_Timers|       COUNT| PERCENT|
               --|------------|--------|
             BUSY| 0x   100001|        |
       L1A_IN_OFW|           3|        |
 OVERFLOW_WARNING| 0x 497B9E52|        |
            READY| 0x82F6C3C36|     100|
              RUN| 0x878F7DA7F|        |

           T2_TTC| BCNTERR|
               --|--------|
            VALUE|       1|

           TTC_Rx|       ENC|       RAW|
               --|----------|----------|
            STATE| RDY (0x0)| RDY (0x8)|           

2016-04-14, djarcaro

2016-03-29, hazen

2016-03-02, hazen

Checking on MultiFED readout software. Tried first with test version 0x24a in S/N 161 and it behaved badly. Reprogramming now to HCAL 0x4043. Test seq:

  daq 2
  wv CONF.EVB.ENABLE_DAQLSC 0     # turn off actual fiber output
  en 1,7 f t
  lt 5 100
  st
  re

This "works" but only one monitor buffer is read, not the two as expected. Not sure what I'm doing wrong. Diving into the code.

Tried 8x larger events and 8x slower rate and xdaq died:

evb::BU_0
InfoSpaces: ApplicationMonitoring
run 102Version 3.7.4
FailedPage last updated: Thu Feb 18 17:16:26 2016 UTC
Caught exception: exception::DiskWriting 'Failed to get value from EVM at http://cms1.bu.edu: 33001/urn:xdaq-application:lid=12/eventCountForLumiSection?ls=25: Couldn't resolve host name' raised at getValueFromEVM(/usr/local/src/xdaq/baseline12/trunk/daq/evb/src/common/bu/RUproxy.cc:553)

2016-02-18, dgastler

2016-02-18, dgastler

Disabling parsing as before on the FEROL, I was able to get the AMC13 to send ~11kByte events at a rad of about 110kHz, which leads to a data rate of 1.23GBps, which seems okay.

There are now no register on the FEROL end that this can be cross-referenced with, so this is entirely based on the fixed fake event size and the rate that the AMC13 reports data going at.

2016-02-12, dgastler

Dominique instructed me on how to get the FEROL to drop the data when it lands and now the CPU use has dropped dramatically and the throughput went way up.

Here are the commands to throw away the data:

>lspci -v
        02:03.0 Memory controller: CMS DAQ group Device fea0 (rev 01)
        Flags: 66MHz, slow devsel
        Memory at dd210000 (32-bit, non-prefetchable) [size=64K]
        Memory at dd200000 (32-bit, non-prefetchable) [size=64K]
>sudo busybox devmem 0xdd210000 32 0

With 1952 Byte events, I was able to max out the AMC13's random event rate at 130kHz.

To try to max out the link, I increased the event size until I got backpressure. This happened at FAKE_DATA_SIZE of 0x40 (6560bytes) with a max rate of about 75kHz.

This gives a throughput of 500MBps, which is what we would expect.

2016-02-10, dgastler

I was seeing a low max rate ~200Hz on 25kB event fragments, so I wanted to investigate.

Double checking my math for event sizes by using both the AMC13's read event and the FEROL's write to disk I have verified my event fragment size equations for the AMC13's fake data generator.

((AMC13_FAKE_DATA_SIZE+3)*nAMCs + (4+nAMCs))*8 Bytes per event fragment.

Choosing 0x10 for the fake data size gives 1952byte events.

Sending this to the FEROL and increasing the rate from 10hz, I didn't see any time in L1A_IN_OFW or OVERFLOW_WARNING until setting the AMC13 to 50khz.

With these settings the AMC13 reports an effective rate of about 30khz.

2016-02-10, dgastler

To get a fresh start on the xdaq software, I purged cms1 of xdaq using:

% sudo yum erase “daq-*”
% sudo yum clean all

CMS1 is already pointing to the xdaq 12 repo, so after the purge, I did the following to re-install xdaq

% sudo yum groupinstall extern_coretools
% sudo yum groupinstall coretools
% sudo yum groupinstall extern_powerpack
% sudo yum groupinstall powerpack
% sudo yum groupinstall database_worksuite
% sudo yum groupinstall general_worksuite
% sudo yum groupinstall hardware_worksuite

After this, the xpci driver no longer worked. I installed all daq-ferol* and daq-xpci* packages from the repo to no effect.

This is because the package that is installed is for another kernel and we now (unlike before) have to compile the driver from source (https://twiki.cern.ch/twiki/bin/viewauth/CMS/CMSFedkitManual)

At this point the fedKit.py script will run and write to disk (most of the time) and for only certain sizes of events.

Here is a working AMC13 sequence:

Using AMC13 software ver:41528
>rg
General reset
>rc
Counter reset
>en 1-12 t f
parsed list "1-12" as mask 0xfff
Enabling TTS as TTC for loop-back
Enabling fake data
AMC13 out of run mode
AMC13 is back in run mode and ready
>wv CONF.AMC.FAKE_DATA_SIZE 0x10
Write to register CONF.AMC.FAKE_DATA_SIZE
>daq 1
SFP0 enabled
Best to do a DAQ reset (rd) after changing link settings
>fed 0 100
>lt c
>wv CONF.AMC.FAKE_DATA_SIZE 0x20
Write to register CONF.AMC.FAKE_DATA_SIZE

Increasing the event size worked until 0x100, where the system stopped recording data.

2016-01-21, dgastler

First, run the fedKit.py with no writing to disk, but set it to dump the next several events "e100".

Then run this script on the connected AMC13

rg
rc
en 1-12 f t
localL1A o 1 1
daq 1
fed 0 100

#size if bytes: ((( CONF.AMC.FAKE_DATA_SIZE + 4)*12) + (2+1+1+12))*8
#2048
wv CONF.AMC.FAKE_DATA_SIZE 0x10
lt 1

#25088
wv CONF.AMC.FAKE_DATA_SIZE 0x100
lt 1
#26624
wv CONF.AMC.FAKE_DATA_SIZE 0x110
lt 1
#28160
wv CONF.AMC.FAKE_DATA_SIZE 0x120
lt 1
#29696
wv CONF.AMC.FAKE_DATA_SIZE 0x130
lt 1
#31232
wv CONF.AMC.FAKE_DATA_SIZE 0x140
lt 1
#32768
wv CONF.AMC.FAKE_DATA_SIZE 0x150
lt 1
#34304
wv CONF.AMC.FAKE_DATA_SIZE 0x160
lt 1
#35840
wv CONF.AMC.FAKE_DATA_SIZE 0x170
lt 1
#37376
wv CONF.AMC.FAKE_DATA_SIZE 0x180
lt 1

This will generate increasing events until the "copy worker" dies with the error here:

21 Jan 2016 17:48:11.625 [139692899636992] FATAL edu.bu.cms1.p: 33001.pt::frl::Application.instance(1) <> - Caught exception: pt::frl::exception::Exception 'Overflow of i2o: 32696' raised at processCopy(/usr/local/src/xdaq/baseline12/trunk/daq/pt/frl/src/common/CopyWork\
er.cc:301)
21 Jan 2016 17:48:11.626 [139693435836160] ERROR edu.bu.cms1.p: 33001.executive::Application.lid(0)  - Unhandled exception occured
Caught exception: pt::frl::exception::Exception 'failed to process copy for stream on index: 0 for copy worker id:0 - current frl event queue elements: 3 last processed input event dump: [[00000000]   F8 01 F9 01 FA 01 FB 01   FC 01 FD 01 FE 01 FF 01   ........ ........
[00000010]   00 02 01 02 02 02 03 02   04 02 05 02 06 02 07 02   ........ ........
[00000020]   08 02 09 02 0A 02 0B 02   0C 02 0D 02 0E 02 0F 02   ........ ........
[00000030]   10 02 11 02 12 02 13 02   14 02 15 02 16 02 17 02   ........ ........
[00000040]   18 02 19 02 1A 02 1B 02   1C 02 1D 02 1E 02 1F 02   ........ ........
[00000050]   20 02 21 02 22 02 23 02   24 02 25 02 26 02 27 02   ........ ........
[00000060]   28 02 29 02 2A 02 2B 02   2C 02 2D 02 2E 02 2F 02   ........ ........
[00000070]   30 02 31 02 32 02 33 02   34 02 35 02 36 02 37 02   0.1.2.3. 4.5.6.7.
[00000080]   38 02 39 02 3A 02 3B 02   3C 02 3D 02 3E 02 3F 02   8.9..... ........
[00000090]   40 02 41 02 42 02 43 02   44 02 45 02 46 02 47 02   ..A.B.C. D.E.F.G.
[000000a0]   48 02 49 02 4A 02 4B 02   4C 02 4D 02 4E 02 4F 02   H.I.J.K. L.M.N.O.
[000000b0]   50 02 51 02 52 02 53 02   54 02 55 02 56 02 57 02   P.Q.R.S. T.U.V.W.
[000000c0]   58 02 59 02 5A 02 5B 02   5C 02 5D 02 5E 02 5F 02   X.Y.Z... ........
[000000d0]   60 02 61 02 62 02 63 02   64 02 65 02 66 02 67 02   ..a.b.c. d.e.f.g.
[000000e0]   68 02 69 02 6A 02 6B 02   6C 02 6D 02 6E 02 6F 02   h.i.j.k. l.m.n.o.
[000000f0]   70 02 71 02 72 02 73 02   74 02 75 02 76 02 77 02   p.q.r.s. t.u.v.w.
]' raised at process(/usr/local/src/xdaq/baseline12/trunk/daq/pt/frl/src/common/CopyWorker.cc:583);
        originated by pt::frl::exception::Exception 'Overflow of i2o: 32696' raised at processCopy(/usr/local/src/xdaq/baseline12/trunk/daq/pt/frl/src/common/CopyWorker.cc:301)

The last working event was event 6 which was 31232 bytes long. (0x140 FAKE_DATA_SIZE) The next event crashes the

2016-01-21, dgastler

https://twiki.cern.ch/twiki/bin/viewauth/CMS/UFEDKIT#Firmware_update

When using the fetKit.py write to disk option, it doesn't always start, and when it does, it won't write to disk and the "BU" xdaq icon has the error:

Caught exception: exception::DiskWriting 'Failed to get value from EVM at http://cms1.bu.edu: 33001/urn:xdaq-application:lid=12/eventCountForLumiSection?ls=3: Couldn't resolve host name' raised at getValueFromEVM(/usr/local/src/xdaq/baseline12/trunk/daq/evb/src/common/bu/RUproxy.cc:539)

21 Jan 2016 15:35:54.755 [139942756808448] FATAL edu.bu.cms1.p: 33001.evb::BU.instance(0) <> - Failed: Caught exception: exception::DiskWriting 'Failed to get value from EVM at http://cms1.bu.edu: 33001/urn:xdaq-application:lid=12/eventCountForLumiSection?ls=3: Couldn't resolve host name' raised at getValueFromEVM(/usr/local/src/xdaq/baseline12/trunk/daq/evb/src/common/bu/RUproxy.cc:539)
21 Jan 2016 15:35:54.755 [139942756808448] WARN  edu.bu.cms1.p: 33001.evb::BU.instance(0).RcmsStateNotifier <> - Unable to notify state change. RCMS state listener has not been found. Has findRcmsStateListener() been called at least once? State:Failed
21 Jan 2016 15:35:54.760 [139943433701120] ERROR edu.bu.cms1.p: 33001.executive::Application.lid(0)  - Unhandled exception occured
Caught exception: exception::DiskWriting 'Failed to get value from EVM at http://cms1.bu.edu: 33001/urn:xdaq-application:lid=12/eventCountForLumiSection?ls=3: Couldn't resolve host name' raised at getValueFromEVM(/usr/local/src/xdaq/baseline12/trunk/daq/evb/src/common/bu/RUproxy.cc:539)

• With single events I've been ramping up the FAKE_DATA_SIZE register to see when xdaq breaks. With fake amcs 1-12 enabled,

>lt 1
detected number after 'lt'
Sending 1 local triggers
Trigger: 0 left
>wv CONF.AMC.FAKE_DATA_SIZE 0x20
Write to register CONF.AMC.FAKE_DATA_SIZE
>lt 1
detected number after 'lt'
Sending 1 local triggers
Trigger: 0 left
>wv CONF.AMC.FAKE_DATA_SIZE 0x40
Write to register CONF.AMC.FAKE_DATA_SIZE
>lt 1
detected number after 'lt'
Sending 1 local triggers
Trigger: 0 left
>wv CONF.AMC.FAKE_DATA_SIZE 0x80
Write to register CONF.AMC.FAKE_DATA_SIZE
>lt 1
detected number after 'lt'
Sending 1 local triggers
Trigger: 0 left
>wv CONF.AMC.FAKE_DATA_SIZE 0x100
Write to register CONF.AMC.FAKE_DATA_SIZE
>lt 1
detected number after 'lt'
Sending 1 local triggers
Trigger: 0 left
>wv CONF.AMC.FAKE_DATA_SIZE 0x200
Write to register CONF.AMC.FAKE_DATA_SIZE
>lt 1
detected number after 'lt'
Sending 1 local triggers
Trigger: 0 left

This broke on the last one causing:

19 Jan 2016 18:00:32.963 [140717694822144] ERROR edu.bu.cms1.p: 33001.executive::Application.lid(0)  - Unhandled exception occured
Caught exception: pt::frl::exception::Exception 'failed to process copy for stream on index: 0 for copy worker id:0 - current frl event queue elements: 2 last processed input event dump: [[00000000]   47 5A 00 00 80 00 01 FE   00 00 00 64 00 0\
0 00 06   GZ...... ...d....
[00000010]   08 64 40 1F 06 00 00 51   10 12 F9 01 40 83 C1 10   .d.....Q ........
[00000020]   00 00 01 00 03 02 00 0F   00 00 02 00 03 02 00 0F   ........ ........
[00000030]   00 00 03 00 03 02 00 0F   00 00 04 00 03 02 00 0F   ........ ........
[00000040]   00 00 05 00 03 02 00 0F   00 00 06 00 03 02 00 0F   ........ ........
[00000050]   00 00 07 00 03 02 00 0F   00 00 08 00 03 02 00 0F   ........ ........
[00000060]   00 00 09 00 03 02 00 0F   00 00 0A 00 03 02 00 0F   ........ ........
[00000070]   00 00 0B 00 03 02 00 0F   00 00 0C 00 03 02 00 0F   ........ ........
[00000080]   03 02 40 1F 06 00 00 01   00 00 21 91 06 00 07 00   ........ ........
[00000090]   08 00 09 00 0A 00 0B 00   0C 00 0D 00 0E 00 0F 00   ........ ........
[000000a0]   10 00 11 00 12 00 13 00   14 00 15 00 16 00 17 00   ........ ........
[000000b0]   18 00 19 00 1A 00 1B 00   1C 00 1D 00 1E 00 1F 00   ........ ........
[000000c0]   20 00 21 00 22 00 23 00   24 00 25 00 26 00 27 00   ........ ........
[000000d0]   28 00 29 00 2A 00 2B 00   2C 00 2D 00 2E 00 2F 00   ........ ........
[000000e0]   30 00 31 00 32 00 33 00   34 00 35 00 36 00 37 00   0.1.2.3. 4.5.6.7.
[000000f0]   38 00 39 00 3A 00 3B 00   3C 00 3D 00 3E 00 3F 00   8.9..... ........
]' raised at process(/usr/local/src/xdaq/baseline12/trunk/daq/pt/frl/src/common/CopyWorker.cc:583);
        originated by pt::frl::exception::Exception 'Overflow of i2o: 32696' raised at processCopy(/usr/local/src/xdaq/baseline12/trunk/daq/pt/frl/src/common/CopyWorker.cc:301)
19 Jan 2016 18:08:32.848 [140716833482496] WARN  edu.bu.cms1.p: 33001.pt::frl::Application.instance(1) <> - Caught exception: tcpla::exception::FailedToReceive 'Failed to receive, connection event reset in PublicServicePoint 10.0.0.5:34000 : U\
UID = '74997085-96d3-4e4b-af75-f3a72507ca06'' raised at processSocketEntries(/cmsnfshome0/nfshome0/dsimelev/baseline12/trunk/daq/tcpla/src/common/PublicServicePoint.cc:920);
        originated by tcpla::exception::ConnectResetByPeer 'Connection reset by peer in PublicServicePoint 10.0.0.5:34000 on file descriptor 9 socket error: Connection reset by peer' raised at receive(/cmsnfshome0/nfshome0/dsimelev/baseline12/\
trunk/daq/tcpla/src/common/PublicServicePoint.cc:560)

2016-01-19, dgastler

Copy worker failed with

19 Jan 2016 16:15:37.684 [140287254984448] FATAL edu.bu.cms1.p: 33001.pt::frl::Application.instance(1) <> - Caught exception: pt::frl::exception::Exception 'Overflow of i2o: 32696' raised at processCopy(/usr/local/src/xdaq/baseline12/trunk/daq/pt/frl/src/common/CopyWork\
er.cc:301)
19 Jan 2016 16:15:37.685 [140287800329984] ERROR edu.bu.cms1.p: 33001.executive::Application.lid(0)  - Unhandled exception occured
Caught exception: pt::frl::exception::Exception 'failed to process copy for stream on index: 0 for copy worker id:0 - current frl event queue elements: 1 last processed input event dump: [[00000000]   47 5A 00 00 80 00 01 FE   00 00 00 64 00 00 00 01   GZ...... ...d....
[00000010]   08 64 50 56 01 00 00 51   00 ED 57 22 C0 81 61 10   .dPV...Q ..W...a.
[00000020]   00 00 01 00 03 04 00 0F   00 00 02 00 03 04 00 0F   ........ ........
[00000030]   00 00 03 00 03 04 00 0F   00 00 04 00 03 04 00 0F   ........ ........
[00000040]   00 00 05 00 03 04 00 0F   00 00 06 00 03 04 00 0F   ........ ........
[00000050]   03 04 50 56 01 00 00 01   00 00 D0 7E 06 00 07 00   ..PV.... ........
[00000060]   08 00 09 00 0A 00 0B 00   0C 00 0D 00 0E 00 0F 00   ........ ........
[00000070]   10 00 11 00 12 00 13 00   14 00 15 00 16 00 17 00   ........ ........
[00000080]   18 00 19 00 1A 00 1B 00   1C 00 1D 00 1E 00 1F 00   ........ ........
[00000090]   20 00 21 00 22 00 23 00   24 00 25 00 26 00 27 00   ........ ........
[000000a0]   28 00 29 00 2A 00 2B 00   2C 00 2D 00 2E 00 2F 00   ........ ........
[000000b0]   30 00 31 00 32 00 33 00   34 00 35 00 36 00 37 00   0.1.2.3. 4.5.6.7.
[000000c0]   38 00 39 00 3A 00 3B 00   3C 00 3D 00 3E 00 3F 00   8.9..... ........
[000000d0]   40 00 41 00 42 00 43 00   44 00 45 00 46 00 47 00   ..A.B.C. D.E.F.G.
[000000e0]   48 00 49 00 4A 00 4B 00   4C 00 4D 00 4E 00 4F 00   H.I.J.K. L.M.N.O.
[000000f0]   50 00 51 00 52 00 53 00   54 00 55 00 56 00 57 00   P.Q.R.S. T.U.V.W.
]' raised at process(/usr/local/src/xdaq/baseline12/trunk/daq/pt/frl/src/common/CopyWorker.cc:583);
        originated by pt::frl::exception::Exception 'Overflow of i2o: 32696' raised at processCopy(/usr/local/src/xdaq/baseline12/trunk/daq/pt/frl/src/common/CopyWorker.cc:301)

* Upping to 2 fake AMCs went up to 4khz ok, filled buffer but did not crash above that.

* Upping the rate from 10hz to 10kHz was fine in this configuration, upping to 20khz overloaded but did not crash the system.

We have a flow that has worked now, will test how reliable it is. THere is no data being saved to disk. Here is the setup of that run:

On the AMC13 (after a reconfigureFPGAs):

>rg
General reset
>rc
Counter reset
>en 1 f t
parsed list "1" as mask 0x1
Enabling fake data
Enabling TTS as TTC for loop-back
AMC13 out of run mode
AMC13 is back in run mode and ready
>daq 1
SFP0 enabled
Best to do a DAQ reset (rd) after changing link settings
>fed 0 100
>localL1A r 10 10
Configure LocalL1A enabled mode=2 burst=10 rate=10 rules=0
WARNING: AMC13 random trigger mode, setting burst to 1 L1A per burst

FedKit:

[dan@cms1 FEROL]$ /opt/xdaq/bin/fedKit.py 
Welcome to the optical FEDkit
=============================
Please select the data source to be used:
  1 - Real AMC13 data source        (L6G_SOURCE)
  2 - Generator core of the AMC13   (L6G_CORE_GENERATOR_SOURCE)
  3 - Loopback at the FEROL         (L6G_LOOPBACK_GENERATOR_SOURCE)
  4 - SLINK data source             (SLINK_SOURCE)
=> [1] 
Please enter the FED id you want to read out [100]: 
Do you want to write the data to disk? [No]:
Starting run...
Running. Point your browser to http://cms1.bu.edu:33001

m   display this Menu
f#  dump the next # FED fragments incl DAQ headers (default 1)
e#  dump the next # Events with FED data only (default 1)
q   stop the run and Quit
            
=> 

Back on the AMC13:

>lt c
Enable continuous local triggers

Fed dump file: http://bucms.bu.edu/twiki/pub/Main/AMC13DebugLogdump_run000052_event00005873_fed0100.txt

Event dump file: http://bucms.bu.edu/twiki/pub/Main/AMC13DebugLogdump_run000052_event00006104.txt

reconfigureFPGAs
wait...
rg
rc
daq 1
fed 0 100
localL1A o 1 1 

fedKit.py 

Welcome to the optical FEDkit
=============================
Please select the data source to be used:
  1 - Real AMC13 data source        (L6G_SOURCE)
  2 - Generator core of the AMC13   (L6G_CORE_GENERATOR_SOURCE)
  3 - Loopback at the FEROL         (L6G_LOOPBACK_GENERATOR_SOURCE)
  4 - SLINK data source             (SLINK_SOURCE)
=> [1] 
Please enter the FED id you want to read out [100]: 
Do you want to write the data to disk? [Yes]:
Please enter the directory where the data shall be written [/tmp]: 
Starting run...
Running. Point your browser to http://cms1.bu.edu:33001

m   display this Menu
f#  dump the next # FED fragments incl DAQ headers (default 1)
e#  dump the next # Events with FED data only (default 1)
q   stop the run and Quit

AMC13:

  lt 5

This still doesn't write to disk, but it does get to the user menu for the fedKit.py script

2015-12-11, dgastler

2015-12-11, dgastler

Hardware: Connect the AMC13 daq link 1 to the FEROL SFP descibed on the following pages.

http://cms-frl.home.cern.ch/cms-frl/ferol/ferol.html

https://twiki.cern.ch/twiki/bin/viewauth/CMS/UFEDKIT

This needs to use the "insert SFP name" SFP for the 5gbit link

Also connect the FEROL's 10Gb link to the 10Gb nic in cms1 using the "find name" SFP.

Software: On CMS1 open two terminals so you can run the fedKit.py and AMC13Tool2.exe.

Setup the AMC13 for fake data taking:

>rg
General reset
>rc
Counter reset
daq 1
SFP0 enabled
Best to do a DAQ reset (rd) after changing link settings
>rd
DAQ reset
>en 1-12 t f
parsed list "1-12" as mask 0xfff
Enabling TTS as TTC for loop-back
Enabling fake data
AMC13 out of run mode
AMC13 is back in run mode and ready
>localL1A o 1 1
Configure LocalL1A enabled mode=0 burst=1 rate=1 rules=0

Setup the FEROL in the daq account as follows:

[daq@cms1 FEROL]$ fedKit.py 
Welcome to the optical FEDkit
=============================
Please select the data source to be used:
  1 - Real AMC13 data source        (L6G_SOURCE)
  2 - Generator core of the AMC13   (L6G_CORE_GENERATOR_SOURCE)
  3 - Loopback at the FEROL         (L6G_LOOPBACK_GENERATOR_SOURCE)
  4 - SLINK data source             (SLINK_SOURCE)
=> [1] 
Please enter the FED id you want to read out [1234]: 
Do you want to write the data to disk? [No]:
Starting run...
Running. Point your browser to http://cms1.bu.edu:33001

m   display this Menu
f#  dump the next # FED fragments incl DAQ headers (default 1)
e#  dump the next # Events with FED data only (default 1)
q   stop the run and Quit
            
=> 

At this point open cms1.bu.edu:33000 in a web browser and go to the FEROL's monitoring page. Now sending triggers in AMC13Tool2.exe should cause the event counter to increment on this page.

2015-11-10, hazen

(At CERN) Successful test of two 5Gb links with different FED numbers. Partial success testing 10Gb links. For details:

https://twiki.cern.ch/twiki/bin/view/Main/AMC13DAQTestingLog

Set to 0x2000 with random 0x400-0x3000 rate is 9.2kHz with a few OFW counts.

2015-11-02, Hazen

Testing firmware 0x4037. Six AMC13 (S/N 284, 303, 306, 259, 308, 309) in AMC1-AMC6. All set for random size from 0x400-0x1000. Using TTTT with random rate at 0x800, which gives a rate of ~ 24kHz with lots of time in OFW.

See a significant number of "AMC13_BAD_LENGTH" and "BP_ERR" in AMC1 (S/N 284).

2015-10-97, Hazen

Finding apparent bugs in AMC13 monitor buffer. Here are some notes for someone to pick up the trail. First, read Monitor Buffer documentation. Then, review the data format documentation.

One can set the size of fake data by writing to CONF.AMC.FAKE_DATA_SIZE. By default this is set to 0x400 which results in unsegmented (single-block) events. If you set this larger than the somewhat arbitrary value of 0x13fb (about 40k bytes) then you get a segmented event. The table below illustrates this:

It's a bit confusing, but essentially you can generate events with different number of blocks by setting FAKE_DATA_SIZE. The bugs start when you have e.g. 2 free buffers and try to store an event which requires 3. Here is the command sequence to trigger this apparent bug:

  > wv CONF.AMC.FAKE_DATA_SIZE 0x100       # small size for single blocks
  > lt 0x3fe                               # generate 1k - 2 triggers
  > st 2 Monitor_Buffer                    # two empty buffers (0x400 - 2)
   Monitor_Buffer| COUNT| VALUE|
               --|------|------|
        AVAILABLE|      |     1|
            EMPTY|      |     0|
       UNREAD_EVT| 0x3FE|      |
  >wv CONF.AMC.FAKE_DATA_SIZE 0x3000
  >lt
  >st 2 Monitor_Buffer
   Monitor_Buffer| COUNT| VALUE|
               --|------|------|
        AVAILABLE|      |     0|           # if AVAILABLE=0 FULL should be 1
            EMPTY|      |     0|
       UNREAD_EVT| 0x3FE|      |
  >re                                      # read one event, freeing one buffer
  >st 2 Monitor_Buffer
   Monitor_Buffer| COUNT| VALUE|
               --|------|------|
        AVAILABLE|      |     0|
            EMPTY|      |     0|
             FULL|      |     1|           # why is it full now?
       UNREAD_EVT| 0x400|      |           # why did this go up?

2015-09-29, Gastler

2015-09-24, hazen/gastler

FEROL connected in mini-crate to CMS4. Shows up in lspci as

  06:04.0 Memory controller: Device ecd6:fea0 (rev 01)

Hook the fibers in "loop-back" on the FEROL (referring to the Manual)

2015-09-01, hazen

http://ohm.bu.edu/~hazen/CMS/TestStand/xDAQ_2015-08-31/

• http://ohm.bu.edu/~hazen/CMS/TestStand/xDAQ_2015-08-31/connections-eric.xml
• http://ohm.bu.edu/~hazen/CMS/TestStand/xDAQ_2015-08-31/run-standalone-eric.sh
• http://ohm.bu.edu/~hazen/CMS/TestStand/xDAQ_2015-08-31/standalone-daq-eric.xml

Note that the hcalUTCA module statement in the XML must appear before the hcalTrig as hcalTrig depends on hcalUTCA and if it comes first it will look in LD_LIBRARY_PATH or somewhere evil and get the wrong so file.

Also found that David's code hack in DTCManager to set the calibration trigger orbit gap has no error check whatsoever and will fail badly for most values of the orbit gap settings. For now edit DTCManager.cc to:

    // Enable calibration events in orbit gap if config doc says so                                                                                                              
    if (m_calibEnable){
      m_dtc->write(amc13::AMC13Simple::T1,"CONF.CAL_ENABLE",1);
      try {
        // Set calibration orbit gap window // substract 3456 since only writing to editable part                                                                                
        m_dtc->write(amc13::AMC13Simple::T1,"CONF.CAL_WINDOW_LOWER_PROG", m_calibLower-3456);
        m_dtc->write(amc13::AMC13Simple::T1,"CONF.CAL_WINDOW_UPPER_PROG", m_calibUpper-3456);
      } catch( std::exception& e) {
        LOG4CPLUS_ERROR(getApplicationLogger(), ::toolbox::toString("DTCManager::init() setting orbit gap to %d-%d", m_calibLower, m_calibUpper));
      }
    }
    else m_dtc->write(amc13::AMC13Simple::T1,"CONF.CAL_ENABLE",0);

Hi Eric,

Ah, now I know what the issue is.  The problem is that hcalTrig is now dependent on the hcalUTCA library, 
so the hcalTrig is pulling in the old hcalUTCA.

Can you move the line with the libhcalUTCA _above_ the hcalTrig line?

Jeremy 

Updated connection file for uHTR in slot 7:

http://ohm.bu.edu/~hazen/CMS/TestStand/xDAQ_2015-08-31/connections-eric.xml

http://ohm.bu.edu/~hazen/CMS/TestStand/xDAQ_2015-08-31/

This seems to work ~ ok but not clear if it is really using locally-compiled libraries!

2015-08-31, hazen

Trying to get HCAL xDAQ running. First challenge is to get a source tree which will build. First, manage to finally get SVN to stop demanding my password every time, using a variation of the instructions here:

http://information-technology.web.cern.ch/book/how-start-working-svn/accessing-svn-repository#accessing-sshlinux

Not certain what I did differently, but had to explicity concatenate my id_rsa.pub to ~/.ssh/authorized_keys. Now it completes with no password prompts!

Trying this (list of pacakges based on the beginning of the top-level Makefile):

SUBPACKAGES := \
	hcalBase \
	hcalDCC \
	hcalHW \
	hcalOnlineDB \
	hcalTrig \
	hcalAux \
	hcalClassic \
	hcalCalib \
	hcalSupervisor \
	hcalMonVis \
	hcalUpgrade \
	hcalUHTR \
	hcalRBX hcalAlarm hcalCCM \
	hcalDCC/tool hcalOnlineDB/web

Here's the install/build sequence:

$ source /home/daqowner/dist/etc/env.sh
$ perl installDAQ_12_4_9.perl --mode=teststand \
 --ownsource=${HOME}/src/12_4_9 \
 --svnuser=ehazen \
--packages=hcalBase,hcalDCC,hcalHW,hcalOnlineDB,hcalTrig,\
hcalAux,hcalClassic,hcalCalib,hcalSupervisor,hcalMonVis,\
hcalUpgrade,hcalUHTR,hcalRBX,hcalAlarm,hcalCCM
$ cd src/12_4_9/hcal
$ make -j 4 |& tee ~/build_12_4_9_try4.log

The build succeeded!

2015-06-30, dzou

While installing LabTools (impact, chipscope, etc.). There was a problem getting teh Xilinx USB Cable to work. Tried typical troubleshoot suggested in Xilinx forums including manual install scripts. But the Cable was not working (power light should turn on when plugged in while drivers installed correctly and permission set correctly).

The eventual problem ended up being this (courtesy of Jeroen Hegeman):

The problem was as usual in a detail: in the USB settings file deployed by Xilinx here

/etc/udev/rules.d/xusbdfwu.rules

an environment variable name was spelled in capitals instead of lower-case: $TEMPFILE instead of $tmpfile. Took me a while to find that again.

2015-04-08, hazen

Installed Magnum mini-crate with FEROL and 10GbE NIC (I think it's an Emulex OCe10102-N).

Downloaded OneConnect-Flash-10.0.803.37.iso from emulex.com. Make USB stick, but MoBo can't boot from it so burn to CD-ROM. But when you boot from the CD-ROM it can't find the utilities it needs!

Here is what I managed:

Copy ISO to both CDROM and USB stick
insert CDROM only and boot from it (F12 for boot menu)
wait for it to time out, failing to find CDROM
insert usbstick

  # mount /dev/sdb /usbdrive
  # cd usbdrive/UFI
  # flash -x -f 10.0.803.31.ufi

reboot

Now the fs#@ing computer hangs on some Emulex firmware handshake and won't boot!

2015-03-17, hazen

Attempting to get FEROL (uFEDKIT) working. Dan has installed a 10GbE NIC in CMS1 and configured it at 10.0.0.5. It responds to pings.

Now using S/N 98. Update firmware to latest 0x401d/0x0027.

Tried running the /opt/xdaq/bin/fedKit.py script but it fails.

2015-03-09, hazen

2013-03-09, hazen

Testing firmware 0x222, 0x27 with software rel 34315 on CMS4. Seeing uHAL timeouts while reading data using 'df' command. Example:

  $ AMC13Tool2.exe -c 192.168.1.82
  $ Address table path "/home/hazen/work/amc13/amc13/etc/amc13" set from AMC13_ADDRESS_TABLE_PATH
  use_ch false
  Created URI from IP address:
    T2: ipbusudp-2.0://192.168.1.82:50001
    T1: ipbusudp-2.0://192.168.1.83:50001
  Using AMC13 software ver:34315
  > en 1-12 f t
  > localL1A o 1 10
  > lt c
  > lt d
  > df test.dat 9999
    ...
   Read 12340 words
   Wrote 12342 words to test.dat
   calling readEvent (53)...

  Caught microHAL exception.
  	Timeout (1000 milliseconds) occurred for UDP receive from target with URI: ipbusudp-2.0://192.168.1.83:50001

This problem does not occur when connecting from cms1.bu.edu. Tried both build 32534 (my current) and Dan's test build with uHAL 2.4. Back to cms4, update to 34988 (trunk) and re-build. No improvement.

Seems like some sort of network problem.

Use controlHub, all is well.

2015-02-09, hazen

TTC firmware superficially working. Lots of clean-up on address table to do. Need a "repeat" feature which is now documented here: AMC13AddressTable (new P_Repeat and P_Offset items). Temporarily implemented using a perl script csv_expand_repeat.pl; should eventually build into the C++.

Today: program new 0x4015 firmware which should fix some DAQLDC register readout bug. Test T2 TTC capture.

2015-02-03, hazen

Testing new spartan TTC firmware 0x26. Move S/N 39 to MCH2 site in Crate 2. Reprogram to 0x21d/0x26.

• Updating MMC to check if problem is fixed...
  • MMC updated to 2.2, eeprom erased
  • T2 NV memory now no longer holds an IP address so uses default (192.168.1.154)
  • Still cannot write to T2 NV memory (' Given data "256" is invalid. ' error)

2015-01-22, dzou

Strange behavior of SN50 board.

• NVMem IP address saved as 192.168.1.65 on T2
• When trying to changed with storeConfig gets error: ' Given data "256" is invalid. '
• Created a special connection file with T2 and T1 ip addresses specified.
• When checking 'fv' gives: *0: SN: 95 T1v: 021c T2v: 0029 cf: /home/dzou/amc13/amc13/etc/amc13/connectionSN50
  • reports strange SN and T2 has some sort special T2 version.
• solder blobs correspond to SN50
• MMC version out of date (current is 2.2 but it has 2.1)
• Updating MMC to check if problem is fixed..

2015-01-13, hazen

Looking at S/N 75/49 returned from Cornell with an alleged TTC data problem. Install in MCH2 site in crate 2.

Nmap scan report for rfc1918.address.not.used.bu.edu (192.168.2.56)
Host is up (0.000087s latency).
MAC Address: 08:00:30:F3:01:9C (Network Research)
Nmap scan report for rfc1918.address.not.used.bu.edu (192.168.2.57)
Host is up (0.000093s latency).
MAC Address: 08:00:30:F3:01:DC (Network Research)

2015-01-06, hazen

Investigating S/N above 128. Take board#99, change to 0xe3 (227). Won't power up properly in crate 2. Move to crate 1 slot 2. Looks good according to scanCrate.pl:

2: MMC: 2.2 IP: 192.168.2.56 192.168.2.57 vv: 0x1002 sv: 0x0021 sn: 227

So IP last octet is 255-2*(sn & 127) and 254-2*(sn & 127). This is OK except that we should probably avoid IP address 128 as it would result in last octet = 255/254.

MAC addresses set as follows

Nmap scan report for rfc1918.address.not.used.bu.edu (192.168.2.56)
Host is up (0.000064s latency).
MAC Address: 08:00:30:F3:01:9C (Network Research)
Nmap scan report for rfc1918.address.not.used.bu.edu (192.168.2.57)
Host is up (0.000074s latency).
MAC Address: 08:00:30:F3:00:5C (Network Research)

The documented scheme is:

  bits 0-5     complement of serial number bits 0-5
  bit 6        =0 for T2 board (lower IP)
               =1 for T1 board (upper IP)
  bit 7, 8     serial number bits 6, 7 (not complemented for S/N < 64 only)

I'm confused.

2014-11-13, hazen

Possible FEROL installation. What about this adapter?

http://www.startech.com/Cards-Adapters/Slot-Extension/PCI-Express-to-PCI-Adapter-Card~PEX1PCI1

2014-11-07, hazen

Recompiling AMC13 T1 Firmware

In Vivado v2014.3 (64-bit) on Ubuntu 14.04:

Download file: http://physics.bu.edu/~wusx/download/AMC13/AMC13XG_HCALv0x400a.xpr.zip

Find K7version constant and change to 0x4aaa. Unzip, open and recompile with default settings. Find .bit file and copy to /tmp. Execute TCL command:

write_cfgmem -loadbit "up 0x0 /tmp/AMC13_T1.bit" -format MCS -size 128 /tmp/AMC13_T1_test.mcs

(guessed at some arguments). Rename, program into flash. It works!

http://ohm.bu.edu/~hazen/CMS/AMC13/AMC13_T1_v0x4aaa_utilization.txt

http://ohm.bu.edu/~hazen/CMS/AMC13/AMC13XG_T1_v0x800a_utilization.txt

  localTtcSignal=true
  internalPeriodicTriggerEnable=true

  if ( SLOT == 2 ) {

(once only) start control hub:

  sudo /etc/init.d/controlhub start

Then: sh run-standaone.sh. Should see ~30 lines of messages with no errors, ending in "Ready.".

Start web browser on cms1 (e.g. "ssh -X cms1; firefox --no-remote").

Go to "localhost:40010" (xDAQ main page). Navigate to:

* hcalSupervisor * Control Panel * Setup (again, ~30 lines of messages ending in "New state is:Ready") * Start

Go back to "localhost:40010" in another tab, open "hcalDTCmanager".

* Peek into... * Basic View

1. Reload firmware for module using software (in principle this should have the effect as powercycle)
2. Rerun initialization

2014-09-02, dzou

Suggestion for troubleshooting AMC13 with uHTR initialization:

1. Reload AMC13 firmware
2. Disable uHTR DAQ Path
3. Power cycle modules

2014-08-21, Eric

To run csv_to_xml.pl script in ...amc13/dev_tools/cfg/addrTableTools, need to install perl module Tree::Simple. Do this with:

   $ sudo yum install perl-Tree-Simple

2014-08-19, Dan, Eric

Started to get xDAQ working. Jeremy now provides a perl script standalone_setup.pl in ...hcalUpgrade/test in the HCAL xDAQ release. It creates standalone-daq.xml which is then used by run-standalone.sh to start xDAQ. How to answer the questions from standalone_setup.pl

Sample connection file: sample_xdaq_connections.xml. Set for crate number 1.

2014-08-08, dzou

• Status display:
  • AMC Link Status:
    • Enabled links
    • Locked links
  • Port Status:
    • AMC Link Versions incorrect
    • Unsynced AMC Ports
  • AMC Bc0 Status: Bc0s locked
  • AMC Counters
• Address Tables and Mr. Wu Spec files?

Places to change code:

1. singleBit_OnOff is a method that checks bits and populates a vector of strings indicating with AMC ports have their relevant bit set (in 0-based enumation). All Status display items from AMC Link Status through Bc0 Status use this to enumerate. (Actions.cc, line 2625)
   • Up front my guess at the fix:
     1. (ss bit shift by i+1? and the if statement to i<=10) or
     2. (loop i starting at i=1 and make the appropriate shifts throughout loop)
2. ctr64_status_amc is the method use in AMC Counters. (Actions.cc, line 2657, ena_amcs[i] )
   • ena_amcs is populated by singleBit_OnOff, so if singleBit_OnOff is made to give a vector of 1-based strings, then it may not be necessary to change ctr64_status_amc at all.

2014-07-18, dzou

Multiple users have experienced problems with receiving/sending Bc0s from their AMC13. Unfortunately, we do not have the same hardware and we are experiencing difficulties providing support, as it is hard to determine if the issue is in fact due to the AMC13.

Below is a place to document tests using our setup with the AMC13 and uHTR to determine setups work/don't work in hopes of illuminating possible problems that may be occurring in the AMC13 side.

All tests using uHTR running most recent firmware (front: 0.C.0, back: 0.E.22) to receive Bc0. AMC13 used is Serial #86 in AMC13 slot of old crate.

Since the uHTR will send a signal back after receiving Bc0s so that the AMC13 can tell if the Bc0s are locked

Note 1: Sending triggers does not successfully lead to building events to monitor buffer, Note 2: Bc0 locked only successfully locked if setup is done after complete power cycle of boards (physically removed from crate)

2014-07-03, hazen

Trying Wu's new 0x1001 version. Not obvious that it works. Meanwhile, try to reprogram S/N90 T2, but somehow this fails and the module is bricked.

>readRepeat 0 0x1080 100000
terminate called after throwing an instance of 'uhal::exception::UdpTimeout'
  what():  Timeout (1000 milliseconds) occurred for UDP receive from target with URI: ipbusudp-2.0://192.168.1.90:50001

Take GbE switch out (connect yellow cable direct from CMS2 to Vadatech MCH). Now flash programming is reliable. Maybe the TTT was causing trouble? Disconnect it.

2014-07-02, hazen

2014-07-02, hazen

S/N 93 flakes out when installed in crate with many other boards (T1 doesn't configure).

Now we have thirteen AMC13 in the crate at once:

Using AMC13Tool found at /home/hazen/bin/AMC13Tool.exe
 1: MMC: 2.2    IP:      192.168.1.62     192.168.1.63 vv: 0x0109 sv: 0x0021 sn:  96 temp:  732
 2: MMC: 2.2    IP:      192.168.1.70     192.168.1.71 vv: 0x0109 sv: 0x0021 sn:  92 temp:  627
 3: MMC: 2.2    IP:     192.168.1.126    192.168.1.127 vv: 0x0108 sv: 0x0021 sn:  64 temp:  761
 4: MMC: 2.2    IP:      192.168.1.60     192.168.1.61 vv: 0x0109 sv: 0x0021 sn:  97 temp:  708
 5: MMC: 2.2    IP:      192.168.1.54     192.168.1.55 vv: 0x0109 sv: 0x0021 sn: 100 temp:  595
 6: MMC: 2.2    IP:      192.168.1.52     192.168.1.53 vv: 0x0109 sv: 0x0021 sn: 101 temp:  686
 7: MMC: 2.2    IP:      192.168.1.72     192.168.1.73 vv: 0x1000 sv: 0x0020 sn:  91 temp:  386
 8: MMC: 2.2    IP:      192.168.1.78     192.168.1.79 vv: 0x1000 sv: 0x0020 sn:  88 temp:  413
 9: MMC: 2.2    IP:      192.168.1.76     192.168.1.77 vv: 0x1000 sv: 0x0020 sn:  89 temp:  370
10: MMC: 2.2    IP:      192.168.1.74     192.168.1.75 vv: 0x1000 sv: 0x0020 sn:  90 temp:  400
11: MMC: 2.2    IP:     192.168.1.106    192.168.1.107 vv: 0x0108 sv: 0x0021 sn:  74 temp:  706
12: MMC: 2.2    IP:      192.168.1.56     192.168.1.57 vv: 0x0109 sv: 0x0021 sn:  99 temp:  668
13: MMC: 2.2    IP:      192.168.1.90     192.168.1.91 vv: 0x0209 sv: 0x0021 sn:  82 temp:  702

2014-07-1, aguld

Testing boards 51, 52, 68, 72, 78, 79, 83, 94

51: T1 has a short in the power supply. When doing chipscope test with T2, the error counter is changing while it is supposed to remain constant.

52: T1 is working well, T2 was shorted out when the 12V power was turned on.

68: The blue indicator light does not turn off when the handle is released, T2 is good. T2 is currently with S/N 78's T1 and T3. Does not have a T3 with it.

72: T1 is working well. When T2 is used on any working T1, the blue indicator light does not turn off when the handle is pushed in.

78: T1 is good. When T2 is used on any working T1, the blue indicator light shuts off after a few seconds, even when the handle is not pushed in.

79: Board was powered up in test stand with T1 and T2 in the wrong slots. T2 does not work at all, but T1 works fine.

83: T1 works. T2 has a shorted 3.3v power and has been put away

94: T1 has power supply damage and is likely permanently damaged. T2 is probably still usable.

2014-06-30, hazen

Trying to fill the 2nd crate. Install AMC13 in slots 1-8 and 13. Scan crate behaves badly (last couple of boards report 255's for IP addresses). Pull slots 1-4 so only 5-8 and 13 are filled. Slot 8 still has a problem. Only 7-8 and 13 filled, still slot 8 is bad.

Now things are totally flaky. Suspect an MCH or software problem. After a few minutes things are ok again with one module.

Accessing Vadatech MCH: web server at http:192.168.2.2:8080 on CMS2 works. Web interface is horrible but with enough poking one can find and recognize the AMC13 sensors and readings.

• • Not able to ping 10/100 port, when connected to Gigabit Ethernet port (NOTE: This is likely because the broadcast of cmssun's eth1 is not set to communicate w/ address outside of 192.168.1 while cms2 is set to for all addresses between 192.168.1.-192.168.255.):

• • cms2 seems to be able to ping 10/100 port (192.168.2.2), but cmssun4 cannot (NOTE: This is likely because the broadcast of cmssun's eth1 is not set to communicate w/ address outside of 192.168.1 while cms2 is set to for all addresses between 192.168.1.-192.168.255.).

2014-06-12, dzou, hazen, aguld

Experienced initial problems/confusion with communicating with Vadatech MCH due to a combination of a bad Ethernet cable and the IP address of the local machine being set to the same IP address as what the Gigabit Ethernet port on the MCH was set to.

Connecting using serial port using minicom:

• Was able to use serial port cable to connect using minicom w/ the following settings (taken from Vadatech MCH "Getting Started" guide)
  • 115200 baud, 8 data bits, no parity, and 1 stop bit
  • In /etc/rc.d/rc.conf found information on IP addresses for Ethernet ports (interface0 is the 10/100 Ethernet port, interface 1 is the Gigabit Ethernet port)
  • Gigabit Ethernet ports set to (192.168.1.2)
  • 10/100 Ethernet port set to (192.168.2.2)
• eth1 of cmssun4 was set to (192.168.1.2), which conflicted with the Gigabit Ethernet port IP
  • change eth1 of cmssun4 to 192.168.1.4

Connecting to Gigabit Ethernet port directly to local computer:

• Connected MCH Gigabit Ethernet port directly to cmssun4 and was able to ping Gigabit Ethernet (192.168.1.2)
  • Was able to connect to AMC13 that was in Vadatech crate, using VM running SLC6 on cmssun4.
    • AMC13Tool able to connect to AMC13.
    • Python scripts (readIPs, applyConfig, etc) confirmed working
    • Used (192.168.1.2) as MCH host IP for python scripts.
  • Not able to ping 10/100 port, when connected to Gigabit Ethernet port (persisting problem):