(r17) AMC13UserManual < BUCMSPublic

Tags: view all tags
THIS MANUAL IS INCOMPLETE but what is written here should be more accurate and up to date
than the old "functional specification".  Please e-mail me if you find mistakes/omissions here (Eric H).

*See Also*

   * [[AMC13Tool2]] command-line software too

*Contents*

%TOC%

---++ DAQ Outputs

The AMC13 implements the S-Link Express protocol using a core provided by CMS CDAQ.
This is currently a 5.0Gb/s link, received by the FedKit (see [[https://twiki.cern.ch/twiki/bin/view/CMS/CMSFedkitManual][CMSFedkitManual]])
or by an AMC13v1 (Virtex-6 version only) running special firmware version 0xff.

The AMC13 has 3 SFP transceivers which may be used for DAQ.  There are 3 possible configurations, determined by the value written to *CONF.SFP.ENABLE_MASK*.  In addition, *CONF.EVB.ENABLE_DAQLSC* must be set to '1'.  The =AMC13Tool2= command *[[AMC13Tool2#daq_config][daq]]* may be used to accomplish this configuration.

| *Configuration* | *Mask value* | *Notes* |
| SFP0 only | 1 | (top) DAQ fiber for AMC1-AMC12 readout |
| SFP0, 1 | 3 | SFP0 for AMC1-AMC6, SFP1 for AMC7-AMC12 |
| SFP0, 1, 2 | 7 | SFP0 for AMC1-AMC4, SFP1 for AMC5-AMC8, SFP2 for AMC9-AMC12 |

After changing the DAQ output configuration the logic and high-speed transceivers must be reset by writing to *ACTION.RESETS.DAQ* (=AMC13Tool2= command *[[http://bucms.bu.edu/twiki/bin/view/BUCMSPublic/AMC13Tool2#rd_DAQ_reset][rd]]*).

The DAQ sender firmware itself was provided by the CMS cDAQ group.  Links to the firmware
in the CERN SVN and it's documentation may be found here:

   * [[https://svnweb.cern.ch/cern/wsvn/cms-firmwsrc/cdaq/slinkexpress/Xilinx/][FIrmware Link]]
   * [[https://svnweb.cern.ch/cern/wsvn/cms-firmwsrc/cdaq/slinkexpress/Xilinx/trunk/Documentation/Optical%20SLINK64_v4.doc][Documentation]] (MS-word format)

---++ TTS

The TTS (Trigger Throttling System) output from the AMC13 is a four-bit code transmitted over
the transmit half of the bottom SFP fiber transciever, and is normally sent to the TCDS system
to control the trigger rate in response to pending buffer overflows.  This system is intended
to be logically compatible with the legacy system documented 
[[http://cmsdoc.cern.ch/cms/TRIDAS/horizontal/RUWG/DAQ_IF_guide/DAQ_IF_guide.html#Physical][HERE]].

The AMC13 outputs the TTS state as the four-bit code described in the link above:

|| 0000 || Disconnected || Hardware Failure or broken cable ||
|| 0001 || Overflow Warning || Imminent buffer overflow ||
|| 0010 || Sync Lost || AMC13 is not synchronized with DAQ due to buffer overflow ||
|| 0100 || Busy || Cannot accept triggers ||
|| 1000 || REady || Ready to accept triggers ||
|| 1100 || Error || Any other state that prevents functioning ||
|| 1111 || Disconnected || Hardware failure or broken cable ||

Any time the AMC13 is not in run mode (such as after power up)
the AMC13 sends state "0100" (busy).

Internally the AMC13 manages the TTS state using a 5 bit internal format; this is
exposed in a few monitoring registers.  This format should be decoded as text
in the latest AMC13 software, but is documented here for completeness:

| Bit 4 | Disconnected |
| Bit 3 | Error |
| Bit 2 | Sync Lost |
| Bit 1 | Busy |
| Bit 0 | Overflow Warning |

(A value of 0 means "ready")

The TTS state is sent over the same 5.0Gb/s link as the DAQ data. There's an input port in our link logic on the AMC side:

<pre>
-- TTS port
    TTSclk            : in  std_logic;  -- clock source which clocks TTS signals
    TTS               : in  std_logic_vector (3 downto 0);
</pre>

When the TTS state changes, a control word is sent across the link to transfer the information with minimum latency to the AMC13. The final TTS output state is simply a priority encoding of the 12 AMC states plus the local AM13 one.

---++ TTC Simulator

The AMC13 has the ability to generate simulated TTC signals and distribute them to AMC cards in the crate.
This allows operation of a stand-alone test setup with only an AMC13 and AMC cards in a single crate without
requiring any external TTC hardware.  The simulated TTC signal will always include a BC0 sent once per LHC orbit.
Four "BGO" channels are provided which can send programmed short or long format TTC commands either
once under program command or periodically.
In addition, ECR (event count reset) and OCR (orbit count reset) TTC commands may be sent by writing
to the *ACTION.LOCAL_TRIG.SEND_ECR* and *ACTION.LOCAL_TRIG.SEND_OCR* registers, respectively.

_N.B._ the OCR and ECR mentioned above will not be reflected in the AMC13 registers until after
the next !L1A, as the current EvN and OrN in the AMC13 are not visible; they are used only to stamp
an event in in response to !L1A.

The "BGO" channels are programmed using registers 0x24-0x27 (*CONF.TTC.BGOn*).
Each of the four channels requires the following settings, where =BGOn= is =BGO0=,
=BGO1=, =BGO2= or =BGO3=.

| *Register* | *Function* |
| =CONF.TTC.BGOn.COMMAND= | Short (bits 0-7) or long (bits 0-31) format TTC command |
| =CONF.TTC.BGOn.LONG_CMD= | Bit '1' for long-format command, '0' for short format |
| =CONF.TTC.BGOn.ENABLE_SINGLE= | Bit '1' to enable single command (trigger with =ACTION.TTC.SINGLE_CMD=) *[1]* |
| =CONF.TTC.BGOn.ORBIT_PRESCALE= | Orbit prescale (prescale is value + 1) |
| =CONF.TTC.BGOn.BX= | Bunch crossing number on which to send command |

*[1]* Only one of four =ENABLE_SINGLE= may be set at one time.
If bit is set to 0 the commands are sent periodically

Locally-generated triggers may be sent in a rather flexible way.  See the next section for details.

The simulated TTC function requires a clock which is transmitted on the output side of the TTC
SFP optical transceiver and must be received on the input side, so a short loop-back cable must be plugged
between the input and output sides of the bottom SFP.

This feature is enabled by setting *CONF.DIAG.FAKE_TTC_ENABLE* to 1.  The local !L1A generator
must also be enabled (*CONF.TTC.ENABLE_INTERNAL_L1A* set to 1).

---+++ TTC Command Details

The AMC13 can transmit and take action on several specific TTC commands. 

| *Command* | *Default Value* | *Programmable?* | *Notes* |
| BC0 | 1 | No | Bunch Count Reset (sent every orbit)... CMS standard? |
| EC0 | 3 | No[1] | Event Count Reset (send at the start of each lumisection to set the EvN to 1) |
| OC0 | 9 | Yes | Orbit Count Reset - reset orbit count to 0 |
| Resync | 0x28 | Yes | Resync after error |

Note that for the programmable commands, there is both a command value register (8 bits)
and a mask value register (8 bits) which allows only specific bits to be matched when decoding
a received command.

[1] This should be programmable

These are taken originally from [[http://cmsdoc.cern.ch/cms/HCAL/document/CountingHouse/TTC/TTCcommands.html][this table]] maintained by HCAL:


---++ Local !L1A Generator

This feature allows the AMC13 to generate !L1A and transmit them over the TTC backplane signals to AMC cards.
It may be used in conjunction with the TTC Simulator described above, or with an external TTC input.

There are 4 modes of operation available:

   * Individual triggers under software control
   * Burst with count and spacing in BX or orbits specified
   * Continuous triggers equally spaced by BX or orbits
   * Random triggers from 2Hz to about 130kHz rate with CMS trigger rules respected

Various registers are used to control the local !L1A generator.  The easiest way to control this feature
is using the method *AMC13::configureLocalL1A()* in the *AMC13* class.  Then, call *AMC13::sendL1ABurst()* to
send a single software-triggered burst, or *AMC13::startContinuousL1A()* and *AMC13::stopContinuousL1A()* to
start or stop continuous triggers.

*AMC13::configureLocalL1A( bool ena, int mode, uint32_t burst, uint32_t rate, int rules)* documentation:

| *Parameter* | *Description* |
| _ena_ | true to enable the L1A generator |
| _mode_ | 0 - periodic triggers spaced by _rate_ orbits at BX=500 |
|^| 1 - periodic triggers spaced by _rate_ bx |
|^| 2 - random trigger at 2* _rate_ Hz |
| _burst_ | number of triggers in a burst (1-4096) |
| _rate_ | sets the rate based on mode (1-65536) |
| _rules_ | set to 0 normally to enforce the "standard" CMS trigger rules |

Brief register-level documentation follows.  Register at offset 0x1c
controls the local !L1A generation through the following bit fields:

   * *CONF.LOCAL_TRIG.RATE* sets the rate or spacing (0 means spacing=1)
   * *CONF.LOCAL_TRIG.NUM_TRIG* sets the burst count (0 means count=1)
   * *CONF.LOCAL_TRIG.TYPE* is the mode (0 for orbit, 2 for BX, 3 for random)
   * *CONF.LOCAL_TRIG.RULES* specifies which [[http://bucms.bu.edu/twiki/bin/view/BUCMSPublic/TTTSoftwareDocumentation#TriggerRuleDescriptions][CMS trigger rules]] are followed

Rule 1 is always enforced.  The *rules* parameter to *AMC13::configureLocalL1A* or
the *CONF.LOCAL_TRIG.RULES* item may be set as follows to suppress other rules:

   * 0 means enforce all rules (1-4)
   * 1 means all except rule 4
   * 2 means enforce rules 1 and 2
   * 3 means enforce only rule 1


---++ TTC History Capture

This feature added to the T2 (Spartan) firmware starting in version 0x26
allows the capture of up to 512 TTC short-format broadcast commands in a buffer.
The commands may originate in the AMC13 itself (if the TTC simulator is being used)
or received externally on the TTC fiber input.

A filter feature is provided which checks incoming commands against a list
of up to 16 entries, and if a match is found the command is discarded rather
than being stored in the history.

Each filter item has the following 3 fields:

<pre>
  bits 0-7   TTC command value to match
  bits 8-15  Mask applied before match.  '1' to ignore specified bit
  bit 16     This filter item is enabled if '1'
</pre>

Several C++ functions are provided which are briefly listed below.
See the nightly 
[[http://cactus.web.cern.ch/cactus/nightly/api/html_dev_amc13/index.html][API documentation]]
for details.

<pre>
    void setTTCHistoryEna( bool ena);                      // enable/disable history capture
    void setTTCFilterEna( bool ena);                       // enable/disable history filter
    void setTTCHistoryFilter( int n, uint32_t filterVal);  // set individual filter item
    uint32_t getTTCHistoryFilter( int n);                  // get individual filter item
    void clearTTCHistoryFilter();                          // clear entire filter list
    void clearTTCHistory();                                // clear capture history (reset count)
    void getTTCHistory( uint32_t* buffer, int nhist);      // get TTC history list (READ DOCS)
    int getTTCHistoryCount();                              // get TTC hsitory count
</pre>




---++ External Clock / Trigger Inputs (g-2)

The following section applies only to the g-2 firmware series (Kintex 0x8000 and up).
The AMC13 clock and external trigger can come from one of 3 sources (optical fiber, internal or external copper signal).
The choices are enumerated in the following table.

| *Clock Source* | *Trigger Source* | *Hardware* | *T1 0x1 bit 15* | *T1 0x1 bit 8* | *T1 0x1 bit 2* |
| Internal | Internal | Loop-back fiber (Note 1) | 0 | 1 | 1 |
| Fiber | Internal | Fiber with TTC input | 0 | 0 | 1 |
| Lemo | Internal | Special T3 with clock input | 1 | 0 | 1 |
| - | - | Not used | 1 | 0 | 0 |

Notes:

   1. SFP transceiver with loop-back fiber from Tx to Rx must be installed in bottom site

---++ Wu's Debugging Guide

<pre>
AMC13 quick trobleshooting with register dump
	Last updated on 3/19/2015

Following description is accurate only for T1 versions
0x4020, 0x225, T2 version 0x27 and later.
Also make sure bit 11-0 of T1 reg 0x5 are all 0, any bit
set to 1 indicates that AMC module has a different backplane
link version as that of the AMC13 T1 firmware.

T1 version is bit 31-16 of T1 reg 0x1
T2 version is bit 15-0 of T2 reg 0x0

a)Keep firmware up to date
  Always check for the latest firmware and upgrade
your system. New versions are released to fix bugs or
adding debugging information, so it is important to
keep your firmware up to date. If you have problems,
upgrade to the latest version and see if that solves
your problem.

b)TTC problems
  Once set up right and TTC works correctly, check regularly
  the following T2 registers:
  0x7 counts bcnt errors, it should have no more than couple
      of counts.
  0x8 counts TTC single errors, it should have no more than couple
      of counts.
  0x9 counts TTC multiple errors, it should have no more than couple
      of counts.
  If TTC does not work, 
    first check T1 reg 0x4:
    if bit0 is 1, TTC optical receiver is absent.
    if bit7 is 1, there's no TTC input signal, check the
                  cabling to TTC source. 
  If AMC13 registers look OK, but AMC modules have TTC
  problem, make sure your TTC decoder has the right timing.
  AMC13 output TTC clock's edge is in the middle of the TTC
  data on the backplane.

c)run stopped because of AMC13
  If run stopped and bit 15-12 of T1 reg 0x19 is not 0x8, check T1 registers
  0xe1a, 0xe1b and 0xe1c. If any of them is non-zero, at least one AMC is
  causing the problem. Each AMC uses one byte, AMC1 using bit7-0 of reg 0xe1a
  and AMC2 using bit15-8 of reg 0xe1a and AMC3 .... The definition of the byte is
  bit7 if set, AMC has been in disconnected state
  bit6 if set, AMC has been in error state
  bit5 if set, AMC has been in out of sync state
  bit4 if set, AMC is in disconnected state
  bit3 if set, AMC is in error state
  bit2 if set, AMC is in out of sync state
  bit1 if set, AMC is in busy state
  bit0 if set, AMC is in overflow warning state

  If AMC13 is in overflow warning or busy states, first check T1 reg 0x0. If bit0
  is set, cDAQ is down. Otherwise, check T1 reg 0xd4, if bit 18-16 are not all 0.
  cDAQ full stopped sending data out. If cDAQ is neither down nor full, and bit 10-8
  are all 0, then it is event builder not building events. Next check T1 register
  0xe0c, if any bit of bit 11-0 is not 0, the corresponding AMC(bit 0 AMC1) has no
  data.

d) data integrity problems
  T1 register 0xb3-0xb5 counts event cmsCRC error for SFP0,SFP1 and SFP2
  T1 register 0xb6-0xb8 counts event length error for SFP0,SFP1 and SFP2
  T1 register space 0x800-0xdff are monitoring counters for AMC modules,
  each AMC module occupies 0x80 32 bit space, AMC1 uses 0x800-0x87F.
  following registers' address is offset address in their own space. Each counter
  occupies two 32 bit space. Even address is the lower 32 bits and odd address is
  the upper 16 bits of the counter.
  offset 0x6-7 is event number of the event mismatch counter
  offset 0x8-9 is Orbit count of the event mismatch counter
  offset 0xa-b is BC count of the event mismatch counter
  offset 0x12-13 is bad EventLength counter
  offset 0x14-15 is trailer Evn mismatch counter
  offset 0x1e-1f is link input Evn skip counter
  offset 0x3a-3b short event at input counter(less than three 64bit words)
  offset 0x3c-3d number padded words for short event
  (offset less than 0x40 are from the backplane link module built inside the AMC module.)
  offset 0x6a-6b is the same as offset 0x6-7, but counted inside AMC13
  offset 0x6c-6d is the same as offset 0xa-b, but counted inside AMC13
  offset 0x6e-6f is the same as offset 0x8-9, but counted inside AMC13
  offset 0x70-71 is the same as offset 0x12-13, but counted inside AMC13
  offset 0x78-79 is bad AMC event CRC counter
  offset 0x7a-7b is TTS state is error counter
  offset 0x7c-7d is TTS state is out of sync counter
  offset 0x7e-7f is TTS state is disconnect counter

  If all these counters are 0, there is no data integrity problem detected.

e) Other T1 registers containing important run information
  0x46 number of L1A received
  0xba	low word of SFP0 sum of event length from CDF trailer
  0xbb	bit 55-32 of SFP0 sum of event length from CDF trailer
  0xbc	low word of SFP1 sum of event length from CDF trailer
  0xbd	bit 55-32 of SFP1 sum of event length from CDF trailer
  0xbe	low word of SFP2 sum of event length from CDF trailer
  0xbf	bit 55-32 of SFP2 sum of event length from CDF trailer
  0xc0	SFP0 built event count
  0xc1	SFP1 built event count
  0xc2	SFP2 built event count
  0xc4	SFP0 built event word count(lower 32 bit)
  0xc5	SFP1 built event word count(lower 32 bit)
  0xc6	SFP2 built event word count(lower 32 bit)
  0xc8	SFP0 built event block count
  0xc9	SFP1 built event block count
  0xca	SFP2 built event block count

  in the range of 0x800-0xdff, for each AMC module:
  offset 0xc-d  number of events received at link input
  offset 0x18-19  number of words received at link input
  offset 0x40-41  number of words received by AMC13 from AMC module
  offset 0x52-53  number of events received by AMC13 from AMC module
  offset 0x72-73  number of event blocks received by AMC13 from AMC module

f) Monitor buffer can buffer up to 0x400 events/blocks, each buffer occupies 
   0x20000 32 bit words.
  you can read any buffer using the following command:
  rv [starting address] [length]
  where starting address = 0x8000000 + (offset x 0x20000)
  e.g. the starting address of the first buffer is 0x8000000
  and the starting address of the second buffer is 0x8020000, etc.
  maximum offset is 0x3ff and the length is in 32 bit words.

Please email comments and suggestions to
wusx@bu.edu with subject as amc13debug
</pre>

-- Main.EricHazen - 23 Mar 2015
Topic revision: r17 - 11 Jun 2015 - EricHazen