0.0.1 DØ Trigger Upgrade

The upgraded DØ trigger system is shown schematically in Figure 1. Trigger decisions at Level 1 use data (trigger, or more specifically Level 1, data) which is provided by the hardware for just this purpose. Such data is not as precise as the final raw data, but is provided in a very short time, allowing queries to take place per crossing time (132ns). Decisions at this level are based upon what is seen in the various detectors (calorimeter, tracker, muon, and preshower), but for the most part without detector cross-correlation. Level 1 Accepts (which occur at 10 kHz) cause trigger data to be sent to level 2, which then makes physics requirements using all detector elements (with the addition of the silicon detector). Level 2 Accepts (which cannot occur much faster than 1 kHz) cause a data acquisition readout cycle into Level 3, where a processor farm uses reconstructed raw data. Level 3 output rates are expected to be in the range of 10-50Hz.

Figure 1: Schematic of the DØ Run II Trigger system.

Level 2 is built around the concept of preprocessor crates as outlined in figure 2. Working in parallel, each preprocessor feeds data into the Global Processor crate for Level 2 decision making. Each type of crate (both preprocessor and global) will consist of assorted VME cards for I/O and communication, with processing done on VME boards which consist of a Dec/ALPHA 21164 chip, running at 500MHz or greater (see figure 3). Note that in this figure, the term “Trigger Bus” is the known as the MBus, and is discussed below.

Since each preprocessor crate will create data for the Level 2 global decision, they are required to be part of the Level 3 DAQ readout. DAQ readout is accomplished in each crate via a VME bus master (the VBD card, built at Brown). This suggests that intra-

Figure 2: Schematic of the Level 2 trigger system.

crate communication for Level 2 business should not rely on the VME backplane in order to prevent L2 accepts from causing L2 query deadtime. Another data bus is therefore required. The Level 2 group will use a custom VME backplane bus built by the CDF Michigan group.

The bus, called “Magic Bus”, or MBus, consists of a card mounted on the VME crate backplane, with bus lines connected to P3 (the bottom row of pins on VME64x crates). The bus consists of 128 data bits and 32 address bits, is asynchronous, and supports both broadcasting and programmed I/O. At this time there are no comprehensive measurements as to maximum bus rates, however bus operation is relatively simple: load address and data lines, raise DSTROBE, the receiver latches the 128 data bits and raises DDONE. DDONE high is the signal to lower DSTROBE (which releases DDONE) and begin another cycle. This sequence is the same for both broadcast mode, and programmed I/O. Estimates for cycle times are around 50ns, which would correspond to a nominal 20MHz cycle time, giving 20MHz×16bytes=320Mbyte/sec throughput. The PCI (64 bits) bus in the ALPHA card is nominally rated at 33MHz, or 240MByte/sec, however the University of Michigan experts agree that the realized performance will be lower than this (more like 80MByte/sec), and is thus the determining factor in getting data in and out of the ALPHA. An MBus derating would therefore have to be more than x2 to become the bottleneck.

Any data used by any crate in the Level 2 system for any Level 2 purpose will come through an MBT card (Magic Bus Transceiver). This card is being designed and built by the Maryland HEP group under the supervision of Professor Baden, and plays a major role in the DØ Level 2 trigger upgrade.

Level 2 data (for decision making) enters the MBT and is bussed into and out of the ALPHA cards via MBus. Trigger framework control (from the Hub) enters the MBT via a mezzanine card and is also bussed to the ALPHA over Mbus. For a more detailed description of the DØ Level 2 system, see the TDR document DØ note 3402, February 1998,

Figure 3: Schematic of the Level 2 ALPHA processor.

http://www.pa.msu.edu:80/hep/d0/ftp/l2/overview/globaltdr/global tdr.pdf

for PDF format, and .ps for POSTSCRIPT. MBT Introduction

Table 1 summarizes the MBT Card. The MBT web page is maintained at Maryland, and can be found at

http://macdrew.physics.umd.edu/dzero/trigger/mbt.html

A functional diagram of the MBT card is seen in Figure 4. The MBT card is a single-width 9U 400mm VME-64 for Physics card which collects all of the support functions that are needed in the ALPHA Processor based Level 2 Trigger Crates onto one card. In order to achieve this, the MBT will have 7 HotLinks serial digital data inputs running at 160 Mb/sec (or 16 MByte/sec in encoded mode, see below), 2 HotLinks data outputs running at the same rate, 1 HotLinks output for Level 1 SCL information fanout purposes, an SCL mezzanine card (for trigger hardware framework communication), a special 128 bit ECL output line which will deliver 16 bytes of information in about 0.5μsec or less (for Level 2 hardware framework communication and other purposes), and both VME and MBus interfaces. We will adopt the philosophy that all operations which do not occur in real time (with respect to data taking) will be over the VME bus, others will be over MBus.

The MBT card is designed around the Altera 10k100 FPGA (henceforth known as “Main”). This gate array is ball-grid-array mounted, and has a large number of gates and I/O pins (approximately 300 pins), and was chosen primarily in order to eliminate the need for an internal data bus. We therefore will keep as resident inside MAIN as many registers, counters, etc. as is possible. All MBus and VME data and address lines are connected to MAIN, as are the input FIFOs, and various other smaller gate arrays. The internal MBT clock will be 80MHz, or 12.5 ns “ticks”.

Internal Clock  80 MHz (12.5 ns)
VME Crate Buses  VME R/W, MBus R/W
Input Data Sources  7 HotLinks receivers, 16MB/sec
Output Data Sources 2 HotLinks transmitters, 16MB/sec

1 SCL Level 1 Fanout, 16MB/sec

SCL Mezzanine Card (MBT Pilot only)
L2 HWFW Output  128 ECL lines @ 5 MHz, or

16 PECL levels

MBT State Register VME Readable

Bit 0=1 indicates READY

Bit 1=0 for data, 1 for test inject

Table 1: MBT Summary.

In each Level 2 crate, there will be a “Pilot” MBT, and possibly one or more “Assistants”, selectable via a switch on the board. A front panel LED will be lit for the PILOT. Differences between the two are detailed in the TDR proposal (available from the MBT website), but mainly have to do with functions not related to data taking (e.g. trigger HUB communication and MBus broadcasting).

The MBT card has four main functions:

1. Receiver for Input. This is used in the Global Crate to receive Preprocessor data and in some of the Preprocessors Crates to receive input data from their Level 1 trigger data transmitters. There are 7 channels of Cypress HotLink input running differential over unshielded twisted pair at 16 MB/s (described below). Data received on these links is buffered and then sent via the MBus to the ALPHA processor boards.

2. Transmitter for Output. This is used in the Preprocessors to send their data to the Global Crate. There will be 2 channels of Cypress HotLink output, also running differential over unshielded twisted pair at 16MB/s. Data sent out on these links comes from the MBus, passes through a FIFO and is then sent out.

3. Communication to the L2 Hardware Framework (L2HWFW). The Global processor will need a way to send the L2 decision result back to the L2HWFW. This will be via 128 lines of Parallel Digital I/O, driven by the MBT card. This can also be used in both the Preprocessor and Global Crates for real-time scaler and state control.

4. Serial Command Link (SCL) Receiver. This is used in both the Preprocessor and Global crates. The L1 Accept, L2 Decision, and SCL Initialize information is received and made available to the ALPHAs via the MBus (see below). In addition, the L1 Accept information will be sent out on shielded twisted pair using the same HotLinks transmitters. The status “Error” and “Busy” for both Level 1 and Level 2 are communicated back to the SCL hub through the SCL.

Figure 4: The MBT card.

Data Transmission

Data into the Level 2 preprocessor crates is transmitted from the Level 1 system over a copper serial link running at 16 MByte/sec. Data from the preprocessor crates is sent to the Global crate over the same link. For the physical layer, we have chosen to use Cypress HotLink chips. The transmitter/receiver chipset is used for point-to-point communication, and serves as building blocks for systems which transfer data over high-speed serial links (fiber, coax, and twisted pair). For technical information, see

http://macdrew.physics.umd.edu/dzero/trigger/mbt.html for the HotLinks receiver transmitter description documents (copied from Cypress).

We will be using the transmission mode “Encoded mode” which consists of 8 data bits encoded with 2 control bits, transmitted serially over positive differential “pseudo” ECL (PECL). The data byte rate (16 MByte/sec) is 1/10th the transmission speed. The output register of the HotLinks receiver presents 8 data bits in parallel, a byte rate clock generated from the incoming (transmitted) data (CKR), a signal for “Data Ready” (\overline{RDY}), a “Special Character” flag bit (SC/\overline{D}), and an error bit (Received Violation Symbol, RVS). The state machine running in the PLD will be keying on \overline{RDY}, SC/\overline{D}, and RVS.

Tests at Maryland indicate that the system is very robust. Using a single transmitter and receiver pair running at 250 MB/s in BIST mode over 100 feet of unshielded balanced twisted pair (UTP) cable used for 1000-Base-T ethernet resulted in no errors after 17 hours of running overnight, or less than 1 error in in 1.53 × 10^{13} bits. This is translated into a bit error rate (BER) of less than 6.5 × 10^{−14}. The “eye” plot is shown in figure 5. Note that 3 of the 4 pairs are grounded.

Figure 5: “Eye” plot for 250MHz in BIST mode over 100 feet of 1000Base-T UTP cable.

Maryland Contribution

The entire MBT project resides at Maryland. The functional design has been a 15 month effort, with Professor Baden working closely with the DØ Level 2 trigger group as lead by Professor Jim Linnemann from Michigan State University. The HEP group electronics engineer Rob Bard has co-led the design with the Physics Department Electronics Group senior engineer John Giganti. HotLinks prototypes have been built and tested here at Maryland. At this time (April 2000) a pre-production prototype MBT card is being built. The previous production modules have been successful, passing all electrical tests. Professor Baden is writing Verilog code for the FPGA, along with help from EE students. The project is on schedule to produce all 30 cards by May 2000, fully programmed and ready for installation. We anticipate that we will meet design specifications, functional specifications, cost, and schedule. VME Rack Engineering

In the fall of 1999, DØ asked the Maryland group for help in engineering the VME racks for use in the Level 2 system. Professor Baden took on this additional responsibility. The work involves the design and production of the cable harness for the power delivery, rack monitoring, engineering of the power supply chassis, and the development of electronics for managing the power in the various supplies (±12, ±5, and + 3.3 volts). The work is almost complete at this writing (April 2000), with delivery of a prototype rack to FNAL schedule for mid April, and necessary safety documentation available soon after.