WBS 6.2.2
Readout Electronics
Evelyn Thomson
Level-3 Manager & CAM The University of Pennsylvania
U.S. ATLAS HL-LHC Upgrade Project Scrubbing Meeting Brookhaven National Laboratory
Upton, NY
Outline
• Technical Details
▪ Deliverable Overview, institutional responsibilities
▪ R&D Status and Plans
▪ Technical Progress in FY18
▪ Pending Issues
▪ Plans for pre-production
▪ Plans for production
• Schedule and Cost
▪ How has the schedule changed since CD-1?
▪ How as cost changed since CD-1
• Risk and Uncertainty
• Closing Remarks
Technical Details
Deliverable Overview
This WBS covers front-end readout ASICs, and powering electronics for the silicon strip tracker.
Basic unit of tracker is a silicon strip sensor, one or two readout hybrids, and one integrated powering board (IPB).
Star chips: each barrel readout hybrid has 10 ABCStar (front-end) and 1 HCCStar (controller) o Penn & UCSC → ABCStar design only, US will pay for 88,631 chips (37.9% of entire tracker).
o Penn & UCSC → HCCStar design, Penn probes wafers, supply 25,536 chips for entire tracker.
Powerboard: IPB implements all LV, HV, monitoring, & power control functions in a single PCB.
o LBNL → design of barrel IPB, test, burn-in, supply 10,976 boards for barrel only.
o CERN provides bPOL12V circuit design and supplies chips, US will pay for barrel only.
o Penn → AMAC design, probe wafers, supply 16,352 chips for entire tracker.
o Yale → design, testing of 16,352 coils for entire tracker.
o BNL → HV Mux design, prototyping, production of 17,888 pieces for entire tracker.
(NB: numbers to install on this slide, yield factors and numbers to fabricate on next slide).
Deliverable Overview
Total number of components larger than number required to install
▪ System yield factor includes losses in downstream assembly
▪ Component yield factor accounts for losses in fabrication of components
▪ Pre-production of 5% for quality assurance
Deliverable
Number to
install
System yield factor
Component yield
factor
Number for
production
Pre-
production (5%)
Total with pre-
production ABCStar (37.9% of
barrel+endcap) 88,631 1.19 1.13 119,482 5,974 125,456
HCCStar (barrel+endcap) 25,536 1.19 1.13 34,424 1,721 36,146
AMAC (barrel+endcap) 16,352 1.19 1.13 22,044 1,102 23,146
Coils (barrel+endcap) 16,352 1.19 1.07 20,910 1,145 22,056
Deliverable Overview
Prototyping phase (ends FY19 *) dates range for sub-projects
✓ Design and testing of prototype ABCStar, HCCStar, AMAC v2 by Penn and UCSC
✓ Radiation hardness, reliability evaluation of GaN-FET based circuit HV Mux by BNL
✓ Coil and shield tests by Yale, order 1000 coil sets by Yale
✓ Design, fabrication, and testing of V2 and V3 of the IPB by LBNL Pre-production phase (FY19 to mid-FY20 *)
▪ Preproduction of ABCStar, HCCStar, AMAC design revisions by Penn and UCSC, and testing of HCCStar and AMAC by Penn
▪ Order pre-production 1150 GaN-FET devices, perform QC tests + irradiation
▪ Fabrication and testing of 21,500 production coil sets by Yale.
▪ Preproduction IPB design revision, fabrication of 250+750 boards, testing by LBNL Production phase (late-FY20 to FY22 *)
▪ Financial contribution to ABCStar production
▪ Production HCCStar and AMAC: fabrication, testing, and distribution by Penn
▪ Production order for 23,000 GaN-FET devices, QC tests including irradiations by BNL
▪ Production IPB fabrication and testing of the IPB, including burn-in, by LBNL
Progress in FY18 and FY19
Star chips prototypes (ABCStar, HCCStar, AMAC V2a)
✓ Submitted for fabrication in June 2018, back from fab in Nov 2018.
✓ Single-chip tests successful, chips operational.
✓ Wafer probing at RAL underway; ordered probe station for Penn.
✓ First hybrid with star chips (Dennis Sperlich, module meeting 5-March-2019)
▪ Extra design effort was required for prototype design due to complexity of multi- level readout scheme, collaboration changes in specifications.
▪ Extensive simulation and verification efforts mitigated risk of chip failure.
▪ BCP-004 in Sept 2018 updated design FTE effort through production.
Powerboards
✓ V3 powerboard fabricated, first batch distributed. Second batch in fabrication for Star chip modules. Noise leakage tests.
▪ Extra costs for pre-production tester boards and production PCB (vendor quotes).
▪ BCP-005 in Feb 2019 updated material costs.
Single chip tests
• Photo from downstairs at Penn
• Photo from RAL of wafer probing
Wafer probe tests
• Photo from downstairs at Penn
• Photo from RAL of wafer probing
Pending Technical Issues
Star chips: implementation of revisions from prototype to pre-production
• Extensive verification required for revisions since pre-production is final version.
• Note well that there should be NO changes between pre-production and production
• Design revisions and verification must be well advanced before FDR in June 2019.
• After FDR, only have 2 months for final changes before pre-production order.
Star chips are operational, issues being tracked on JIRA. Critical issues are:
• ABCStar
▪ Improve PRLP decoder to guard against bad input.
• HCCStar
▪ 640 MHz clock signal too weak, fix description of output pin.
▪ AM comparator switch (change resistor type from PMOS to LPPMOS)
• AMAC
▪ Damage to AMAC from linPOL12V during radiation testing
▪ Load eFuse id with broadcast command instead of writing to register.
▪ AM comparator switch (change resistor type from PMOS to LPPMOS)
▪ More SEU mitigation.
Details on ABCStar
• ABCStar bad input
▪ Reporters: Ben Rosser in simulation (cocotb), Bruce Gallop with ABCStar
▪ Problem: Send a "PR" on the LP line-- the PR prefix 0b101 followed by seven bits of L0 tag and a trailing 0- and then try to send a LP with the correct prefix (110) a few clocks later, the subsequent LP is ignored unless a logic reset is issued first.
▪ Diagnosis: ABCStar's PRLP decoder is not protected against bad inputs.
This could be a serious problem, because the HCCStar's PRLP output serializer is not hardened against SEUs-- meaning it's not entirely unexpected to have random noise on those lines.
• To do: identify other bad input cases with simulation tests,
single chip tests, module tests
Details on 640 MHz output
• 640 MHz output – to be expanded
▪ Output pin not defined as “clock” in Cadence
▪ Extra capacitance on line
▪ Signal too weak
▪ Change output pin…
Plans for Pre-Pro and Production
Star chips - Paul Keener at Penn is the international activity co-ordinator for ASICs
▪ Schedule in P6 consistent with international schedule
▪ Design revisions for pre-production already in progress (next slide)
▪ Wafer probe station arrives at Penn in March 2019 (next slide)
▪ FDR in June 2019, pre-production order in August 2019, chips available Jan 2020
▪ PRR in June 2020, production order in July 2020, first chips available Feb 2021 Powerboards
▪ FDR in June 2019 as part of modules
▪ Pre-production set of 250+750 parts from Fall 2019 through Summer 2020 for PRR
▪ Fabrication vendor and component loading vendor have been found
▪ Developing large scale testing and thermal cycling system for vendor External dependencies
▪ CERN bPOL12V
HCCStar design revisions in P6
Activity ID Activity Name Start Finish Duration Float Hours Labor Total Cost RE320290
Perform Initial Functional Tests of
HCC* on Single Chip PCB 10/29/2018 3/1/2019 86 1 732 28,219
RE320470
Review HCC* Functional Test
Results from Single Chip PCB 3/12/2019 3/25/2019 10 1 315 $25,221
RE320490 Revise HDL Design 3/12/2019 4/22/2019 30 2 250 $32,130
RE320540
Review HCCstar Chip Design Block
by Block Check 3/26/2019 4/26/2019 24 1 804 $51,826
RE320550
Performance Review of HCCstar
Design 4/29/2019 4/29/2019 1 1 85 $11,698
RE320570
Simulate Design Changes in Star
Chipset and Hybrid 5/1/2019 6/5/2019 25 1 721 $18,688
RE320580
Revise Physical Design &
Verification 4/23/2019 6/4/2019 30 2 698 $24,380
RE320590 FDR Preparation 6/6/2019 6/13/2019 6 1 115 $13,741
RE320620 Design Revision 6/14/2019 7/26/2019 30 1 175 $20,643
RE320630 Revision Simulation 6/14/2019 8/21/2019 48 1 387 $13,002
Design and layout revisions take 30 days
• Allow for 3 cycles
HCCStar design revisions in P6
Activity ID Activity Name Start Finish Duration Float Hours Labor Total Cost RE320290
Perform Initial Functional Tests of
HCC* on Single Chip PCB 10/29/2018 3/1/2019 86 1 732 28,219
RE320470
Review HCC* Functional Test
Results from Single Chip PCB 3/12/2019 3/25/2019 10 1 315 $25,221
RE320490 Revise HDL Design 3/12/2019 4/22/2019 30 2 250 $32,130
RE320540
Review HCCstar Chip Design Block
by Block Check 3/26/2019 4/26/2019 24 1 804 $51,826
RE320550
Performance Review of HCCstar
Design 4/29/2019 4/29/2019 1 1 85 $11,698
RE320570
Simulate Design Changes in Star
Chipset and Hybrid 5/1/2019 6/5/2019 25 1 721 $18,688
RE320580
Revise Physical Design &
Verification 4/23/2019 6/4/2019 30 2 698 $24,380
RE320590 FDR Preparation 6/6/2019 6/13/2019 6 1 115 $13,741
Extensive simulation and verification effort
• Significant uncosted labor hours here
Schedule and Cost
Schedule
Star chips are on critical path:
▪ ABCStar in hand and works, design is US deliverable
▪ HCCStar in hand and works, US deliverable Powerboard dependencies
▪ Coil – prototype and pre-pro design in hand and 1000 more ordered, low risk
▪ bPOL12V – from CERN. Concern is final operating voltage may be lower than planned for, leading to lower efficiency
▪ LinPOL12V – from CERN. Already in hand
▪ HV Mux – pending down select, US controls access to this
▪ AMAC – V2a in hand and works, US deliverable
Deliverabley Prototype Phase Pre-Production Production
Start End Start End Start End
HV-mux 2015 2017-10-12 2017-10-13 2019-04-17 2019-04-18 2020-12-24 IPB 2016-10-03 2019-04-08 2019-04-09 2020-03-02 2020-02-26 2023-04-05 ABCStar 2016-10-03 2019-03-11 2019-03-12 2020-01-23 2020-01-24 2020-06-18
Milestones
MILESTONE DATE TASK
BNL: HV mux decision 17-APR-2019 RE140380 BNL: HV mux prod order 20-JAN-2020 RE140390 BNL: HV mux delivery complete 24-DEC-2020 RE140560 LBNL: PB pre-pro end 02-MAR-2020 RE261220 LBNL: PB prod end 05-APR-2023 RE262500A Yale: prod coils delivered 18-NOV-2019 RE530630 Penn/UCSC: ABC end of design 18-JUN-2020 RE410250 Penn: HCC pre-pro end 23-JAN-2020 RE320850 Penn: HCC prod first chips 18-FEB-2021 RE321180 Penn: HCC prod end 18-MAY-2021 RE321450 Penn: AMAC pre-pro end 23-JAN-2020 RE350630 Penn: AMAC prod first chips 18-FEB-2021 RE350830 Penn: AMAC prod end 03-AUG-2021 RE351050
Items needed for CD-3a
BNL HV MUX test RE140420M-RE140530M 1/17/2020 71.5
CERN
HV MUX
parts RE140190M 1/17/2020 895.4
LBNL PB RE261330M - RE261430M 8/20 - 12/20 402.5
CERN BPOL12V RE261405M 6/2/2020 210.4
CERN ABC RE310390M 6/18/2020 853
CERN HCC AMAC RE321030M 7/8/2020 472.8
Penn HCC AMAC Don't know where this is 6/20/2020 10
Yale Coil RE530450M 2/6/2019 55.7
List of material for PRODUCTION that has to be purchased before January 2021
Double-check in P6, improve format
Cost change after CD-1: BCP-004
• To be added
Cost change after CD-1: BCP-005
• LBNL FY18+ Powerboards added preproduction tester boards and production PCB order boards, updated industry quotes, documented in new BOE
WBS BCP-004 Now in P6 Delta
FY18+
Delta FY17 Total Delta
6.02.02 Readout
Electronics 4,618,368 5,499,605 164,673 716,564 881,237
6.02.02.01-BNL 383,815 480,340 1,332 95,192 96,524
6.02.02.02-LBNL 1,841,527 2,121,778 264,478 15,773 280,251
6.02.02.03-Penn 1,880,772 2,342,056 2 461,282 461,284
6.02.02.04-UCSC 355,674 346,214 (101,139) 91,679 (9,460)
6.02.02.05-Yale 156,579 209,217 (0) 52,638 52,638
Risk and Uncertainty
Risk and Uncertainty
• 6.2.2 Readout Electronics Threat RD-06-02-02-001 Active 17-Oct-17 1-Jan-19
• Title: HV Mux does not converge and we need to resort to making choices based on available cabling. We plan an HV mux on the powerboard, per module. We are still doing a technology selection and the parts are boutique. While promising we could end up with no solution
• Mitigation: One of two alternative technologies was chosen in FY17 to focus the resources on the most promising solution and maximize the chances of making it work. A program of
testing and irradiation is being pursued to validate the technical performance.
• Response: If the technology fails we can use extra lines on the stave bus tape to supply the additional HV lines.
• Comment: The technology is novel and may not be commercially available in the un- packaged form we need in the experiment. If it fails this will impact the tape design and granularity of HV control on the strip modules. As a last ditch solution, to preserve the
schedule, we might build some staves w/o HV Mux while continuing to work on the HV Mux solution. Maximum cost is due to a 1 year engineering delay.
• RE140180
Risk and Uncertainty
• 6.2.2 Readout Electronics Threat RD-06-02-02-002 Active 17-Oct-17 27-May-20
• Title: Chip run fails or radiation effects cause concern for readout at high rate and we need to do another run.Everything else is dependent on chip fabrication and performance both electrical and radiation. Chip iterations can take nearly 1 year so any failure of technical issues propagates a delay
• Mitigation: A thorough program of chip performance simulation during design phase, ASIC design review, and post-fabrication testing, including irradiations.
• Response: Determine cause of the problem through chip testing and its performance simulations, revise design, review with panel of experienced ASIC experts, resubmit for fabrication.
• Comment: Detect problem 2 - 4 months. Determine cause, 1 month Revise circuit, layout, or compiler issue, 1 month. Review by experts, 2 weeks, Revised fabrication 3 months. There are many scenarios where the chip may need to be refabricated even though it is mostly functional, in this case the loss would likely be cost of labor to fix the problem and cost of refabrications.
This problem also propagates into the other production activities causing project wide delays.
We estimate that, due to time scales associated with the root cause determination and assembly sequence in the project, we could loose about half a year worth of labor time. At flat top
production with corresponds to $2.5M.
• RE350490
Risk and Uncertainty
• 6.2.2 Readout Electronics Threat RD-06-02-02-003 Active 17-Oct-17 27-May-20
• Title: Chip set not compatible with lpGBT which won't be ready until just before or after production wafer orders for HCC are placed. The lpGBT sites on the end-of-stave and communicates with the outside world. HCC ties directly to lpGBT
• Mitigation: lpGBT will need to be tested with HCC in a realistic physical prototype. Point to point 640 Mbps and multi-drop conditions should be fully explored to determine the level of
communication robustness. It may be that EOS components are able to mitigate the problem or that a new submission of an HCC is required. It is unlikely that stave or endcap prototyping will be halted due to partially reliable communications. The HCC will retain a 320Mbps data rate in
addition to the default of 640Mbps.
• Response: Move to fall back data transmission rates to allow continued testing of prototypes.
Determine lowest cost mitigation that provides low or no bandwidth compromises. Implement correction on EOS or Hybrid or on HCC ASIC.
• Comment: Prototypes of the HCC LVDS transmitters and receivers have been fabricated and work at their design frequency. The lpGBT is being designed by CERN Microelectronics and in
collaboration with other institutions with proven communications circuit design ability. The risk of total failure is low, but ther may be some reliability issues that need work. This would also cause
Closing Remarks
• Status: on schedule for FDR in June 2019
▪ First prototype star chips operational in single chip tests
▪ First module with star chips
▪ Powerboard V3 first batch fabricated and distributed
• Costs
▪ BCP-004 updated costs for readout electronics design FTE effort
▪ BCP-005 updated costs for powerboard materials
• Risks still remaining
▪ Unforeseen issue with star chips on module and stave tests
▪ End-of-stave communication lpGBT problem
BACKUP
Bio Sketch of L3 Manager
• Evelyn Thomson, Associate Professor, University of Pennsylvania
• ATLAS: 2007-present
• ATLAS responsibilities
▪ L3 manager for readout electronics
▪ Supervisor of Dr. Jeff Dandoy and graduate student James Heinlein on ITk
▪ Supervisor of graduate students Dominick Olivito, Ian Dyckes on TRT
▪ ATLAS speaker’s committee, US ATLAS speaker’s committee
• Previous experiments
▪ CDF – Top quark physics group leader, track trigger (XFT)
▪ ALEPH
Institute Capabilities
Institute Deliverable Work type Facility
BNL HV-mux Evaluation, Irradiation Lab benches, gamma source LBNL Powerboard Design, evaluation Extensive engineering
facilities: PCB design, evaluation
Penn ABCstar, HCCstar, AMAC
Design, testing CAD design system, circuit testing labs, wafer probe station
UCSC ABCstar, HCCstar Design CAD design systems
Yale Coils Evaluation Renovated evaluation lab
Powerboard Costs in BoE
The power board estimate is described in Attachment 3, supported by Attachments 4-8.
• The passive components cost $2.58 per board, and the total cost is $39k for parts for 15,000 boards.
These costs are well known from catalogs. Attachment 4 has the bill of materials from January 2017, which shows a higher cost of $3.88 per board. This needs to be updated.
• The fabrication of specialized shield boxes costs $4.14 each, and the total cost is $62k for 15,000 boxes.
Attachment 5 has the quote from Tech-Etch from June 2017 shows a cost of $3.41 per box for orders of over 5,000 boxes before including the costs of procurements and escalations. The quote from April 2018 should be updated to include quotes for orders over 500 parts.
• The fabrication of the flexible circuit of the IPB costs $10.19 per board, and the total cost is $153k for 15,000 boards. Attachment 6 has the quote from epec from August 2017, which shows $4.93 per board for orders of 1000 boards before including the costs of procurements and escalations. This quote needs to be updated.
• The loading of components onto the IPB costs $58.50 per board, and the total cost is $877k for 15,000 boards. Attachment 7 has the current quote from AmTech from January 2019 , which shows $50.00 per board before including the costs of procurement and escalations.
• The DC-DC converter bPOL12V and linPOL12V chips are supplied by CERN. Attachment 8 shows estimates of the cost of $210k for bPOL12V and $33k for linPOL12V.
• The HVMux is a separate WBS item 6.2.2.1.4 designed and tested by BNL.
• The AMAC chip is a separate WBS item 6.2.2.3.5 designed and tested by Penn.
• The coil is a separate WBS item 6.2.2.5.3 supplied by Yale.
• Attachment 3 also estimates the cost of 1500 test boards at $118k, based on engineering experience.
• Attachment 3 also estimates the cost of 50 active boards at $20k, based on engineering experience.
Risk and Uncertainty
• Summarize main risks for this deliverable (highest ranked)
▪ These should be copied directly from the Risk Register
• Discuss risks which should be retired or downgraded
• Discuss new risks which have been/should be added
• New guidance is to add additional risks, discuss these
▪ See risk guidance slide from PO, next slide after this!
▪ All external dependencies now become risks
▪ Risks from loss of key personnel
▪ These new risks will have to be described in a mature way by the May
2019 Director’s review
Risk Checklist from PO
• Basic maintenance of all risks
▪ Check if risks are still active
▪ Revisit tasks affected column in RR, especially after recent RLS changes
▪ Recheck comments section for each risk. Is there an explanation for how risk probabilities and impacts were estimated? Can a reviewer reconstruct the impact ranges from the information provided?
▪ Check that every external dependency has a risk associated with it.
• Add new risks
▪ Risk of loss of key personnel. One risk per deliverable.
▪ Risk of needing additional labor force due to unexpected complexities that arise. Estimate as the labor cost of one prototype iteration.
• Maturity scores
▪ Check that all tasks in P6 have maturity scores assigned
▪ Retune if necessary the scores (for example if R&D performed to date has improved the maturity)
