MX with enhanced SCB2 – cards not coming online

25 03 2015

We just installed an MX in the lab for a customer type-approval test (TAT) and none of the cards came online. 

The output of “show chassis hardware” showed that there were FPCs installed, but not the MICs that were in them:

imtech@nge001.pre> show chassis hardware
Hardware inventory:
Item             Version  Part number  Serial number     Description
Chassis                                JN1249BDBAFA      MX960
Midplane         REV 04   750-047849   ACRD2400          Enhanced MX960 Backplane
FPM Board        REV 03   710-014974   CADE9287          Front Panel Display
PDM              Rev 03   740-013110   QCS181650BM       Power Distribution Module
PEM 0            Rev 11   740-027760   QCS1806N0MP       PS 4.1kW; 200-240V AC in
PEM 1            Rev 11   740-027760   QCS1806N0SK       PS 4.1kW; 200-240V AC in
PEM 2            Rev 11   740-027760   QCS1806N07S       PS 4.1kW; 200-240V AC in
PEM 3            Rev 11   740-027760   QCS1812N02D       PS 4.1kW; 200-240V AC in
Routing Engine 0 REV 01   740-051822   9013061577        RE-S-1800x4
Routing Engine 1 REV 01   740-051822   9013056762        RE-S-1800x4
CB 0             REV 01   750-055976   CACX9090          Enhanced MX SCB 2
CB 1             REV 01   750-055976   CACZ4497          Enhanced MX SCB 2
CB 2             REV 01   750-055976   CADA1721          Enhanced MX SCB 2
FPC 0            REV 05   750-044444   CAAM5562          MPCE Type 2 3D P
  CPU
FPC 1            REV 35   750-028467   CAAP9738          MPC 3D 16x 10GE
  CPU
FPC 2            REV 28   750-037355   CACZ6946          MPC4E 3D 2CGE+8XGE
  CPU
FPC 3            REV 29   750-037358   CADH7370          MPC4E 3D 32XGE
  CPU
FPC 4            REV 14   750-045372   CADC4195          MPCE Type 3 3D
  CPU
Fan Tray 0       REV 08   740-031521   ACDB4968          Enhanced Fan Tray
Fan Tray 1       REV 08   740-031521   ACDB4925          Enhanced Fan Tray

All a bit odd. Pressing the button on the front of the chassis made no difference, and requesting an FPC to come online at the CLI resulted in this:

imtech@nge001.pre> request chassis fpc restart slot 0
FPC 0 misconfigured

It turns out all of them were ‘misconfigured’:

imtech@nge001.pre> show chassis fpc
                     Temp  CPU Utilization (%)   Memory    Utilization (%)
Slot State            (C)  Total  Interrupt      DRAM (MB) Heap     Buffer
  0  Offline         ---FPC misconfiguration---
  1  Offline         ---FPC misconfiguration---
  2  Offline         ---FPC misconfiguration---
  3  Offline         ---FPC misconfiguration---
  4  Offline         ---FPC misconfiguration---
  5  Empty
  6  Empty
  7  Empty
  8  Empty
  9  Empty
 10  Empty
 11  Empty

The command ‘show chassis fpcs’ produced no output whatsoever:

imtech@nge001.pre> show chassis fabric fpcs

But the SCBs seemed to be online (in redundant mode) and operating normally:

imtech@nge001.pre> show chassis fabric plane
Fabric management PLANE state
Plane 0
  Plane state: ACTIVE
Plane 1
  Plane state: ACTIVE
Plane 2
  Plane state: ACTIVE
Plane 3
  Plane state: ACTIVE
Plane 4
  Plane state: SPARE
Plane 5
  Plane state: SPARE

imtech@nge001.pre> show chassis fabric summary
Plane   State    Uptime
 0      Online   23 hours, 13 minutes, 11 seconds
 1      Online   23 hours, 13 minutes, 5 seconds
 2      Online   23 hours, 12 minutes, 58 seconds
 3      Online   23 hours, 12 minutes, 52 seconds
 4      Spare    23 hours, 12 minutes, 46 seconds
 5      Spare    23 hours, 12 minutes, 40 seconds

So the conclusion was that there must be an FPC to SCB communication problem

An attempt was made to set the redundancy type (since it wasn’t specified in the almost-empty config) to see if this kicked the FPCs into life, but it didn’t help:

[edit]
imtech@nge001.pre# set chassis fabric redundancy-mode redundant

[edit]
imtech@nge001.pre# commit and-quit
re0:
configuration check succeeds
re1:
commit complete
re0:
commit complete
Exiting configuration mode

imtech@nge001.pre>

imtech@nge001.pre> show chassis fabric plane
Fabric management PLANE state
Plane 0
  Plane state: ACTIVE
Plane 1
  Plane state: ACTIVE
Plane 2
  Plane state: ACTIVE
Plane 3
  Plane state: ACTIVE
Plane 4
  Plane state: SPARE
Plane 5
  Plane state: SPARE

imtech@nge001.pre> show chassis fabric fpcs

Looking at the latest entries in the messages log file, the cause of the issue became obvious – the chassis needed to be in enhanced-ip mode to support these new SCBE2 fabric cards.  Since the config was blank, the required command was not there.  You can see below that the system is taking the FPCs offline because the chassis isn’t configured correctly:

imtech@nge001.pre> show log messages | last 30
Mar 25 12:11:16  nge001.pre chassisd[1703]: CHASSISD_FRU_OFFLINE_NOTICE: Taking FPC 1 offline: FPC misconfiguration
Mar 25 12:11:16  nge001.pre chassisd[1703]: It is required to configure enhanced_ip or enhanced_ethernet network service mode with MX Enhanced SCBE 2 board to power up FPC, current mode = 0x0
Mar 25 12:11:16  nge001.pre chassisd[1703]: CHASSISD_SNMP_TRAP10: SNMP trap generated: Fru Offline (jnxFruContentsIndex 7, jnxFruL1Index 2, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: MPC 3D 16x 10GE @ 1/*/*, jnxFruType 3, jnxFruSlot 1, jnxFruOfflineReason 27, jnxFruLastPowerOff 0, jnxFruLastPowerOn 0)
Mar 25 12:11:16  nge001.pre chassisd[1703]: CHASSISD_FRU_OFFLINE_NOTICE: Taking FPC 2 offline: FPC misconfiguration
Mar 25 12:11:16  nge001.pre chassisd[1703]: CHASSISD_SNMP_TRAP10: SNMP trap generated: Fru Offline (jnxFruContentsIndex 7, jnxFruL1Index 3, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: MPC4E 3D 2CGE+8XGE @ 2/*/*, jnxFruType 3, jnxFruSlot 2, jnxFruOfflineReason 27, jnxFruLastPowerOff 0, jnxFruLastPowerOn 0)
Mar 25 12:11:16  nge001.pre chassisd[1703]: It is required to configure enhanced_ip or enhanced_ethernet network service mode with MX Enhanced SCBE 2 board to power up FPC, current mode = 0x0
Mar 25 12:11:16  nge001.pre chassisd[1703]: CHASSISD_FRU_OFFLINE_NOTICE: Taking FPC 3 offline: FPC misconfiguration
Mar 25 12:11:16  nge001.pre chassisd[1703]: CHASSISD_SNMP_TRAP10: SNMP trap generated: Fru Offline (jnxFruContentsIndex 7, jnxFruL1Index 4, jnxFruL2Index 0, jnxFruL3Index 0, jnxFruName FPC: MPC4E 3D 32XGE @ 3/*/*, jnxFruType 3, jnxFruSlot 3, jnxFruOfflineReason 27, jnxFruLastPowerOff 0, jnxFruLastPowerOn 0)

To resolve this, simply enter the following command and reboot the MX:

[edit]
imtech@nge001.pre# set chassis network-services enhanced-ip

[edit]
imtech@nge001.pre# commit and-quit
re0:
[edit]
  'chassis'
    WARNING: Chassis configuration for network services has been changed. A system reboot is mandatory.  Please reboot the system NOW. Continuing without a reboot might result in unexpected system behavior.
configuration check succeeds
re1:
commit complete
re0:
commit complete
Exiting configuration mode
Advertisements

Actions

Information

8 responses

8 02 2016
Aditya Jambhale

Check SCB version,if it is Enhanced MX SCB 2 use below command to change chassis mode and reboot the router.

request chassis network service mode Enhanced-IP

reboot is required to apply these changes.

17 03 2016
DataPlumber

I think that’s what my post says to do, but thanks anyway.

21 07 2016
James

Interesting. I have a couple of MX960s in the lab and am trialling the MPC3E NG FPC, with the DWDM tunable MIC (100G). I’m running Junos 14.1R5.5 and am using SCBE-MX CBs.. This should be a suitable mix for the FPCs, and yet I get a very similar issue when trying to fire the FPCs up.

show chassis fpc:
5 Offline —FPC misconfiguration—

I have tried enhanced ip mode (was already on actually) and both redundant and increased-bandwidth mode.

If I find out what conditions I need to bring it up, I’ll let you know 🙂

J

22 07 2016
James

I found the solution.. Despite the documentation saying 14.1R4 is supported for the MPC (and I’m on R5), the MIC3-100G-DWDM isn’t supported until 15.1F5..

Back to the drawing board!

25 07 2016
DataPlumber

Thanks for sharing that – it would’ve tripped me up in the end…

25 07 2016
James

Now, upgrading to 15.1F6 and its new version of FreeBSD underlaying Junos.. Not going too smoothly 😀

20 08 2016
fr0d0

Are you running the

Juniper MPC-3D-16XGE-SFPP-R-B

or plain

Juniper MPC-3D-16XGE-SFPP

Curious cause documentations shows that you can only run enhanced-ip with -R-B

26 05 2017
Dave

thank you, this saved us. shocking how little information regarding this is available online

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s




%d bloggers like this: