Skip to main content

Observability User Guide

4.4. Location connection failures workflow

This drill-down workflow is designed to analyze connection failures for your locations, which are configured using analytics labels. It includes eleven dashboards organized across four hierarchical levels, drilling down from the Locations Overview dashboard to the more detailed Troubleshooting dashboards. Dashboards in this workflow can traverse up, down, and across to switch views and perspectives on the same data set. Zoom out by using the browser's back button or navigate laterally to dashboards with the same level of detail but from different perspectives, such as endpoints, protocols,monitoring points, or peers. To learn more, see Navigate the workflowsNavigate the workflows

The following diagram gives you an overview of the different dashboards and their hierarchical relationship:

location_connection_failures_workflow_diagram_noshadow.png

The following information summarizes each level of the workflow:

Location Connection Failures Overview

The Locations Overview is the top level of the Location Connection Failures workflow. Here, all of your custom locations that have been active for the last 24 hours are listed.

Using a time-series database, metrics such as server latency, client latency, percentage of inbound and outbound connection failures, and the total number of inbound and outbound connections for each location are visualized and analyzed in real-time. The default metric displayed on the graph is server latency.

Level 1

The Level 1 dashboard is designed to help you determine the health and performance of your locations. It compares connection metrics across various aspects of the service, such as monitoring points, endpoints, protocols, and peers. The default metric displayed on the graph is failure % - inbound connections.

When the host acts as a server, metrics are displayed for the percentage of inbound connection failures, inbound connections, SYN Rx, and SYNACK Tx. Conversely, metrics are shown for the percentage of outbound connection failures, outbound connections, SYN Tx, and SYNACK Rx when the host acts as a client. You will see both sets of metrics if the host has acted as a server and a client.

One variable for Resolution is available at the top of the dashboard. To learn more about this variable, see Dashboard variables.

Level 1 dashboards include:

  • Connection Failures for a Location by Endpoints

  • Connection Failures for a Location by Monitoring Points

  • Connection Failures for a Location by Protocols

  • Connection Failures for a Location by Peers

Level 2

Level 2 dashboards help narrow down the source of connection issues. You select a specific location and examine metrics at a distinct monitoring point, endpoint, or peer associated with that location. The same metrics are available for Level 1 and Level 2 dashboards, which focus only on connections. The default metric displayed on the graph is failure % - inbound connections.

When the host acts as a server, metrics are displayed for the percentage of inbound connection failures, inbound connections, SYN Rx, and SYNACK Tx. Conversely, metrics are shown for the percentage of outbound connection failures, outbound connections, SYN Tx, and SYNACK Rx when the host acts as a client. You will see both sets of metrics if the host has acted as a server and a client.

One variable for Resolution is available at the top of the dashboard. To learn more about this variable, see Dashboard variables.

Level 2 dashboards include:

  • Connection Failures for a Location and an Endpoint by Monitoring Points

  • Connection Failures for a Location and a Monitoring Point by Endpoints

  • Connection Failures for a Location and a Protocol by Endpoints

  • Connection Failures for a Location with a Peer by Monitoring Points

Level 3

The Level 3 dashboard, Troubleshooting Location (Endpoint Summary), is used to troubleshoot connection issues for a location at specific endpoints. If the IP address is known, you can start troubleshooting directly from this dashboard. As a starting point, it provides various metrics such as throughput Tx, throughput Rx, packets Tx, packets Rx, bytes Tx, bytes Rx, maximum flows Tx, maximum flows Rx, and server latency. The default metric displayed on the graph is throughput Tx. Additional metrics are grouped to help troubleshoot different types of observed issues. To learn more, see Metrics for advanced analysis.

Four variables are available at the top of the dashboard: Endpoint, Service, cStor, and Resolution. To learn more about these variables, see Dashboard variables.

PCAP files can be downloaded for a particular endpoint from the cStor Download tag or a specified range using the cPacket Capture panel. To learn more, see Capture packet data.

Level 4

The level 4 dashboard, Troubleshooting Location (Endpoint Conversations), is utilized for advanced and targeted analysis of a specific data point or narrow time range. An on-demand fetch of the analytics is always required using the cPacket Analytics panel. To learn more, see On-demand analytics collection.

Six variables are available at the top of the dashboard: Endpoint, Peer, Location, cStor, Resolution, and Include Excluded Locations. To learn more about these variables, see Dashboard variables.

See Level 3 for metric and packet capture information.

In this section:

A location is a user-defined group of endpoints that identify a physical location within your network topology. Locations can be nested, where one location is entirely contained within another, such as a data center rack, a hospital floor, or a specific building within a campus.  To define Locations, use the Locations analytics label in the cClear UI.

An endpoint is a server, client, or peer as identified by a unique IP address.

A protocol is the combination of TCP or UDP and a port number used to identify a specific application, also known as a network application. For example, DNS uses port 53, and HTTPS uses port 443. To define protocols, use the Protocols analytics label in the cClear UI.

A monitoring point is the location in the network where the monitoring stack captures packets, typically a TAP, SPAN port, or remote SPAN such as ERSPAN, VXLAN, or GRE tunnel. Each monitoring point is uniquely identified by a combination of a cStor and a packet brokering port, such as 'Hoboken FW1 External'. Sources for this identifier include VLANs, cVu ports, or GENEVE VNI. To define monitoring points, use the Monitoring Points analytics label in the cClear UI.

A peer is a collection of endpoints on the other end of a monitoring point from the entity being monitored, such as a client or server.

Server latency is the round-trip time between the client sending a TCP packet and the server acknowledging the packet, as observed by the monitoring point.

Client latency is the round-trip time between the server sending a TCP packet and the client acknowledging the packet as observed by the monitoring point.

A TCP connection error is any failure in establishing, maintaining, or closing a TCP connection.

The total number of connections that have successfully completed a three-way TCP handshake (SYN-SYNACK-ACK).

The percentage of inbound connection attempts that failed to complete a three-way TCP handshake (SYN-SYNACK-ACK) where the host is a server.

A server is an endpoint that receives a SYN packet from a client and responds with a SYN-ACK packet.

The total number of inbound connections that have successfully completed a three-way TCP handshake (SYN-SYNACK-ACK) where the host is a server.

The total number of SYN packets received for the specific endpoint or application where the host is a server.

The total number of SYN-ACK packets transmitted for the specific endpoint or application where the host is a server.

The percentage of outbound connection attempts that failed to complete a three-way TCP handshake (SYN-SYNACK-ACK) where the host is a client.

The total number of outbound connections that have successfully completed a three-way TCP handshake (SYN-SYNACK-ACK) where the host is a client.

The total number of SYN packets transmitted for the specific endpoint or application where the host is a client.

The total number of SYN-ACK packets received for the specific endpoint or application where the host is a client.

A client is an endpoint that initiates the TCP connection by sending a SYN packet.

The total data in bits per second transmitted by the endpoint or application calculated over the resolution of the dashboard.

The total data in bits per second received by the endpoint or application calculated over the resolution of the dashboard.

The total number of packets an endpoint or application transmitted.

The total number of packets an endpoint or application received.

The total number of bytes an endpoint or application transmitted.

The total number of bytes an endpoint or application received.

The maximum number of transmitted flows by a host as observed by a monitoring point.

The maximum number of received flows by a host as observed by a monitoring point.