Skip to main content

Observability User Guide

4.3. Location latency workflow

This drill-down workflow is designed to analyze latency in your locations , which are configured using analytics labels. It includes eleven dashboards organized across four hierarchical levels, drilling down from the Locations Overview dashboard to the more detailed Troubleshooting dashboards. Dashboards in this workflow can traverse up, down, and across to switch views and perspectives on the same data set. Zoom out by using the browser's back button or navigate laterally to dashboards with the same level of detail but from different perspectives, such as endpoints, protocols,monitoring points, or peers. To learn more, see Navigate the workflows.Navigate the workflows

The following diagram gives you an overview of the different dashboards and their hierarchical relationship:

location_latency_workflow_diagram_noshadow.png

The following information summarizes each level of the workflow:

Location Latency Overview

The Locations Overview is the top level of the Location Latency workflow. Here, all of your custom locations that have been active for the last 24 hours are listed.

Using a time-series database, metrics such as server latency, client latency, percentage of inbound and outbound connection failures, and the total number of inbound and outbound connections for each location are visualized and analyzed in real-time. The default metric displayed on the graph is server latency.

Level 1

The Level 1 dashboard is designed to help determine if an individual location has a latency issue. It compares latency metrics across various aspects of the location, such as monitoring points, endpoints, protocols, and peers. The default metric displayed on the graph is server latency.

When the host acts as a server, metrics for server latency, peer client latency, and network latency are displayed. Conversely, metrics are shown for client latency, peer server latency, and peer network latency when the host acts as a client. You will see both sets of metrics if the host has acted as a server and a client.

One variable for Resolution is available at the top of the dashboard. To learn more about this variable, see Dashboard variables.

Level 1 dashboards include:

  • Latency for a Location by Endpoints

  • Latency for a Location by Monitoring Points

  • Latency for a Location by Protocols

  • Latency for a Location by Peers

Level 2

Level 2 dashboards help narrow down the source of latency. You select a specific location and examine metrics at a distinct monitoring point, endpoint, or peer associated with that location. The same metrics are available for Level 1 and Level 2 dashboards, which focus only on latency. The default metric displayed on the graph is server latency.

When the host acts as a server, metrics for server latency, peer client latency, and network latency are displayed. Conversely, metrics are shown for client latency, peer server latency, and peer network latency when the host acts as a client. You will see both sets of metrics if the host has acted as a server and a client.

One variable for Resolution is available at the top of the dashboard. To learn more about this variable, see Dashboard variables.

Level 2 dashboards include:

  • Latency for a Location at an Endpoint by Monitoring Points

  • Latency for a Service at a Monitoring Point by Endpoints

  • Latency for a Service at a Monitoring Point by Peers

  • Latency for a Service with a Peer by Monitoring Points

Level 3

The Level 3 dashboard, Troubleshooting Location (Endpoint Summary), is used to troubleshoot latency issues for a location at specific endpoints. If the IP address is known, you can start troubleshooting directly from this dashboard. As a starting point, it provides various metrics such as throughput Tx, throughput Rx, packets Tx, packets Rx, bytes Tx, bytes Rx, maximum flows Tx, maximum flows Rx, and server latency. The default metric displayed on the graph is throughput Tx. Additional metrics are grouped to help troubleshoot different types of observed issues. To learn more, see Metrics for advanced analysis.

Four variables are available at the top of the dashboard: Endpoint, Location, cStor, and Resolution. To learn more about these variables, see Dashboard variables.

PCAP files can be downloaded for a particular endpoint from the cStor Download tag or a specified range using the cPacket Capture panel. To learn more, see Capture packet data.

Level 4

The level 4 dashboard, Troubleshooting Location (Endpoint Conversations), is utilized for advanced and targeted analysis of a specific data point or narrow time range. An on-demand fetch of the analytics is always required using the cPacket Analytics panel. To learn more, see On-demand analytics collection.

Six variables are available at the top of the dashboard: Endpoint, Peer, Location, cStor, Resolution, and Include Excluded Locations. To learn more about these variables, see Dashboard variables.

See Level 3 for metric and packet capture information.

In this section:

A location is a user-defined group of endpoints that identify a physical location within your network topology. Locations can be nested, where one location is entirely contained within another, such as a data center rack, a hospital floor, or a specific building within a campus.  To define Locations, use the Locations analytics label in the cClear UI.

An endpoint is a server, client, or peer as identified by a unique IP address.

A protocol is the combination of TCP or UDP and a port number used to identify a specific application, also known as a network application. For example, DNS uses port 53, and HTTPS uses port 443. To define protocols, use the Protocols analytics label in the cClear UI.

A monitoring point is the location in the network where the monitoring stack captures packets, typically a TAP, SPAN port, or remote SPAN such as ERSPAN, VXLAN, or GRE tunnel. Each monitoring point is uniquely identified by a combination of a cStor and a packet brokering port, such as 'Hoboken FW1 External'. Sources for this identifier include VLANs, cVu ports, or GENEVE VNI. To define monitoring points, use the Monitoring Points analytics label in the cClear UI.

A peer is a collection of endpoints on the other end of a monitoring point from the entity being monitored, such as a client or server.

Server latency is the round-trip time between the client sending a TCP packet and the server acknowledging the packet, as observed by the monitoring point.

Client latency is the round-trip time between the server sending a TCP packet and the client acknowledging the packet as observed by the monitoring point.

A TCP connection error is any failure in establishing, maintaining, or closing a TCP connection.

The total number of connections that have successfully completed a three-way TCP handshake (SYN-SYNACK-ACK).

A host refers to the endpoint, such as a server or client, that serves as the reference point for analysis. It clarifies the direction in which a specific metric is observed. Every connection goes from the host to the peer. Terminologies like Transmit (Tx), Receive (Rx), outbound, and inbound are always interpreted from the host's perspective. For instance, an outbound connection refers to the host sending data, while an inbound connection means the host is receiving data.

A server is an endpoint that receives a SYN packet from a client and responds with a SYN-ACK packet.

Peer client latency is the round trip time between the server sending a TCP packet and the client acknowledging the packet as observed by the monitoring point.

Network latency is the response time as measured between the client’s initial connection request (SYN packet) and the server’s response (SYN-ACK packet), as observed by the monitoring point.

Peer server latency is the round-trip time between the client sending a TCP packet and the peer server acknowledging the packet as observed by the monitoring point.

Peer network latency is the response time as measured between the client’s initial connection request (SYN packet) and the peer server’s response (SYN-ACK packet) as observed by the monitoring point.

A client is an endpoint that initiates the TCP connection by sending a SYN packet.

The total data in bits per second transmitted by the endpoint or application calculated over the resolution of the dashboard.

The total data in bits per second received by the endpoint or application calculated over the resolution of the dashboard.

The total number of packets an endpoint or application transmitted.

The total number of packets an endpoint or application received.

The total number of bytes an endpoint or application transmitted.

The total number of bytes an endpoint or application received.

The maximum number of transmitted flows by a host as observed by a monitoring point.

The maximum number of received flows by a host as observed by a monitoring point.