Splunk

MCP Manager forwards logs directly to Splunk’s HTTP Event Collector (HEC), so a Splunk Cloud Platform or Splunk Enterprise HEC endpoint can be the logs destination with no OpenTelemetry Collector in between. Set Logs format to Splunk HTTP Event Collector (HEC), paste your HEC URL, add the token header, and each MCP log record arrives in Splunk as one event with sourcetype mcpm:mcp. This guide covers the Splunk-specific details. For what MCP Manager sends, how forwarding behaves, who can configure it, and general troubleshooting, see Export to SIEM.

Configuring log forwarding requires the Manage OpenTelemetry collector capability and an Enterprise plan that includes the OpenTelemetry integration. If you do not see the Logging → Integrations panel, see Who can set up log export.

Logs go to HEC; traces go to Splunk Observability Cloud. Splunk keeps logs in Splunk Cloud Platform or Splunk Enterprise, ingested over HEC, while Splunk Observability Cloud accepts only OTLP traces and metrics. The Logs format setting governs logs only — traces are always exported as OTLP, so a traces destination in Splunk Observability Cloud works alongside a HEC logs destination. See Send traces to Splunk Observability Cloud.

What you’ll need

A Splunk Cloud Platform or Splunk Enterprise deployment with HTTP Event Collector enabled.
A HEC token, and the index it writes to (the token’s default index is used, since MCP Manager sets no index on its events).
Access to MCP Manager with the Manage OpenTelemetry collector capability.

Step 1: Determine your HEC URL

MCP Manager posts to the URL you enter exactly as written, so paste the full collector path. Splunk accepts JSON events at either /services/collector or /services/collector/event; both work with MCP Manager.

Deployment	HEC URL form
Splunk Cloud Platform (AWS)	`https://http-inputs-<host>.splunkcloud.com/services/collector` on port `443`
Splunk Cloud Platform (Google Cloud, Azure)	`https://http-inputs.<host>.splunkcloud.com/services/collector` on port `443`
Splunk Cloud Platform free trial	`https://http-inputs-<host>.splunkcloud.com:8088/services/collector`
Splunk Enterprise	`https://<your-splunk-host>:8088/services/collector`

Replace <host> with your Splunk Cloud stack name (the first label of your Splunk Web URL). Splunk Cloud Platform HEC listens on port 443 for paid stacks and on 8088 for free trials; Splunk Enterprise defaults to 8088. MCP Manager does not follow redirects, so a URL that answers with a 3xx fails the export rather than being retried at the new location.

Step 2: Create a HEC token in Splunk

In Splunk Web, go to Settings → Add Data → Monitor → HTTP Event Collector, name the token, choose the index it should write to, and submit. Copy the token value.

Leave indexer acknowledgment off for this token. When acknowledgment is enabled, HEC requires an X-Splunk-Request-Channel header on every request and answers 400 with HEC code 10 (Data channel is missing) without it. MCP Manager does not send a channel identifier or poll the acknowledgment endpoint. If your Splunk administrator requires acknowledgment on the token, add X-Splunk-Request-Channel with a GUID value under Request headers in Step 3; MCP Manager then treats HEC’s code: 0 reply as delivered without waiting for the index acknowledgment.

Step 3: Connect MCP Manager to Splunk HEC

Splunk authenticates HEC requests with an Authorization header whose value is the keyword Splunk followed by a space and the token.

Open the OpenTelemetry collector panel

In MCP Manager, go to Logs → Integrations and find the OpenTelemetry collector panel. Select Edit if a collector is already configured.

Enter the HEC URL as the logs collector URL

In Logs collector URL, paste the HEC URL from Step 1, for example https://http-inputs-acme.splunkcloud.com/services/collector. The Logs format selector stays disabled until this field has a value, because the format applies to the logs destination.

Set the logs format to Splunk HEC

In Logs format, choose Splunk HTTP Event Collector (HEC). The default, OpenTelemetry (OTLP/HTTP), would post OTLP JSON that HEC rejects with 400. A hint under the selector repeats the URL and header requirements once HEC is selected.

Add the Authorization header

Under Request headers, add one header:

Field name: Authorization
Field value: Splunk <your-hec-token>

Paste the token itself after Splunk ; do not prefix it with Bearer. MCP Manager sets Content-Type and Content-Length on every request itself, and ignores a configured header with either of those names (or Host or Transfer-Encoding), logging a warning on the gateway that names the ignored header.

Save the configuration

Select Save. MCP Manager stores the URL, the format, and the encrypted header value, and the panel switches to a read-only view showing Logs format: Splunk HTTP Event Collector (HEC). Saving confirms storage only — it does not confirm delivery.

To configure the same thing without the UI, call set_otel_configuration through the Admin API and MCP server with logsFormat set to splunk_hec:

{
  "logsCollectorUrl": "https://http-inputs-acme.splunkcloud.com/services/collector",
  "logsFormat": "splunk_hec",
  "headers": { "Authorization": "Splunk <your-hec-token>" }
}

Omitting logsFormat keeps the stored format, and a collector saved before the setting existed behaves as otlp. Switching an existing collector between formats on the same URL takes effect for new records within a few seconds; records already queued are flushed to the previous destination first.

Step 4: Verify logs are flowing

Trigger an MCP call through a gateway (a tools/list call is enough), wait a few seconds for the batch to flush, then search Splunk:

Splunk search

index=<your-index> sourcetype="mcpm:mcp"

Each MCP message produces several events sharing one mcpm.correlation_id field — at minimum a proxy_request_success and a proxy_response_success — so a single tools/list call shows up as a small group. If nothing arrives, check Alerts in MCP Manager for a “Failed to export telemetry logs to OTEL collector” alert. For Splunk HEC the alert names the HTTP status and, when Splunk sent one, the HEC reply code and its meaning — for example Splunk HEC code 4 (Invalid token) — so you can tell a bad token from a wrong index without reading Splunk’s own logs. See Troubleshooting for what each code means.

How each log record becomes a Splunk event

MCP Manager maps each OpenTelemetry log record to one HEC event the same way the OpenTelemetry Collector’s splunk_hec exporter does, so events look the same whether they arrive from MCP Manager directly or through a Collector. The log body becomes the event payload and every attribute becomes an indexed field.

HEC field	Value
`time`	The record’s timestamp as epoch seconds with sub-second precision
`host`	The `service.name` resource attribute — `mcp-manager` in production
`source`	`mcp-manager`
`sourcetype`	`mcpm:mcp`
`event`	The log record body — the MCP request/response record as shaped by Export content
`fields`	Every log attribute, flattened with dots, plus the trace and severity fields below

The fields object carries the record’s metadata as indexed fields: user.email, user.full_name, http.response.status_code, mcpm.mcp.method, mcpm.mcp.server_name, mcpm.mcp.server_url, mcpm.mcp.gateway_name, mcpm.mcp.host_name, mcpm.mcp.feature_type, mcpm.mcp.feature_name, mcpm.log.type, and mcpm.correlation_id, along with trace_id, span_id, otel.log.severity.text, otel.log.severity.number, service.name, and service.version. Attributes that are empty on a given record are omitted rather than indexed as null, and nested attribute maps are flattened into dotted keys. The event payload follows the workspace’s Export content setting (see Control what MCP Manager exports). With Full data the event carries the request and response headers and bodies; with Metadata only the same record arrives with its headers and body set to null, so field extraction and searches by mcpm.mcp.method or user.email keep working while message contents stay out of Splunk.

How MCP Manager delivers to HEC

MCP Manager batches log records before sending and reports every delivery problem as an alert. The numbers that shape what you see in Splunk:

Batching. Records are queued and flushed as one export every 5 seconds, or as soon as 512 records are waiting, so an event typically appears in Splunk a few seconds after the MCP call.
Request size. Each export is split into HTTP requests of at most 1,000,000 bytes of newline-delimited JSON, the limit Splunk Cloud Platform documents for HEC request content. Requests are sent one after another under a single 30-second deadline for the whole export.
Oversized records. A single record whose event alone exceeds 1,000,000 bytes is never sent. It is reported as its own alert that names the byte limit, and the rest of the batch is delivered. Under Full data a very large tool result can hit this; Metadata only cannot.
One attempt. A failed request is not retried. Splunk’s 503 (HEC code 9, Server is busy) fails that export and raises an alert like any other error, and the records in it are not re-sent.
Confirmed delivery only. MCP Manager counts records as delivered only when Splunk answers HTTP 200 with {"text":"Success","code":0}. Any other reply, including a 2xx whose body is not a HEC reply (for example an HTML page from a proxy in front of Splunk), is reported as a failure. MCP Manager reads at most 64 KiB of a reply; a longer or cut-off reply is reported as incomplete.

When a request fails partway through a multi-request export, the alert counts only the records that were not delivered and says how many earlier records of the same batch reached Splunk, so the loss figure is exact.

Troubleshooting

Splunk answers every HEC request with a JSON reply carrying a numeric code. MCP Manager parses it and puts both the HTTP status and the code’s meaning in the alert, and uses the code in alert deduplication, so two different Splunk rejections behind the same HTTP 400 (for example an invalid token index and a malformed event) raise two alerts instead of one hiding the other. The codes you are most likely to see:

HTTP status	HEC code	Meaning	What to do
`403`	`1`	Token disabled	Enable the token in Splunk Web, or create a new one
`401`	`2`	Token is required	The `Authorization` header is missing — check the header name
`401`	`3`	Invalid authorization	The value is not `Splunk <token>` — check the keyword and spacing
`403`	`4`	Invalid token	The token value is wrong or was deleted
`400`	`6`	Invalid data format	Splunk rejected the event JSON — confirm Logs format is Splunk HEC, not OTLP
`400`	`7`	Incorrect index	The token’s allowed indexes exclude its default index
`503`	`9`	Server is busy	Splunk is throttling; those records are not retried
`400`	`10`	Data channel is missing	Indexer acknowledgment is on for the token — see Step 2
`400`	`15`	Error in handling indexed fields	A field value Splunk could not index; check the `fields` in a rejected event

A 404 means the URL path is wrong, since MCP Manager appends nothing: confirm it ends in /services/collector or /services/collector/event. A 401 or 403 with codes 1 to 4 points at the token or the header, and a 400 with code 6 almost always means the collector was saved with the OTLP format while pointed at a HEC URL.

Export is failing with 400 and HEC code 6 (Invalid data format)

The request reached HEC but the body was not HEC event JSON. Open the collector panel and confirm Logs format reads Splunk HTTP Event Collector (HEC). If it reads OpenTelemetry (OTLP/HTTP), select Edit, change the format, and save; MCP Manager rebuilds the exporter and the next batch is sent as HEC events.

Export is failing with 400 and HEC code 10 (Data channel is missing)

The token has indexer acknowledgment enabled. Either disable it on the token in Splunk Web, or add a request header named X-Splunk-Request-Channel with a GUID value in MCP Manager. With the header present, Splunk answers code: 0 plus an ackId, and MCP Manager treats that as delivered.

Events arrive but the index is wrong

MCP Manager sends no index field, so every event lands in the token’s default index. Change the default index on the token in Splunk Web rather than looking for a setting in MCP Manager.

An alert says a record exceeded the request limit

One MCP log record was larger than 1,000,000 bytes on its own, which is more than a single HEC request may carry on Splunk Cloud Platform, so MCP Manager set it aside and delivered the rest of the batch. The alert names the record’s correlation ID. If large tool results are routine, set Export content to Metadata only for the collector, or keep Full data and accept that records over the limit are reported rather than sent.

Test the HEC endpoint manually

From any machine with outbound HTTPS access, send one event with your token:

terminal

curl -v -X POST "https://http-inputs-<host>.splunkcloud.com/services/collector" \
  -H "Authorization: Splunk <your-hec-token>" \
  -H "Content-Type: application/json" \
  -d '{"event":"mcp manager hec test","sourcetype":"mcpm:mcp"}'

{"text":"Success","code":0} with HTTP 200 means the URL and token are valid. A 401 or 403 with codes 1 to 4 is a token problem, 404 is a wrong path, and 400 with code 10 means indexer acknowledgment is enabled on the token.

Send traces to Splunk Observability Cloud

Splunk Observability Cloud accepts OTLP traces directly, so request traces need no Collector: set Traces collector URL to your realm’s trace endpoint, https://ingest.<realm>.observability.splunkcloud.com/v2/trace/otlp, and add an X-SF-Token request header carrying an Observability Cloud access token. Traces are always exported as OTLP regardless of the Logs format setting, and the same Request headers are sent with both signals, so a workspace that sends logs to HEC and traces to Observability Cloud carries both the Authorization: Splunk … header and X-SF-Token; each endpoint ignores the header it does not use. Because MCP Manager sends OTLP/HTTP JSON and Splunk’s trace intake documents protobuf, verify delivery before relying on it, and fall back to a self-hosted Collector for traces if JSON is rejected. Splunk Observability Cloud has no OTLP logs endpoint — its /v2/trace/otlp and /v2/datapoint/otlp paths take traces and metrics only — which is why logs go to HEC instead.

Find your realm

Every Splunk Observability Cloud host is realm-specific (for example us0, us1, eu0, eu1, ap0). Your realm determines the ingest host https://ingest.<realm>.observability.splunkcloud.com. Find your realm in the Splunk Observability Cloud UI before configuring the traces endpoint.

Self-hosted Collector

Put a Collector in front when you need to filter, enrich, or fan out before Splunk.

Export to SIEM

What MCP Manager sends, the Export content setting, and general troubleshooting.

Admin API: Logging

Configure the collector and the logs format programmatically with set_otel_configuration.

Viewing Logs

The log model and the data each forwarded record carries.

External sources

Set up and use HTTP Event Collector

Creating a HEC token in Splunk Web, the Splunk Cloud Platform URL forms, and the Authorization: Splunk header.

Splunk Cloud Platform service details

The documented 1 MB maximum content length for HEC requests.

OpenTelemetry Collector splunk_hec exporter

The reference mapping from OpenTelemetry log records to HEC events that MCP Manager follows.

Splunk OTLP/HTTP exporter reference

The /v2/trace/otlp and /v2/datapoint/otlp paths with X-SF-Token auth — and no logs endpoint.

Find your realm and API endpoints

How to find your realm, which determines every Observability Cloud ingest host.

Get Started

MCP Gateway Concepts

Features

Industries

Tutorials

MCP Server Guides

Security

Deployment

Enterprise

Admin API & MCP

Build Your Own MCP Server

Advanced

What you’ll need

Step 1: Determine your HEC URL

Step 2: Create a HEC token in Splunk

Step 3: Connect MCP Manager to Splunk HEC

Step 4: Verify logs are flowing

How each log record becomes a Splunk event

How MCP Manager delivers to HEC

Troubleshooting

Send traces to Splunk Observability Cloud

Find your realm

Further reading

Self-hosted Collector

Export to SIEM

Admin API: Logging

Viewing Logs

External sources

Set up and use HTTP Event Collector

Splunk Cloud Platform service details

OpenTelemetry Collector splunk_hec exporter

Splunk OTLP/HTTP exporter reference

Find your realm and API endpoints

​What you’ll need

​Step 1: Determine your HEC URL

​Step 2: Create a HEC token in Splunk

​Step 3: Connect MCP Manager to Splunk HEC

​Step 4: Verify logs are flowing

​How each log record becomes a Splunk event

​How MCP Manager delivers to HEC

​Troubleshooting

​Send traces to Splunk Observability Cloud

​Find your realm

​Further reading

Self-hosted Collector

Export to SIEM

Admin API: Logging

Viewing Logs

​External sources

Set up and use HTTP Event Collector

Splunk Cloud Platform service details

OpenTelemetry Collector splunk_hec exporter

Splunk OTLP/HTTP exporter reference

Find your realm and API endpoints

What you’ll need

Step 1: Determine your HEC URL

Step 2: Create a HEC token in Splunk

Step 3: Connect MCP Manager to Splunk HEC

Step 4: Verify logs are flowing

How each log record becomes a Splunk event

How MCP Manager delivers to HEC

Troubleshooting

Send traces to Splunk Observability Cloud

Find your realm

Further reading

External sources