Validate Sophos XDR Data Lake Search and exports

Data Lake Search under Security Operations > Data Lake Search queries events and detections ingested in the tenant. Before the first query, define the investigation question, required source, and time window. An empty result becomes meaningful only after ingestion, schema, and filters have been checked.

This guide describes the current Security Operations interface. Live Discover and Live Endpoint Search distinguish historical data from direct device queries and retain the different Classic procedures.

Choose the source and search method

Before searching, check under Integrations > Configured Data Sources that the expected source supplies current data and maps to the required schema. A green connection alone is insufficient. Integration acceptance explains this distinction.

Data Lake Search offers AI Search, Query Builder, and Query Editor. AI Search generates a suggestion from a natural-language question. Check its schema, conditions, and time range before using it for analysis; AI does not replace source acceptance. Query Builder represents conditions visually, while Query Editor uses Advanced Search Query Language. Use the pin icon to make the preferred method the default.

Start with a small, known source and a short period. In Query Builder, first choose Detections or Events, then the event type for events. In Query Editor, FROM selects the schema. Do not mix detections and events in the same query.

Understand Schema Library and logical fields

A schema is a structured event category, such as auth, process, or netflow. Schema Library shows schemas, searchable fields, and Logical Types. Not every field populated by a parser is necessarily searchable. Check mapping against fields actually available to search, rather than a similarly named vendor field.

Logical fields prefixed with @, such as @ip or @user, combine relevant fields from different schemas. A multi-schema search merges results per schema; a condition affects only schemas containing the field. This is not a relational join. Filtering event fields in a detection search requires the corresponding schema prefix, such as process.commandline.

The following harmless template shows a short search in the authentication schema. example-user is an arbitrary placeholder and must be replaced with an identity approved for the investigation:

FROM auth WHERE @user CONTAINS 'example-user' earliest=-1h

CONTAINS searches substrings; it does not guarantee an exact identity match. If definitive account evidence is required, check the actual identity field, comparison operator, and source. Do not combine identically named local and domain accounts without checking them.

Distinguish time range, retention, and raw-data access

For normalized Events, the schema help dated October 2, 2026 allows a window of at most 31 days per query. The start date may lie anywhere within data actually retained. For longer investigations, use coordinated windows with documented boundaries. Detections can be searched for any period; their available history and permissions still require verification.

These 31 days are not a retention period. Next-Gen SIEM may provide longer retention without allowing an unlimited single query.

For @raw and original_data, the SQL-to-QL guide dated September 15, 2026 describes separate access to the last 20 days. A raw-data request outside this scope is rejected, not silently shortened. The older query-language page also mentions 20-day increments. This does not promise that splitting requests makes older raw data available. Before promising historical raw-data evidence outside the documented scope, a confirmed product-supported path must exist. This is neither general 20-day retention nor evidence of physical data deletion.

Account for QL limits and comparison pitfalls

The query-language help specifies 100 concurrently running Query Units per tenant and a 10-TB scan limit per query. One search may occupy several Query Units; a hundred users do not each receive a hundred parallel searches. Narrow broad logical searches and long periods accordingly. A failed or limited search must not be interpreted as zero events.

Regular expressions for events use a Java-compatible engine; detection searches use a Lucene-based engine. Do not copy a saved expression blindly between these paths. Hostname resolution is also a special case with case-sensitive literal comparison and limited operators.

For arrays, conditions may match different entries in the same event. A match on a header name and header value, for example, does not prove they belong to the same header entry. If this association matters to the finding, inspect the underlying data individually.

Check results and save the search

After selecting the period, run the search with Search or Shift + Enter. Full Screen enlarges the results view. The column-header menu can filter, show/hide, pin, or size fields; columns are rearranged by dragging them.

For reliable findings, check representative rows for the correct tenant, source, event time with time zone, identity, and action. Visible row counts or a positive detection do not replace field validation. For a source test, compare the event with the original evidence.

In AI Search and Query Editor, Save Query opens the save dialog. In Query Builder, choose Save Search first. Then enter name, category, and optional description, and save with Save Query; Bookmark this Query adds it to personal bookmarks. A useful name identifies purpose and source without including secrets or personal raw data. Searches are recorded in the tenant’s search history. Saved query text is not a permanent archive of results.

Save complete exports before they expire

Use the Actions menu above the results table to export the full result set or an intentionally selected subset. Then open Security Operations > Data Exports. Check its status separately from successful search execution:

• Processing: The export is still running.
• Completed: The file is ready to download.
• Errored: Investigate the error and use Retry in the three-dot menu when appropriate for the cause.
• Cancelled: The export was cancelled; it is not complete evidence.

Export files are limited to 100,000 rows per file. For larger volumes, divide time windows, filters, or search parameters appropriately and document multiple requests. The number and coverage of partial files must together match the required investigation scope. A downloaded file does not prove completeness for a larger search.

File Expiration shows when the file expires. Use Download to save it to the approved location before that time. Do not invent a universal fixed TTL from one example. Cancel stops a running export; Delete irreversibly deletes the export. Neither action confirms deletion of the original telemetry from the data lake.

Export evidence includes the query, explicit time boundaries and time zone, selected sources/schemas, export scope, status, row count, and retention in the destination system. Agree access and sharing of sensitive telemetry in advance. For a custom, fully paginated client, the Search GraphQL runbook explains Search-URN, partial failures, cursors, and expiry.

Investigate unexpected results

When matches are missing, check tenant, data source, schema, time window, and conditions first. Do not assume older availability by repeatedly trying new windows after a raw-data error. When scan or concurrency limits are reached, reduce the request and document the split instead of starting uncontrolled retries.

If the search preview is correct but a running detection or suppression rule behaves differently, check current rule operations. Streaming and suppression engines process data at different times and with different constraints from a historical search.