Sophos Endpoint Data Collection and Live Discover

The Data Collection and Investigation Policy controls two independent functions: Live Response connections and the upload of endpoint telemetry to the Sophos Data Lake. Live Discover then uses either current data from a connected device or historical data from the Data Lake.

These functions require Sophos EDR, XDR or MDR. Live Response has separate permissions and is also covered in the dedicated response article.

Quick workflow for Classic: Under Threat Analysis Center > Live Discover, first select the appropriate Data source, then choose Query category and Query. For an Endpoint query, deliberately limit the target devices under Devices, then select Run Query. The result is reliable only when the result rows and Telemetry match the selected source, time range and expected devices.

Before querying, check whether the tenant uses Threat Analysis Center or Security Operations. Threat Analysis Center remains available as Classic view. Security Operations separates Data Lake Search for historical data from Live Endpoint Search for queries on connected devices. The existing Live Discover steps, SQL examples, scheduling details and UI limits below describe Classic unless another interface is explicitly named. A tenant upgrade does not automatically invalidate them or translate them into QL.

Security Operations: Data Lake Search

Under Security Operations > Data Lake Search, you can search the tenant’s Detections and events using three methods. This interface uses Advanced Search query language, or QL; do not copy Classic Data Lake SQL into it unchanged.

• AI Search translates a natural-language question into a query. Enter a question or prepared prompt, select Search, and check the generated query alongside the results. Browse Prompts or View All opens the prompts; Sophos Only, category chips and a search term narrow the selection. Refine the prompt and search again if needed.
• Query Builder builds the query visually from fields and operators. Set the date and time range before Search or Shift + Enter. For an initial check, use a short period containing a known event and verify that the source and filters actually capture it.
• Query Editor lets you write QL queries. Basics shows syntax fundamentals and examples, and Cheat Sheet shows operators and functions. Search Help provides contextual Build With Me assistance for schemas and fields. Here too, set the date and time range before Search or Shift + Enter. SQL examples in the later Classic sections are not QL templates.

The pin button next to the method name sets your preferred search method. Every executed search is automatically added to the tenant’s search history; an executed search is therefore not merely a local draft. Limit prompts and queries to the investigation question and do not enter secrets. An empty results table is meaningful only after checking the time range, source and filters; it does not prove the absence of an incident.

To save in AI Search and Query Editor, select Save Query; in Query Builder, select Save Search first. Enter a name and category, optionally add a description and Bookmark this Query, and finish with Save Query. Reopen and check the saved query under Saved Queries. A bookmark also places it in My Bookmarked Queries.

Full Screen enlarges the results view. The column-header menu offers Pin, Autosize, Reset, Filter and options to show or hide columns; drag column headers to change their order. Use Actions above the table to export all results or a selected subset. Then check export status and download on Data Exports. Do not document a filtered view or subset export as the complete investigation scope.

Keep Live Endpoint Search separate from historical APIs

Live Endpoint Search continues to run osquery-style SQL queries on current, connected endpoints. The Live Discover REST-to-GraphQL migration concerns this live path, not historical Data Lake queries. The fact that the LES catalogue can list saved queries with the source dataLake does not establish Data Lake execution through LES. Do not migrate an existing historical API integration solely on the basis of this migration guide.

Since 12 October 2026, the Search GraphQL API v1 has been released for historical Fusion Data Lake searches using QL. Its workflow comprises search start, status checks and paginated result retrieval; it is separate from LES and its osquery-style live SQL. The Search GraphQL runbook explains requests, Search URNs, partial results, cursors, expiry and comparison with an existing SQL client. Data Lake Search remains the interactive path and the XDR Query API remains available. Neither this release nor the LES migration guide automatically translates Classic SQL into QL or requires an immediate API cutover.

Existing REST clients and an optional LES cutover

Live Discover REST is explicitly excluded from the deprecation of the Classic Cases/Detections APIs. An existing client does not need to be migrated solely because of this announcement. For a new Live Endpoint client or a deliberately planned transition, assess LES separately; the historical XDR Query API and the Classic SQL queries retained above remain separate from this.

LES associates target devices with a session before starting the query: first run createLiveDiscoverSession, save the returned session name, then assign the selected devices to the session using endpointSessions. Only then start runLiveDiscoverQuery with that name as sessionId and the still-required matchEndpoints: { all: false }. Do not use direct filters in matchEndpoints as a shortcut, even if the schema permits them: the migration guide identifies only the session workflow as a supported integration method. Start with a single approved pilot device and a narrowly scoped read-only SQL query; reconcile target assignment, run status, result rows and Telemetry.

Before cutover, check saved query and category IDs for portability; the guide does not confirm that they can be carried over unchanged. Also clarify result retention, subsequent availability of completed runs and a replacement for the previous enrichment pivot management with the API owner. LES accepts only one filter group per selection call; do not guess the meaning of multiple previous REST filter groups. tamperProtectionEnabled becomes tamperProtection as a list, and loosely structured string/JSON filters require their own value validation.

Implement run, catalogue and result pagination separately. Results contain the SQL output columns as a JSON-encoded string in additionalColumns; metadata.columns describes names and types. For subsequent pages, return the supplied pages.nextKey unchanged as params.pageFromKey. HTTP 200 is not sufficient: also check the GraphQL errors array and the actual run/endpoint status. LES shares the limit of 1'000 runs per day and 10 per minute with scheduled queries; the REST API instead specifies 500 runs per day and 10 per minute. These API-specific values do not universally replace the Classic UI limits described further below. Data volume, the device watchdog and result limits must still be considered during the pilot.

Live Endpoint or Data Lake

Live Endpoint queries are based on osquery and run directly on the target devices. Data Lake queries run against telemetry previously uploaded by devices, can be scheduled and, depending on the licence, access up to 30 or 90 days of data.

A Data Lake time span in Live Discover determines the historical range queried. It is not an execution schedule. A single query can cover no more than 30 days. To cover 90 days, use several bounded queries.

Select and test queries

Prepared Endpoint and Data Lake queries are available under Threat Analysis Center > Live Discover. The workflow deliberately separates the source, query and target:

1. Under Data source, select either Endpoint for the current state of connected devices or Data Lake for historical telemetry that has already been uploaded.
2. Under Query category, select a category, then choose the required query under Query. In the query details, check which operating systems are supported and the System Impact that Sophos assigns to the query.
3. For an Endpoint query, first select a single pilot device under Devices. If necessary, use Show filters, then refresh the list of selected devices. For a Data Lake query, specify a short, known time range instead.
4. Select Run Query and wait for the result.

Test a new or changed query first on a single pilot device or over a short Data Lake time range. Expect result rows that match the selected device or time range. Under Telemetry, also verify that the target responded, the level of System Impact, and the volume of data transferred. If the result remains empty, first check Data source, the time range, the supported operating system and, for Endpoint queries, the device’s connection status. For a Data Lake query, also check whether uploads are enabled under My Products > Endpoint > Policies > Data Collection and Investigation, or under My Products > Server for servers.

Designer Mode allows custom osquery or SQL queries. Sophos Support assists with prepared Sophos queries and the data path, but not with the specialist development of arbitrary custom queries. The Community and Professional Services are available for that purpose.

Query results provide pivots, enrichments and actions. A suspicious SHA-256 can lead to another query, an Intelix or VirusTotal check, a Threat Graph or Live Response.

If you query DNS Protection data, see Analyse Sophos DNS Protection reports and Live Discover for the DNS Protection selection, the corresponding Data Lake schema and DNS-specific validation.

Control Data Lake uploads

Uploads are enabled by default for Endpoint Protection. They can be restricted for groups under My Products > Endpoint > Policies > Data Collection and Investigation.

Turning uploads off reduces available detections and historical investigation. An event collection exclusion should only be used when instructed by Sophos Support because it reduces telemetry and therefore detection or investigation capabilities.

Network traffic can temporarily increase after activation. On small links or sensitive segments, monitor the change in a pilot group first.

Event Journals on the device

Sophos stores Event Journals locally on Windows, macOS and Linux devices. They provide data for Endpoint queries. There is no fixed retention period; it depends on the platform, activity and configured storage.

Under Global Settings > Products and Services > Endpoint and Server > Event Journals, the maximum size can be set between 300 and 30,000 MB. The default is 5,250 MB. An additional percentage of the disk can optionally be set as a limit. If both settings apply, Sophos uses the lower value.

Central configuration of these size limits is available only with an XDR, MDR or MDR Plus licence. Local Event Journals still exist on protected Windows, macOS and Linux devices, but their thresholds cannot be adjusted tenant-wide without the appropriate licence.

The setting applies to all affected devices and cannot differ by endpoint. Reducing it can discard existing journal data and shorten the investigation period.

Retention and storage limits

XDR stores Data Lake data for up to 90 days, EDR for up to 30 days. The optional Central Data Storage – 1 yr pack extends this to one year. These periods are maximums; if the storage pool is exhausted, older data is removed sooner.

Each device can upload no more than 2 GB per day. When the limit is reached, uploads pause until the reset. On Windows, the limit resets at midnight local time; on Linux, it resets every 24 hours after the XDR agent starts. Data that was not uploaded is not sent later, but remains available for a direct Endpoint query.

The Endpoint pool averages 20 MB per licence per day. Sustained over-usage reduces the history available in practice. A tenant that formally supports 90 days can therefore contain less queryable data.

Find the cause of a daily upload limit

The event Data Lake uploads stopped because the daily limit reached is reported on Windows endpoints and Windows or Linux servers. Scheduled Hydration Queries on the individual device produce these uploads. XDR and MDR both have a daily upload limit of 2 GB per device. Once it is reached, no further Hydration Query data is uploaded until the next day; the allowance then resets and the scheduled queries resume until the day ends or the limit is reached again.

To find the Hydration Query with the largest upload volume, sign in to Sophos Fusion (formerly Sophos Central) and go to Threat Analysis Center > Live Discover. Turn on Designer Mode, select Create a new query, enter a name, and select Data Lake as the source. Open Show variable editor, give each variable a descriptive name, and add the variable types Device Name and Date. Supply the affected device name and investigation date in YYYY-MM-DD format, then run this query:

SELECT
  query_name,
  sum(upload_size) / (1000.0 * 1000) AS total_upload_size_bytes_mb
FROM xdr_data
WHERE date_trunc('day', ingestion_timestamp) = date (from_unixtime($$date$$))
  AND meta_hostname = '$$hostname$$'
GROUP BY query_name, date_trunc('day', ingestion_timestamp)
ORDER BY total_upload_size_bytes_mb DESC

Alternatively, this no-variable query ranks every uploading device and query by day. Replace the example start date before running it:

SELECT
  query_name,
  meta_hostname,
  date_trunc('day', ingestion_timestamp) AS date,
  sum(upload_size) / (1000.0 * 1000) AS total_upload_size_bytes_mb
FROM xdr_data
WHERE date_trunc('day', ingestion_timestamp) >= date '2021-09-17'
GROUP BY query_name, meta_hostname, date_trunc('day', ingestion_timestamp)
ORDER BY total_upload_size_bytes_mb DESC

The largest result identifies the Hydration Query to examine, not automatically a process to block. Check that query in the current Sophos Data Lake schema viewer to see which result columns it populates, then group by a useful column. For example, if running_processes_windows_sophos is largest, break its data down by process path:

SELECT
  path,
  sum(upload_size) / (1000.0 * 1000) AS total_upload_size_bytes_mb
FROM xdr_data
WHERE date_trunc('day', ingestion_timestamp) = date '$$date$$'
  AND meta_hostname = '$$hostname$$'
  AND query_name = 'running_processes_windows_sophos'
GROUP BY path, date_trunc('day', ingestion_timestamp)
ORDER BY total_upload_size_bytes_mb DESC

In Sophos’s example, this Hydration Query contributed about 72 MB and one started-process path accounted for more than 50 per cent of its daily upload; the actual process names were hidden because they are environment-specific. Once the dominant action is known, add constraints or inspect other populated columns such as command line, user name, or parent process to identify a change or fix that reduces the result volume. Investigate cause, legitimacy, and possible misconfiguration before changing or blocking anything.

Guardrails

Sophos limits queries to protect endpoints and the tenant:

• UI: no more than 15 queries per minute,
• scheduled and API queries combined: 1,000 per day and 10 per minute,
• Endpoint Watchdog: termination above 30 per cent CPU for 12 seconds or more than 256 MB RAM,
• no more than 10 MB response data per device and query,
• no more than 1 MB per row,
• no more than 100,000 rows from all responding devices.

An empty result can therefore also be caused by an exceeded row size. Telemetry below the query result shows status, System Impact and transferred data volume for each device.

Narrow down Query cancelled systematically

Query cancelled is not a HAFNIUM-specific error. On Windows, the memory watchdog can stop a Live Query. Use SophosLiveQueryService.log to identify the query name and actual resource error, then reduce the queried period, for example from 15 to 7 days, and retest on one device. If it persists, shorten the period again or narrow the query semantically.

After downgrading Sophos Core Agent 2025.2 to an older LTS or FTS version, SophosOsquery.exe may not start because File Attestation changed. The log then contains Exception whilst running LiveQueryRunner. In this specific case, restart the device; rewriting the query does not resolve the version boundary.

AI Search

In the Classic view under Threat Analysis Center > AI Search, AI Search creates Data Lake queries from natural language. Detections and Windows endpoint data can be searched without writing SQL syntax. The results table shows no more than 1,000 matches. Do not extrapolate these platform and result limits to the new Data Lake Search described above.

Start with a few hours or one day here as well. Large time ranges can be slow or fail. Review the generated SQL query before treating the result as complete.

Custom categories and scheduled queries

Organise reusable queries into clearly named custom categories. The query documentation should include a description, supported operating systems, data source, variables and expected output. A copied query is not considered trustworthy merely because its SQL syntax executes successfully. Sophos is rolling out custom categories gradually, so the feature might not yet be visible in every tenant.

Scheduled queries are suitable for recurring hypotheses and controls and are available only for Data Lake queries. Set the frequency, day, end date and queried time period deliberately. Each admin can have no more than 100 actively scheduled queries or reports that share this format. Queries that are too broad or run very frequently can also reach the general query limits and generate large volumes of personal telemetry.

Operations and privacy

Data Lake telemetry and Live Discover can contain process names, command lines, users, file hashes, IP addresses and other operational data. Roles, purpose, retention and export should therefore be documented.

Query categories organise frequently used queries. Deleting a custom category does not delete the queries it contains; they remain available under All Queries or other categories.

Related articles

Isolate a Sophos Endpoint and investigate with Live Response explains isolation and Live Response. Sophos Endpoint XDR: Investigation and Threat Graph covers the subsequent assessment of detections and Threat Graphs.

Frequently asked questions

For the current Security Operations interface, Data Lake Search and exports explains schema selection, 31-day event windows, the separate raw-data limit, and current exports. The Classic limits retained above are not transferred to this interface.