Sophos Endpoint Data Collection and Live Discover

The Data Collection and Investigation Policy controls two independent functions: Live Response connections and the upload of endpoint telemetry to the Sophos Data Lake. Live Discover then uses either current data from a connected device or historical data from the Data Lake.

These functions require Sophos EDR, XDR or MDR. Live Response has separate permissions and is also covered in the dedicated response article.

Live Endpoint or Data Lake

Live Endpoint queries are based on osquery and run directly on the target devices. Data Lake queries run against telemetry previously uploaded by devices, can be scheduled and, depending on the licence, access up to 30 or 90 days of data.

A Data Lake time span in Live Discover determines the historical range queried. It is not an execution schedule. A single query can cover no more than 30 days. To cover 90 days, use several bounded queries.

Control Data Lake uploads

Uploads are enabled by default for Endpoint Protection. They can be restricted for groups under My Products > Endpoint > Policies > Data Collection and Investigation.

Turning uploads off reduces available detections and historical investigation. An event collection exclusion should only be used when instructed by Sophos Support because it reduces telemetry and therefore detection or investigation capabilities.

Network traffic can temporarily increase after activation. On small links or sensitive segments, monitor the change in a pilot group first.

Event Journals on the device

Sophos stores Event Journals locally on Windows, macOS and Linux devices. They provide data for Endpoint queries. There is no fixed retention period; it depends on the platform, activity and configured storage.

Under Global Settings > Products and Services > Endpoint and Server > Event Journals, the maximum size can be set between 300 and 30,000 MB. The default is 5,250 MB. An additional percentage of the disk can optionally be set as a limit. If both settings apply, Sophos uses the lower value.

Central configuration of these size limits is available only with a full XDR or MDR licence. Local Event Journals still exist on protected Windows, macOS, and Linux devices, but their thresholds cannot be adjusted tenant-wide without the appropriate licence.

The setting applies to all affected devices and cannot differ by endpoint. Reducing it can discard existing journal data and shorten the investigation period.

Retention and storage limits

XDR stores Data Lake data for up to 90 days, EDR for up to 30 days. An optional storage package extends this to one year. These periods are maximums; if the storage pool is exhausted, older data is removed sooner.

Each device can upload no more than 2 GB per day. When the limit is reached, uploads pause until the reset. Data that was not uploaded is not sent later, although it may still be available for a direct Endpoint query.

The Endpoint pool averages 20 MB per licence per day. Sustained over-usage reduces the history available in practice. A tenant that formally supports 90 days can therefore contain less queryable data.

Find the cause of a daily upload limit

Data Lake uploads originate from recurring Hydration Queries on devices. When an endpoint reaches its daily limit, identify the query generating the largest data volume rather than bypassing the limit. In Threat Analysis Center > Live Discover, create a Data Lake query in Designer Mode against xdr_data and group upload_size by query_name, meta_hostname, and day.

For one device and date, use this basic pattern:

SELECT
  query_name,
  sum(upload_size) / (1000.0 * 1000) AS total_upload_size_mb
FROM xdr_data
WHERE date_trunc('day', ingestion_timestamp) = date '$$date$$'
  AND meta_hostname = '$$hostname$$'
GROUP BY query_name, date_trunc('day', ingestion_timestamp)
ORDER BY total_upload_size_mb DESC

Define date and hostname in Variable Editor as a date in YYYY-MM-DD format and a device name. Then group the largest Hydration Query further using a suitable current-schema field such as process path, user, or Parent Process. This reveals which local activity produces unusual result volume. Do not block a process merely for high telemetry volume; investigate cause, legitimacy, and possible misconfiguration first.

Select and test queries

Prepared Endpoint and Data Lake queries are available under Threat Analysis Center > Live Discover. System Impact shows the observed effect of a query. Test a new or changed query first on one pilot device and over a short period.

Designer Mode allows custom osquery or SQL queries. Sophos Support assists with prepared Sophos queries and the data path, but not with the specialist development of arbitrary custom queries. The Community and Professional Services are available for that purpose.

Query results provide pivots, enrichments and actions. A suspicious SHA-256 can lead to another query, an Intelix or VirusTotal check, a Threat Graph or Live Response.

Guardrails

Sophos limits queries to protect endpoints and the tenant:

• UI: no more than 15 queries per minute,
• scheduled and API queries combined: 1,000 per day and 10 per minute,
• Endpoint Watchdog: termination above 30 per cent CPU for 12 seconds or more than 256 MB RAM,
• no more than 10 MB response data per device and query,
• no more than 1 MB per row,
• no more than 100,000 rows from all responding devices.

An empty result can therefore also be caused by an exceeded row size. Telemetry below the query result shows status, System Impact and transferred data volume for each device.

Narrow down Query cancelled systematically

Query cancelled is not a HAFNIUM-specific error. On Windows, the memory watchdog can stop a Live Query. Use SophosLiveQueryService.log to identify the query name and actual resource error, then reduce the queried period, for example from 15 to 7 days, and retest on one device. If it persists, shorten the period again or narrow the query semantically.

After downgrading Sophos Core Agent 2025.2 to an older LTS or FTS version, SophosOsquery.exe may not start because File Attestation changed. The log then contains Exception whilst running LiveQueryRunner. In this specific case, restart the device; rewriting the query does not resolve the version boundary.

AI Search

AI Search creates Data Lake queries from natural language. Detections and Windows Endpoint data can be searched without writing SQL syntax. The results table shows no more than 1,000 matches.

Start with a few hours or one day here as well. Large time ranges can be slow or fail. Review the generated SQL query before treating the result as complete.

Custom categories and scheduled queries

Organise reusable queries into clearly named custom categories. The query documentation should include a description, supported operating systems, data source, variables and expected output. A copied query is not considered trustworthy merely because its SQL syntax executes successfully.

Scheduled queries are suitable for recurring hypotheses and controls. Set the schedule, target devices, Data Lake scope, result recipients and retention deliberately. Queries that are too broad or run very frequently can reach limits and generate large volumes of personal telemetry.

Configure investigation email notifications separately in the Threat Analysis Center preferences. They do not replace a central alert rule and require a monitored recipient.

Operations and privacy

Data Lake telemetry and Live Discover can contain process names, command lines, users, file hashes, IP addresses and other operational data. Roles, purpose, retention and export should therefore be documented.

Query categories organise frequently used queries. Deleting a custom category does not delete the queries it contains; they remain available under All Queries or other categories.

Related articles

Isolate a Sophos Endpoint and investigate with Live Response explains isolation and Live Response. The subsequent assessment of detections and Threat Graphs is covered in the separate XDR investigation article.

Frequently asked questions