Sophos Endpoint Data Collection and Live Discover

The Data Collection and Investigation Policy controls two independent functions: Live Response connections and the upload of endpoint telemetry to the Sophos Data Lake. Live Discover then uses either current data from a connected device or historical data from the Data Lake.

These functions require Sophos EDR, XDR or MDR. Live Response has separate permissions and is also covered in the dedicated response article.

Quick workflow: Under Threat Analysis Center > Live Discover, first select the appropriate Data source, then choose Query category and Query. For an Endpoint query, deliberately limit the target devices under Devices, then select Run Query. The result is reliable only when the result rows and Telemetry match the selected source, time range and expected devices.

Live Endpoint or Data Lake

Live Endpoint queries are based on osquery and run directly on the target devices. Data Lake queries run against telemetry previously uploaded by devices, can be scheduled and, depending on the licence, access up to 30 or 90 days of data.

A Data Lake time span in Live Discover determines the historical range queried. It is not an execution schedule. A single query can cover no more than 30 days. To cover 90 days, use several bounded queries.

Select and test queries

Prepared Endpoint and Data Lake queries are available under Threat Analysis Center > Live Discover. The workflow deliberately separates the source, query and target:

1. Under Data source, select either Endpoint for the current state of connected devices or Data Lake for historical telemetry that has already been uploaded.
2. Under Query category, select a category, then choose the required query under Query. In the query details, check which operating systems are supported and the System Impact that Sophos assigns to the query.
3. For an Endpoint query, first select a single pilot device under Devices. If necessary, use Show filters, then refresh the list of selected devices. For a Data Lake query, specify a short, known time range instead.
4. Select Run Query and wait for the result.

Test a new or changed query first on a single pilot device or over a short Data Lake time range. Expect result rows that match the selected device or time range. Under Telemetry, also verify that the target responded, the level of System Impact, and the volume of data transferred. If the result remains empty, first check Data source, the time range, the supported operating system and, for Endpoint queries, the device’s connection status. For a Data Lake query, also check whether uploads are enabled under My Products > Endpoint > Policies > Data Collection and Investigation, or under My Products > Server for servers.

Designer Mode allows custom osquery or SQL queries. Sophos Support assists with prepared Sophos queries and the data path, but not with the specialist development of arbitrary custom queries. The Community and Professional Services are available for that purpose.

Query results provide pivots, enrichments and actions. A suspicious SHA-256 can lead to another query, an Intelix or VirusTotal check, a Threat Graph or Live Response.

If you query DNS Protection data, see Analyse Sophos DNS Protection reports and Live Discover for the DNS Protection selection, the corresponding Data Lake schema and DNS-specific validation.

Control Data Lake uploads

Uploads are enabled by default for Endpoint Protection. They can be restricted for groups under My Products > Endpoint > Policies > Data Collection and Investigation.

Turning uploads off reduces available detections and historical investigation. An event collection exclusion should only be used when instructed by Sophos Support because it reduces telemetry and therefore detection or investigation capabilities.

Network traffic can temporarily increase after activation. On small links or sensitive segments, monitor the change in a pilot group first.

Event Journals on the device

Sophos stores Event Journals locally on Windows, macOS and Linux devices. They provide data for Endpoint queries. There is no fixed retention period; it depends on the platform, activity and configured storage.

Under Global Settings > Products and Services > Endpoint and Server > Event Journals, the maximum size can be set between 300 and 30,000 MB. The default is 5,250 MB. An additional percentage of the disk can optionally be set as a limit. If both settings apply, Sophos uses the lower value.

Central configuration of these size limits is available only with an XDR, MDR or MDR Plus licence. Local Event Journals still exist on protected Windows, macOS and Linux devices, but their thresholds cannot be adjusted tenant-wide without the appropriate licence.

The setting applies to all affected devices and cannot differ by endpoint. Reducing it can discard existing journal data and shorten the investigation period.

Retention and storage limits

XDR stores Data Lake data for up to 90 days, EDR for up to 30 days. The optional Central Data Storage – 1 yr pack extends this to one year. These periods are maximums; if the storage pool is exhausted, older data is removed sooner.

Each device can upload no more than 2 GB per day. When the limit is reached, uploads pause until the reset. On Windows, the limit resets at midnight local time; on Linux, it resets every 24 hours after the XDR agent starts. Data that was not uploaded is not sent later, but remains available for a direct Endpoint query.

The Endpoint pool averages 20 MB per licence per day. Sustained over-usage reduces the history available in practice. A tenant that formally supports 90 days can therefore contain less queryable data.

Find the cause of a daily upload limit

The event Data Lake uploads stopped because the daily limit reached is reported on Windows endpoints and Windows or Linux servers. Scheduled Hydration Queries on the individual device produce these uploads. XDR and MDR both have a daily upload limit of 2 GB per device. Once it is reached, no further Hydration Query data is uploaded until the next day; the allowance then resets and the scheduled queries resume until the day ends or the limit is reached again.

To find the Hydration Query with the largest upload volume, sign in to Sophos Fusion (formerly Sophos Central) and go to Threat Analysis Center > Live Discover. Turn on Designer Mode, select Create a new query, enter a name, and select Data Lake as the source. Open Show variable editor, give each variable a descriptive name, and add the variable types Device Name and Date. Supply the affected device name and investigation date in YYYY-MM-DD format, then run this query:

SELECT
  query_name,
  sum(upload_size) / (1000.0 * 1000) AS total_upload_size_bytes_mb
FROM xdr_data
WHERE date_trunc('day', ingestion_timestamp) = date (from_unixtime($$date$$))
  AND meta_hostname = '$$hostname$$'
GROUP BY query_name, date_trunc('day', ingestion_timestamp)
ORDER BY total_upload_size_bytes_mb DESC

Alternatively, this no-variable query ranks every uploading device and query by day. Replace the example start date before running it:

SELECT
  query_name,
  meta_hostname,
  date_trunc('day', ingestion_timestamp) AS date,
  sum(upload_size) / (1000.0 * 1000) AS total_upload_size_bytes_mb
FROM xdr_data
WHERE date_trunc('day', ingestion_timestamp) >= date '2021-09-17'
GROUP BY query_name, meta_hostname, date_trunc('day', ingestion_timestamp)
ORDER BY total_upload_size_bytes_mb DESC

The largest result identifies the Hydration Query to examine, not automatically a process to block. Check that query in the current Sophos Data Lake schema viewer to see which result columns it populates, then group by a useful column. For example, if running_processes_windows_sophos is largest, break its data down by process path:

SELECT
  path,
  sum(upload_size) / (1000.0 * 1000) AS total_upload_size_bytes_mb
FROM xdr_data
WHERE date_trunc('day', ingestion_timestamp) = date '$$date$$'
  AND meta_hostname = '$$hostname$$'
  AND query_name = 'running_processes_windows_sophos'
GROUP BY path, date_trunc('day', ingestion_timestamp)
ORDER BY total_upload_size_bytes_mb DESC

In Sophos’s example, this Hydration Query contributed about 72 MB and one started-process path accounted for more than 50 per cent of its daily upload; the actual process names were hidden because they are environment-specific. Once the dominant action is known, add constraints or inspect other populated columns such as command line, user name, or parent process to identify a change or fix that reduces the result volume. Investigate cause, legitimacy, and possible misconfiguration before changing or blocking anything.

Guardrails

Sophos limits queries to protect endpoints and the tenant:

• UI: no more than 15 queries per minute,
• scheduled and API queries combined: 1,000 per day and 10 per minute,
• Endpoint Watchdog: termination above 30 per cent CPU for 12 seconds or more than 256 MB RAM,
• no more than 10 MB response data per device and query,
• no more than 1 MB per row,
• no more than 100,000 rows from all responding devices.

An empty result can therefore also be caused by an exceeded row size. Telemetry below the query result shows status, System Impact and transferred data volume for each device.

Narrow down Query cancelled systematically

Query cancelled is not a HAFNIUM-specific error. On Windows, the memory watchdog can stop a Live Query. Use SophosLiveQueryService.log to identify the query name and actual resource error, then reduce the queried period, for example from 15 to 7 days, and retest on one device. If it persists, shorten the period again or narrow the query semantically.

After downgrading Sophos Core Agent 2025.2 to an older LTS or FTS version, SophosOsquery.exe may not start because File Attestation changed. The log then contains Exception whilst running LiveQueryRunner. In this specific case, restart the device; rewriting the query does not resolve the version boundary.

AI Search

AI Search creates Data Lake queries from natural language. Detections and Windows Endpoint data can be searched without writing SQL syntax. The results table shows no more than 1,000 matches.

Start with a few hours or one day here as well. Large time ranges can be slow or fail. Review the generated SQL query before treating the result as complete.

Custom categories and scheduled queries

Organise reusable queries into clearly named custom categories. The query documentation should include a description, supported operating systems, data source, variables and expected output. A copied query is not considered trustworthy merely because its SQL syntax executes successfully. Sophos is rolling out custom categories gradually, so the feature might not yet be visible in every tenant.

Scheduled queries are suitable for recurring hypotheses and controls and are available only for Data Lake queries. Set the frequency, day, end date and queried time period deliberately. Each admin can have no more than 100 actively scheduled queries or reports that share this format. Queries that are too broad or run very frequently can also reach the general query limits and generate large volumes of personal telemetry.

Operations and privacy

Data Lake telemetry and Live Discover can contain process names, command lines, users, file hashes, IP addresses and other operational data. Roles, purpose, retention and export should therefore be documented.

Query categories organise frequently used queries. Deleting a custom category does not delete the queries it contains; they remain available under All Queries or other categories.

Related articles

Isolate a Sophos Endpoint and investigate with Live Response explains isolation and Live Response. Sophos Endpoint XDR: Investigation and Threat Graph covers the subsequent assessment of detections and Threat Graphs.

Frequently asked questions