Architecture

Sentinel Data Lake

Design Sentinel retention and analytics as tiers: keep detection-driving data in Analytics, push verbose or historical data into cheaper table plans or long-term retention, and use ADLS Gen2 plus Azure Data Explorer when you need a decoupled analytical lake.

Architecture Model

Sentinel data lake tiered architecture

flowchart TD
  DS1[High value security data] --> AL[Analytics table plan]
  DS2[Verbose operational data] --> BL[Basic table plan]
  DS3[Low touch audit data] --> AUX[Auxiliary table plan]
  AL --> DET[Sentinel analytics rules]
  AL --> RET[Long term retention]
  BL --> SEARCH[Search jobs]
  AUX --> SEARCH
  RET --> SEARCH
  AL --> EXPORT[Log Analytics data export]
  BL --> EXPORT
  EXPORT --> ADLS[Storage account or Event Hubs]
  ADLS --> ADX[Azure Data Explorer]
  ADX --> EXT[External tables and historical KQL]

Azure Monitor Logs table plansMicrosoft Learn Log Analytics data export rulesMicrosoft Learn

Tier Decision Framework

Table tier decision flow

flowchart TD
  A[New log source] --> B{Feeds detections or active triage}
  B -->|Yes| C[Analytics plan]
  B -->|No| D{Needs low cost interactive table query}
  D -->|Yes| E[Basic plan]
  D -->|No| F{Compliance archive or rare audit}
  F -->|Yes| G[Auxiliary or long term retention]
  F -->|No| H{Needs multi year external analytics}
  H -->|Yes| I[Export to ADLS and query with ADX]
  H -->|No| J[Do not ingest yet]

Tier	Use for	Constraint	Reference
Analytics	Detection, active triage, multi-table hunting, workbooks, and Sentinel analytics rules	Query price is included. Analytics retention is 30 days by default for Azure Monitor Logs and 90 days for Microsoft Sentinel and Application Insights; total retention can extend up to 12 years.	Azure Monitor Logs table plans
Basic	High-volume troubleshooting or operational data that still needs table-scoped queries	Reduced ingestion cost. Queries are charged by data scanned and are limited to one Basic/Auxiliary table, with supported lookup/union patterns into Analytics tables.	Query data in Basic and Auxiliary tables
Auxiliary	Low-touch verbose audit data and compliance data with infrequent queries	Lowest ingestion tier for supported custom logs. Queries are unoptimized and charged by data scanned. Data export rules and restore are not supported.	Azure Monitor Logs table plans
Long-term retention	Compliance and occasional historical investigation after interactive retention	Total retention can extend supported tables up to 12 years. Use search jobs for historical investigation rather than normal interactive Analytics queries.	Manage data retention
ADLS Gen2 + ADX	External long-term analytical lake, historical joins, and large decoupled datasets	Native Sentinel scheduled analytics rules cannot directly use external ADX/ADLS-only data; keep detection-driving data in Analytics.	Query data in Azure Data Lake using Azure Data Explorer

Table Plan Feature Matrix

These are current Azure Monitor Logs table-plan behaviors to validate before moving a Sentinel-connected table out of Analytics. Dollar rates vary by region and agreement, so use Azure pricing for the actual scan and ingestion rates.

Feature	Analytics	Basic	Auxiliary
Query price included	Yes	No	No
Query model	Full KQL	Single Basic/Auxiliary table, with limited lookup/union to Analytics tables	Single Basic/Auxiliary table, with limited lookup/union to Analytics tables
Query performance	Optimized	Optimized for supported Basic query patterns	Slower and not optimized for real-time analysis
Alerts	Supported	Simple log alerts	Not supported
Data export rules	Supported	Supported	Not supported
Search jobs	Supported	Supported	Supported
Restore	Supported	Supported	Not supported
Total retention	Up to 12 years	Up to 12 years	Up to 12 years

Access Control Model

Sentinel data lake access is not a single RBAC switch. Validate the portal role, workspace role, data source, and any row-level scope with a test account before treating access as production-ready.

Control	Use for
Defender unified RBAC	Use for Defender portal roles, Sentinel data lake data-operation permissions, and Sentinel scope assignments where supported.
Microsoft Entra global roles	Can provide broad data lake access across workspaces. Avoid using broad global roles as the normal analyst access pattern.
Azure RBAC	Still governs direct Log Analytics workspace access and Azure resource management outside the Defender portal.
Sentinel row-level scoping	Preview control that uses scope tags and unified RBAC assignments to limit rows returned to scoped users.
Data lake managed identity	Needs workspace permissions such as Log Analytics Contributor when creating new custom tables in the analytics tier.

Defender unified RBACMicrosoft Learn Sentinel roles and permissionsMicrosoft Learn Sentinel row-level scopingMicrosoft Learn Defender XDR Unified RBACSite guide

Implementation Patterns

Continuous export to ADLS Gen2

Data export rules continuously export newly arriving records for selected supported tables to Storage or Event Hubs. They are not a historical backfill mechanism; use export jobs for historical subsets.

hcl

resource "azurerm_storage_account" "datalake" {
  name                     = "stsentinellake${var.environment}"
  resource_group_name      = azurerm_resource_group.security.name
  location                 = azurerm_resource_group.security.location
  account_tier             = "Standard"
  account_replication_type = "ZRS"
  account_kind             = "StorageV2"
  is_hns_enabled           = true
}

resource "azurerm_log_analytics_data_export_rule" "sentinel_tables" {
  name                    = "export-sentinel-tables"
  resource_group_name     = azurerm_resource_group.security.name
  workspace_resource_id   = azurerm_log_analytics_workspace.sentinel.id
  destination_resource_id = azurerm_storage_account.datalake.id
  enabled                 = true

  table_names = [
    "SecurityEvent",
    "CommonSecurityLog"
  ]
}

Table plan configuration

hcl

resource "azurerm_log_analytics_workspace_table" "common_security_log" {
  workspace_id = azurerm_log_analytics_workspace.sentinel.id
  name         = "CommonSecurityLog"
  plan         = "Basic"
}

azurerm_log_analytics_data_export_ruleTerraform Registry Microsoft.OperationalInsights workspaces/dataExportsMicrosoft Learn

Query Patterns

Search job for Basic, Auxiliary, or long-term retained data

http

PUT https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.OperationalInsights/workspaces/{workspaceName}/tables/CommonSecurityLog_SRCH?api-version=2025-07-01
Authorization: Bearer <arm-token>
Content-Type: application/json

{
  "properties": {
    "searchResults": {
      "query": "CommonSecurityLog | where SourceIP == '198.51.100.101'",
      "limit": 1500,
      "startSearchTime": "2026-01-01T00:00:00.000Z",
      "endSearchTime": "2026-01-08T00:00:00.000Z"
    }
  }
}

kql

CommonSecurityLog_SRCH
| project TimeGenerated, SourceIP, DestinationIP, DeviceAction, Message
| order by TimeGenerated desc

ADX external table over lake storage

kql

.create external table SentinelCommonSecurityLog(
  TimeGenerated: datetime,
  SourceIP: string,
  DestinationIP: string,
  DeviceAction: string,
  Message: string
)
kind=blob
partition by (Date: datetime = bin(TimeGenerated, 1d))
dataformat=multijson
(
  h@'https://stsentinellake.blob.core.windows.net/am-CommonSecurityLog;managed_identity=system'
)

kql

external_table("SentinelCommonSecurityLog")
| where TimeGenerated between (datetime(2026-01-01) .. datetime(2026-02-01))
| where DeviceAction =~ "deny"
| summarize Blocks=count() by SourceIP, DestinationIP
| top 20 by Blocks

Search jobsMicrosoft Learn Query data in Azure Data Lake using Azure Data ExplorerMicrosoft Learn

Cost Model

Use current Azure pricing pages or the Azure pricing calculator for dollar values. The reliable part of the model is the unit breakdown: ingestion, retention, data scanned by searches, storage, transactions, and ADX compute.

text

monthly_ingest_gb = daily_gb * 30
analytics_monthly = monthly_ingest_gb * analytics_ingest_rate
basic_or_aux_monthly = monthly_ingest_gb * table_plan_ingest_rate
retention_gb = daily_gb * retained_days
retention_monthly = retention_gb * retention_rate_per_gb_month
search_monthly = scanned_gb_per_search * searches_per_month * search_scan_rate
adls_monthly = stored_gb * storage_rate_per_gb_month + transaction_costs
adx_monthly = cluster_compute_monthly + ingestion_or_external_query_costs

Microsoft Sentinel pricingAzure pricing Azure Monitor pricingAzure pricing Azure Data Lake Storage pricingAzure pricing Azure Data Explorer pricingAzure pricing

Operational Pitfalls

Pitfall	Mitigation
Detection data moved out of Analytics	Sentinel analytics rules need data in Sentinel-supported query paths. Keep alert-driving tables in Analytics and export a copy if you need lake retention.
Unscoped search jobs	Search charges are scan-based for Basic and Auxiliary tables. Require explicit time windows and column filters for every historical query.
Unsupported data export source	Data export rules support Analytics and Basic table plans, not Auxiliary, and destinations must be in the workspace region.
Small-file ADLS layout	Tiny JSON blobs make external querying expensive and slow. Use partitioning, batching, and Parquet where the downstream ADX pattern supports it.
Schema drift	Custom table changes do not automatically update every DCR, external table, or ADX mapping. Treat schema mapping as versioned code.

Automation Coverage

Sentinel Data Lake Architecture

Method	Support	Reference or Gap
Portal	Supported	Reference
PowerShell	Supported	Reference PowerShell is useful for search jobs and selected operational tasks; use ARM/Bicep/Terraform for durable data export and table plan configuration.
REST API	Supported	Reference
Graph	Gap	Sentinel data lake configuration is Azure Monitor, Storage, Event Hubs, and Azure Data Explorer infrastructure, not a Microsoft Graph Security configuration surface.
ARM	Supported	Reference
Bicep	Supported	Reference
Terraform	Supported	azurermazurerm_log_analytics_data_export_rule Reference Pair data export rules with storage, Event Hubs, table plan, ADX, and RBAC resources; not every downstream lake schema decision is handled by the export rule.
GitHub Actions	Supported	Reference
Azure DevOps	Supported	Reference