Architecture

Sentinel Data Lake

Design Sentinel retention and analytics as tiers: keep detection-driving data in Analytics, push verbose or historical data into cheaper table plans or long-term retention, and use ADLS Gen2 plus Azure Data Explorer when you need a decoupled analytical lake.

Architecture Model

Sentinel data lake tiered architecture

flowchart TD
  DS1[High value security data] --> AL[Analytics table plan]
  DS2[Verbose operational data] --> BL[Basic table plan]
  DS3[Low touch audit data] --> AUX[Auxiliary table plan]
  AL --> DET[Sentinel analytics rules]
  AL --> RET[Long term retention]
  BL --> SEARCH[Search jobs]
  AUX --> SEARCH
  RET --> SEARCH
  AL --> EXPORT[Log Analytics data export]
  BL --> EXPORT
  EXPORT --> ADLS[Storage account or Event Hubs]
  ADLS --> ADX[Azure Data Explorer]
  ADX --> EXT[External tables and historical KQL]

Azure Monitor Logs table plansMicrosoft Learn Log Analytics data export rulesMicrosoft Learn

Tier Decision Framework

Table tier decision flow

flowchart TD
  A[New log source] --> B{Feeds detections or active triage}
  B -->|Yes| C[Analytics plan]
  B -->|No| D{Needs low cost interactive table query}
  D -->|Yes| E[Basic plan]
  D -->|No| F{Compliance archive or rare audit}
  F -->|Yes| G[Auxiliary or long term retention]
  F -->|No| H{Needs multi year external analytics}
  H -->|Yes| I[Export to ADLS and query with ADX]
  H -->|No| J[Do not ingest yet]

Tier	Use for	Constraint	Reference
Analytics	Detection, active triage, multi-table hunting, workbooks, and Sentinel analytics rules	Full Sentinel analytics behavior; Analytics retention is configurable and Sentinel defaults differ from generic Log Analytics defaults.	Azure Monitor Logs table plans
Basic	High-volume troubleshooting or operational data that still needs table-scoped queries	Optimized for single-table queries with limitations; Basic and Auxiliary query charges are based on data scanned.	Query data in Basic and Auxiliary tables
Auxiliary	Low-touch verbose audit data and compliance data with infrequent queries	Supported by Sentinel, but with feature limitations and unoptimized query behavior; data export rules do not support Auxiliary tables.	Azure Monitor Logs table plans
Long-term retention	Compliance and occasional historical investigation after interactive retention	Data is accessed through search jobs rather than normal interactive table features.	Manage data retention
ADLS Gen2 + ADX	External long-term analytical lake, historical joins, and large decoupled datasets	Native Sentinel scheduled analytics rules cannot directly use external ADX/ADLS-only data; keep detection-driving data in Analytics.	Query data in Azure Data Lake using Azure Data Explorer

Implementation Patterns

Continuous export to ADLS Gen2

Data export rules continuously export newly arriving records for selected supported tables to Storage or Event Hubs. They are not a historical backfill mechanism; use export jobs for historical subsets.

hcl

resource "azurerm_storage_account" "datalake" {
  name                     = "stsentinellake${var.environment}"
  resource_group_name      = azurerm_resource_group.security.name
  location                 = azurerm_resource_group.security.location
  account_tier             = "Standard"
  account_replication_type = "ZRS"
  account_kind             = "StorageV2"
  is_hns_enabled           = true
}

resource "azurerm_log_analytics_data_export_rule" "sentinel_tables" {
  name                    = "export-sentinel-tables"
  resource_group_name     = azurerm_resource_group.security.name
  workspace_resource_id   = azurerm_log_analytics_workspace.sentinel.id
  destination_resource_id = azurerm_storage_account.datalake.id
  enabled                 = true

  table_names = [
    "SecurityEvent",
    "CommonSecurityLog"
  ]
}

Table plan configuration

hcl

resource "azurerm_log_analytics_workspace_table" "common_security_log" {
  workspace_id = azurerm_log_analytics_workspace.sentinel.id
  name         = "CommonSecurityLog"
  plan         = "Basic"
}

azurerm_log_analytics_data_export_ruleTerraform Registry Microsoft.OperationalInsights workspaces/dataExportsMicrosoft Learn

Query Patterns

Search job for Basic, Auxiliary, or long-term retained data

http

PUT https://management.azure.com/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.OperationalInsights/workspaces/{workspaceName}/tables/CommonSecurityLog_SRCH?api-version=2025-07-01
Authorization: Bearer <arm-token>
Content-Type: application/json

{
  "properties": {
    "searchResults": {
      "query": "CommonSecurityLog | where SourceIP == '198.51.100.101'",
      "limit": 1500,
      "startSearchTime": "2026-01-01T00:00:00.000Z",
      "endSearchTime": "2026-01-08T00:00:00.000Z"
    }
  }
}

kql

CommonSecurityLog_SRCH
| project TimeGenerated, SourceIP, DestinationIP, DeviceAction, Message
| order by TimeGenerated desc

ADX external table over lake storage

kql

.create external table SentinelCommonSecurityLog(
  TimeGenerated: datetime,
  SourceIP: string,
  DestinationIP: string,
  DeviceAction: string,
  Message: string
)
kind=blob
partition by (Date: datetime = bin(TimeGenerated, 1d))
dataformat=multijson
(
  h@'https://stsentinellake.blob.core.windows.net/am-CommonSecurityLog;managed_identity=system'
)

kql

external_table("SentinelCommonSecurityLog")
| where TimeGenerated between (datetime(2026-01-01) .. datetime(2026-02-01))
| where DeviceAction =~ "deny"
| summarize Blocks=count() by SourceIP, DestinationIP
| top 20 by Blocks

Search jobsMicrosoft Learn Query data in Azure Data Lake using Azure Data ExplorerMicrosoft Learn

Cost Model

Use current Azure pricing pages or the Azure pricing calculator for dollar values. The reliable part of the model is the unit breakdown: ingestion, retention, data scanned by searches, storage, transactions, and ADX compute.

text

monthly_ingest_gb = daily_gb * 30
analytics_monthly = monthly_ingest_gb * analytics_ingest_rate
basic_or_aux_monthly = monthly_ingest_gb * table_plan_ingest_rate
retention_gb = daily_gb * retained_days
retention_monthly = retention_gb * retention_rate_per_gb_month
search_monthly = scanned_gb_per_search * searches_per_month * search_scan_rate
adls_monthly = stored_gb * storage_rate_per_gb_month + transaction_costs
adx_monthly = cluster_compute_monthly + ingestion_or_external_query_costs

Microsoft Sentinel pricingAzure pricing Azure Monitor pricingAzure pricing Azure Data Lake Storage pricingAzure pricing Azure Data Explorer pricingAzure pricing

Operational Pitfalls

Pitfall	Mitigation
Detection data moved out of Analytics	Sentinel analytics rules need data in Sentinel-supported query paths. Keep alert-driving tables in Analytics and export a copy if you need lake retention.
Unscoped search jobs	Search charges are scan-based for Basic and Auxiliary tables. Require explicit time windows and column filters for every historical query.
Unsupported data export source	Data export rules support Analytics and Basic table plans, not Auxiliary, and destinations must be in the workspace region.
Small-file ADLS layout	Tiny JSON blobs make external querying expensive and slow. Use partitioning, batching, and Parquet where the downstream ADX pattern supports it.
Schema drift	Custom table changes do not automatically update every DCR, external table, or ADX mapping. Treat schema mapping as versioned code.

Automation Coverage

Sentinel Data Lake Architecture

Method	Support	Reference or Gap
Portal	Supported	Reference
PowerShell	Supported	Reference PowerShell is useful for search jobs and selected operational tasks; use ARM/Bicep/Terraform for durable data export and table plan configuration.
REST API	Supported	Reference
Graph	Gap	Sentinel data lake configuration is Azure Monitor, Storage, Event Hubs, and Azure Data Explorer infrastructure, not a Microsoft Graph Security configuration surface.
ARM	Supported	Reference
Bicep	Supported	Reference
Terraform	Supported	azurermazurerm_log_analytics_data_export_rule Reference Pair data export rules with storage, Event Hubs, table plan, ADX, and RBAC resources; not every downstream lake schema decision is handled by the export rule.
GitHub Actions	Supported	Reference
Azure DevOps	Supported	Reference