Azure Sentinel SIEM Log Ingestion

Page Contents

The Azure Sentinel SIEM built by Microsoft comes with inbuilt ‘data connectors’ to assist you with speeding up the process for log ingestion/onboarding.

As of the date of writing this there are currently 63 data connectors supported ‘out of the box’ in Azure Sentinel, although these logs do not actually get ingested to Azure Sentinel but rather Azure log analytics workspace. For more of an overview check out my other blog page for a Sentinel Overview.

Azure Sentinel Data Connectors View

Below is the current portal and details that are shown for each of the data connectors available ‘out of the box’. Each connector comes with a description, prebuilt workbooks, queries and some with analytic rule templates that can be used to detect threats from the source.

On the ‘open connector page’ there are details as to how to configure each log source but keep in mind that this method may not work for your business and its environment. For example if you have Microsoft products deployed in a separate Azure tenant than the native data connector cannot be used. An alternative method to get these logs would be by using an Azure logic app that hits the API in the different tenant.

Sentinel Data Connectors

Some of these built in data connectors aren’t as straight forwarded as just configuring a few tools to forward it’s events or creating an IAM role in an AWS account. A few of these connectors require additional infrastructure to be deployed either on premise or to the cloud depending on your organisation. This infrastructure can either be the recommended CEF Forwarders which parse and push the logs to the log analytics API over HTTPS (even if deployed in Azure as there is no internal endpoint).

In order to get the logs to the CEF Forwarders it is recommended to syslog the required logs from sources such as, Firewalls, IDS/IPS, on-premise AV solutions, DHCP and a few others. However I would recommend if going down this approach of using the CEF Forwarders than some kind of load balancing should be put it place prior to the Forwarders to ensure the availability of the logs if/when something is bound to go wrong, this will additional help with any accreditation that might need to be met where logs cannot be missing e.g. PCI Compliance.

To the left is a visual representation of the CEF Forwarding using the agent provided by Microsoft. This agent requires a Workspace ID and Primary Key in order to work.

Windows MMA Agent

The Windows MMA agent or otherwise know as Microsoft Monitoring Agent is an agent that sits on the any Windows OS system and pushes the logs to the log analytics workspace over HTTPS to the API, it does this by using the log analytics workspace ID and Primary key, note that the MMA agent can only push to 1 log analytics workspace for log ingestion. A setting in the Security Event data connector can be configured to pull what events matter to you, however there are only 3 settings currently available and cannot be changed with this agent. These settings are as per below

All events – All Windows security and AppLocker events.

Common – A standard set of events for auditing purposes. A full user audit trail is included in this set. For example, it contains both user sign-in and user sign-out events (event IDs 4624, 4634). There are also auditing actions such as security group changes, key domain controller Kerberos operations, and other types of events in line with accepted best practices.The Common event set may contain some types of events that aren’t so common. This is because the main point of the Common set is to reduce the volume of events to a more manageable level, while still maintaining full audit trail capability.

Minimal – A small set of events that might indicate potential threats. This set does not contain a full audit trail. It covers only events that might indicate a successful breach, and other important events that have very low rates of occurrence. For example, it contains successful and failed user logons (event IDs 4624, 4625), but it doesn’t contain sign-out information (4634) which, while important for auditing, is not meaningful for breach detection and has relatively high volume. Most of the data volume of this set is comprised of sign-in events and process creation events (event ID 4688).

None – No security or AppLocker events. (This setting is used to disable the connector.)

Event Setting	Event ID’s
Common	1, 299, 300, 324, 340, 403, 404, 410, 411, 412, 413, 431, 500, 501, 1100, 1102, 1107, 1108, 4608, 4610, 4611, 4614, 4622, 4624, 4625, 4634, 4647, 4648, 4649, 4657, 4661, 4662, 4663, 4665, 4666, 4667, 4688, 4670, 4672, 4673, 4674, 4675, 4689, 4697, 4700, 4702, 4704, 4705, 4716, 4717, 4718, 4719, 4720, 4722, 4723, 4724, 4725, 4726, 4727, 4728, 4729, 4733, 4732, 4735, 4737, 4738, 4739, 4740, 4742, 4744, 4745, 4746, 4750, 4751, 4752, 4754, 4755, 4756, 4757, 4760, 4761, 4762, 4764, 4767, 4768, 4771, 4774, 4778, 4779, 4781, 4793, 4797, 4798, 4799, 4800, 4801, 4802, 4803, 4825, 4826, 4870, 4886, 4887, 4888, 4893, 4898, 4902, 4904, 4905, 4907, 4931, 4932, 4933, 4946, 4948, 4956, 4985, 5024, 5033, 5059, 5136, 5137, 5140, 5145, 5632, 6144, 6145, 6272, 6273, 6278, 6416, 6423, 6424, 8001, 8002, 8003, 8004, 8005, 8006, 8007, 8222, 26401, 30004
Minimal	1102, 4624, 4625, 4657, 4663, 4688, 4700, 4702, 4719, 4720, 4722, 4723, 4724, 4727, 4728, 4732, 4735, 4737, 4739, 4740, 4754, 4755, 4756, 4767, 4799, 4825, 4946, 4948, 4956, 5024, 5033, 8001, 8002, 8003, 8004, 8005, 8006, 8007, 8222

Note that although there is limited configuration for what events you would like to pull with the MMA agent there will be new agent coming called the AMA agent where you can configure exactly what event ID’s should be pulled from the system. Documentation for this new agent can be found here.

If you are managing a segregated environment or would like to reduce the egress from machines hitting the log analytics API directly then you could use a mixture of the AMA agents on the end clients which are pointed to a/multiple Azure log analytics gateways.

Azure Sentinel Security Event data connector settings

Unsupported log types

Alternatively for log types that are not supported natively, other infrastructure will need to be deployed to the environment to support these sources if the source cannot use the log analytics API directly. Options are down to the business and what they are comfortable with but a typical alternative may be the use of logstash to do the filtering and parsing to the API.

Logstash has a supported output plugin to hit the external endpoint with relatively little configuration required if the filtering has already been performed at the source. Note that if you intend to send logs to Azure blob prior to logstash then this method will not work unless a custom input plugin is built, as there is no supported input plugin other than one non official plugin built by Microsoft employees several years ago which no longer supports the latest logstash version. In order to get logs off of a blob storage and into log analytics an alternative solution can be setup using an Azure Logic App or Azure Function, both will work although the logic app is more visual to build but all can be deployed via Terraform or alternative solution if required.

For more detail on Azure Sentinel check out the other blog pages below and the Sentinel GitHub repository which contains a list of data connectors including many currently not supported.

Azure Sentinel Overview

Azure Sentinel Analytic Rules

Sentinel GitHub Data Connectors

Azure Sentinel Threat Hunting