Parse AWS CloudTrail logs → regex, Grok, Wazuh & rsyslog

What an AWS CloudTrail line looks like

The JSON sample below is fed verbatim into the engine to produce every parser on this page.

{"eventVersion":"1.09","eventTime":"2026-07-03T14:22:15Z","eventSource":"s3.amazonaws.com","eventName":"GetObject","awsRegion":"eu-central-1","sourceIPAddress":"203.0.113.45","userIdentity":{"type":"IAMUser","userName":"jdoe"},"requestParameters":{"bucketName":"onber-logs"}}
{"eventVersion":"1.09","eventTime":"2026-07-03T14:22:31Z","eventSource":"signin.amazonaws.com","eventName":"ConsoleLogin","awsRegion":"us-east-1","sourceIPAddress":"198.51.100.77","userIdentity":{"type":"IAMUser","userName":"admin"},"responseElements":{"ConsoleLogin":"Failure"}}

Detected fields

The engine classified this sample as json and consolidated 10 fields across 2 lines. Fields marked literal were identical on every sample line, so they are baked into the pattern as anchors rather than captured.

• eventversion : number · literal
• eventtime : timestamp
• eventsource : hostname
• eventname : quoted_string
• awsregion : quoted_string
• sourceipaddress : ipv4
• useridentity_type : quoted_string · literal
• useridentity_username : username
• requestparameters_bucketname : quoted_string
• responseelements_consolelogin : quoted_string

Regex (named capture groups)

# sample: {"eventVersion":"1.09","eventTime":"2026-07-03T14:22:15Z","eventSource":"s3.amazonaws.com","eventName":"GetObject","awsRegion":"eu-central-1","sourceIPAddress":"203.0.113.45","userIdentity":{"type":"IAMUser","userName":"jdoe"},"requestParameters":{"bucketName":"onber-logs"}}
# groups: eventtime=2026-07-03T14:22:15Z, eventsource=s3.amazonaws.com, eventname=GetObject, awsregion=eu-central-1, sourceipaddress=203.0.113.45, useridentity_username=jdoe, requestparameters_bucketname=onber-logs
^(?=.*?"eventVersion":"1\.09")(?=.*?"eventTime":"(?<eventtime>[^"]*)")(?=.*?"eventSource":"(?<eventsource>[^"]*)")(?=.*?"eventName":"(?<eventname>[^"]*)")(?=.*?"awsRegion":"(?<awsregion>[^"]*)")(?=.*?"sourceIPAddress":"(?<sourceipaddress>[^"]*)")(?=.*?"type":"IAMUser")(?=.*?"userName":"(?<useridentity_username>[^"]*)")(?=.*?"bucketName":"(?<requestparameters_bucketname>[^"]*)"|)(?=.*?"ConsoleLogin":"(?<responseelements_consolelogin>[^"]*)"|).*$

• note input is JSON — use a JSON parser (jq, Logstash json filter, …) instead of a regex where possible
• note a single linear template could not reproduce every input line — fields are captured with order-independent lookaheads instead

Grok pattern (Logstash / Elastic)

# custom patterns
AWS_CLOUDTRAIL_NOTDQUOTE [^"]*

\{"eventVersion":"1\.09","eventTime":"%{TIMESTAMP_ISO8601:eventtime}","eventSource":"%{HOSTNAME:eventsource}","eventName":"%{AWS_CLOUDTRAIL_NOTDQUOTE:eventname}","awsRegion":"%{AWS_CLOUDTRAIL_NOTDQUOTE:awsregion}","sourceIPAddress":"%{IPV4:sourceipaddress}","userIdentity":\{"type":"IAMUser","userName":"%{USERNAME:useridentity_username}(?:"\},"requestParameters":\{"bucketName":"%{AWS_CLOUDTRAIL_NOTDQUOTE:requestparameters_bucketname})?(?:"\},"responseElements":\{"ConsoleLogin":"%{AWS_CLOUDTRAIL_NOTDQUOTE:responseelements_consolelogin})?

• note json input — consider the Logstash json codec/filter instead of grok
• note constant field "eventversion" embedded as literal anchor "1.09" (varying=false)
• note constant field "useridentity_type" embedded as literal anchor "IAMUser" (varying=false)
• note 2 optional field(s) wrapped in (?:…)? inline regex — grok has no native optional syntax
• note custom patterns emitted — save the '# custom patterns' block to a file in your patterns_dir

Wazuh decoder (OS_Regex XML)

<!--
  Generated by LogForge - Wazuh decoder (OS_Regex dialect, not PCRE)
  sample: {"eventVersion":"1.09","eventTime":"2026-07-03T14:22:15Z","eventSource":"s3.amazonaws.com","eventName":"GetObject","awsRegion":"eu-central-1","sourceIPAddress":
  test with: /var/ossec/bin/wazuh-logtest
-->

<decoder name="aws-cloudtrail-json">
  <prematch>^{</prematch>
  <plugin_decoder>JSON_Decoder</plugin_decoder>
</decoder>

<!-- ============================================================
     ALERT RULE (starter) — put this in a RULES file, e.g.
     /var/ossec/etc/rules/local_rules.xml. Decoders and rules live
     in SEPARATE files. The rule matches the decoder above through
     <decoded_as>; set <level> and add <field>/<match> conditions so
     it alerts only on the events you care about. Rule ids 100000+
     are the user range — change them if they collide with yours.
     ============================================================ -->
<group name="aws-cloudtrail,">
  <rule id="100000" level="3">
    <decoded_as>aws-cloudtrail-json</decoded_as>
    <description>aws-cloudtrail: decoded event</description>
  </rule>
</group>

• note JSON input: emitted a JSON_Decoder plugin decoder — Wazuh extracts every key automatically as dynamic fields (nested keys become dotted names)
• note field names above are what the other LogForge generators use; JSON_Decoder will use the raw JSON keys instead
• note added a starter alert <rule> (level 3, matched to the decoder via <decoded_as>) — put it in a RULES file (not the decoders file), set the level, and add <field>/<match> conditions; the commented example child rule shows the pattern
• note decoder order and prematch specificity may need site-specific tuning (other decoders in your ruleset can shadow these) — validate with /var/ossec/bin/wazuh-logtest

Wazuh's OS_Regex is not PCRE — a bare . is a literal dot and \. matches any character. Test Wazuh OS_Regex patterns →

rsyslog template / liblognorm rulebase

version=2
# aws_cloudtrail — liblognorm v2 rulebase (generated by LogForge)
# Usage with rsyslog (mmnormalize runs liblognorm):
#   module(load="mmnormalize")
#   action(type="mmnormalize" rulebase="/etc/rsyslog.d/aws_cloudtrail.rb" useRawMsg="on")
# Literal "%" is escaped as "%%"; raw tabs are written as \x09.
rule=aws_cloudtrail:{"eventVersion":"1.09","eventTime":"%eventtime:date-rfc5424%","eventSource":"%eventsource:char-to{"extradata":"\""}%","eventName":"%eventname:char-to{"extradata":"\""}%","awsRegion":"%awsregion:char-to{"extradata":"\""}%","sourceIPAddress":"%sourceipaddress:ipv4%","userIdentity":{"type":"IAMUser","userName":"%useridentity_username:char-to{"extradata":"\""}%"},"requestParameters":{"bucketName":"%requestparameters_bucketname:char-to{"extradata":"\""}%"},"responseElements":{"ConsoleLogin":"%responseelements_consolelogin:char-to{"extradata":"\""}%"}}
rule=aws_cloudtrail:{"eventVersion":"1.09","eventTime":"%eventtime:date-rfc5424%","eventSource":"%eventsource:char-to{"extradata":"\""}%","eventName":"%eventname:char-to{"extradata":"\""}%","awsRegion":"%awsregion:char-to{"extradata":"\""}%","sourceIPAddress":"%sourceipaddress:ipv4%","userIdentity":{"type":"IAMUser","userName":"%useridentity_username:char-to{"extradata":"\""}%"

• note json structure: rsyslog mmjsonparse handles CEE/JSON natively — consider action(type="mmjsonparse") instead of this rulebase
• note trailing literal "\"}}" reconstructed from line 1
• note chosen parser types: eventtime=date-rfc5424, eventsource=char-to("), eventname=char-to("), awsregion=char-to("), sourceipaddress=ipv4, useridentity_username=char-to("), requestparameters_bucketname=char-to("), responseelements_consolelogin=char-to(")
• note optional columns (requestparameters_bucketname, responseelements_consolelogin): liblognorm has no optional parts within a single rule — emitted a second rule variant with only the always-present columns (max 2 variants; lines with other column combinations will not match and need extra rule= lines)

Splunk

# props.conf  (search-time extraction)
[<REPLACE_WITH_SOURCETYPE>]
EXTRACT-logforge = (?=.*?"eventVersion":"1\.09")(?=.*?"eventTime":"(?<eventtime>[^"]*)")(?=.*?"eventSource":"(?<eventsource>[^"]*)")(?=.*?"eventName":"(?<eventname>[^"]*)")(?=.*?"awsRegion":"(?<awsregion>[^"]*)")(?=.*?"sourceIPAddress":"(?<sourceipaddress>[^"]*)")(?=.*?"type":"IAMUser")(?=.*?"userName":"(?<useridentity_username>[^"]*)")(?=.*?"bucketName":"(?<requestparameters_bucketname>[^"]*)"|)(?=.*?"ConsoleLogin":"(?<responseelements_consolelogin>[^"]*)"|).*

# Quick search-time test in SPL:
# | rex field=_raw "(?=.*?\"eventVersion\":\"1\\.09\")(?=.*?\"eventTime\":\"(?<eventtime>[^\"]*)\")(?=.*?\"eventSource\":\"(?<eventsource>[^\"]*)\")(?=.*?\"eventName\":\"(?<eventname>[^\"]*)\")(?=.*?\"awsRegion\":\"(?<awsregion>[^\"]*)\")(?=.*?\"sourceIPAddress\":\"(?<sourceipaddress>[^\"]*)\")(?=.*?\"type\":\"IAMUser\")(?=.*?\"userName\":\"(?<useridentity_username>[^\"]*)\")(?=.*?\"bucketName\":\"(?<requestparameters_bucketname>[^\"]*)\"|)(?=.*?\"ConsoleLogin\":\"(?<responseelements_consolelogin>[^\"]*)\"|).*"

• note regex: input is JSON — use a JSON parser (jq, Logstash json filter, …) instead of a regex where possible
• note regex: a single linear template could not reproduce every input line — fields are captured with order-independent lookaheads instead
• note EXTRACT-<class> names must be unique within a sourcetype stanza — rename EXTRACT-logforge if you already use that class for this sourcetype
• note a timestamp field was detected: this EXTRACT only makes it a searchable field. To set the event _time at index time, add TIME_PREFIX and TIME_FORMAT to this props.conf stanza (TIME_FORMAT uses Splunk strptime, e.g. %Y-%m-%dT%H:%M:%S) — this generator does not guess the strptime format.

ES ingest

PUT _ingest/pipeline/aws-cloudtrail
{
  "description": "LogForge-generated ingest pipeline for aws-cloudtrail",
  "processors": [
    {
      "json": {
        "field": "message",
        "add_to_root": true
      }
    }
  ]
}

• note grok: json input — consider the Logstash json codec/filter instead of grok
• note grok: constant field "eventversion" embedded as literal anchor "1.09" (varying=false)
• note grok: constant field "useridentity_type" embedded as literal anchor "IAMUser" (varying=false)
• note grok: 2 optional field(s) wrapped in (?:…)? inline regex — grok has no native optional syntax
• note grok: custom patterns emitted — save the '# custom patterns' block to a file in your patterns_dir
• note json structure: emitted a { json: { field: "message", add_to_root: true } } processor — it parses the JSON line and lifts every field to the top level of the record, so a grok pattern is unnecessary
• note test in Kibana Dev Tools with: POST _ingest/pipeline/aws-cloudtrail/_simulate (supply a docs[] array whose _source.message holds a sample line)

Graylog

# Grok patterns to add under System > Grok Patterns:
# (Graylog needs these custom patterns installed globally BEFORE the rule/extractor below will work.)
#   AWS_CLOUDTRAIL_NOTDQUOTE [^"]*

# --- Graylog processing pipeline rule (primary) ---
# Paste under System > Pipelines > Manage rules, then attach the rule to a pipeline stage.
rule "aws-cloudtrail-parse"
when
  has_field("message")
then
  let gp = grok(pattern: "\\{\"eventVersion\":\"1\\.09\",\"eventTime\":\"%{TIMESTAMP_ISO8601:eventtime}\",\"eventSource\":\"%{HOSTNAME:eventsource}\",\"eventName\":\"%{AWS_CLOUDTRAIL_NOTDQUOTE:eventname}\",\"awsRegion\":\"%{AWS_CLOUDTRAIL_NOTDQUOTE:awsregion}\",\"sourceIPAddress\":\"%{IPV4:sourceipaddress}\",\"userIdentity\":\\{\"type\":\"IAMUser\",\"userName\":\"%{USERNAME:useridentity_username}(?:\"\\},\"requestParameters\":\\{\"bucketName\":\"%{AWS_CLOUDTRAIL_NOTDQUOTE:requestparameters_bucketname})?(?:\"\\},\"responseElements\":\\{\"ConsoleLogin\":\"%{AWS_CLOUDTRAIL_NOTDQUOTE:responseelements_consolelogin})?", value: to_string($message.message), only_named_captures: true);
  set_fields(gp);
end

# --- Graylog import-ready extractor JSON (secondary) ---
# Save as a .json file and import under System > Inputs > (input) > Manage extractors > Actions > Import extractors.
{
  "extractors": [
    {
      "title": "aws-cloudtrail",
      "extractor_type": "grok",
      "converters": [],
      "order": 0,
      "cursor_strategy": "copy",
      "source_field": "message",
      "target_field": "",
      "extractor_config": {
        "grok_pattern": "\\{\"eventVersion\":\"1\\.09\",\"eventTime\":\"%{TIMESTAMP_ISO8601:eventtime}\",\"eventSource\":\"%{HOSTNAME:eventsource}\",\"eventName\":\"%{AWS_CLOUDTRAIL_NOTDQUOTE:eventname}\",\"awsRegion\":\"%{AWS_CLOUDTRAIL_NOTDQUOTE:awsregion}\",\"sourceIPAddress\":\"%{IPV4:sourceipaddress}\",\"userIdentity\":\\{\"type\":\"IAMUser\",\"userName\":\"%{USERNAME:useridentity_username}(?:\"\\},\"requestParameters\":\\{\"bucketName\":\"%{AWS_CLOUDTRAIL_NOTDQUOTE:requestparameters_bucketname})?(?:\"\\},\"responseElements\":\\{\"ConsoleLogin\":\"%{AWS_CLOUDTRAIL_NOTDQUOTE:responseelements_consolelogin})?",
        "named_captures_only": true
      },
      "condition_type": "none",
      "condition_value": ""
    }
  ],
  "version": "5.0.0"
}

• note grok: json input — consider the Logstash json codec/filter instead of grok
• note grok: constant field "eventversion" embedded as literal anchor "1.09" (varying=false)
• note grok: constant field "useridentity_type" embedded as literal anchor "IAMUser" (varying=false)
• note grok: 2 optional field(s) wrapped in (?:…)? inline regex — grok has no native optional syntax
• note grok: custom patterns emitted — save the '# custom patterns' block to a file in your patterns_dir
• note 1 custom grok pattern(s) (AWS_CLOUDTRAIL_NOTDQUOTE) must be installed globally first under System > Grok Patterns — see the block at the top of the output
• note primary artifact is the processing-pipeline rule; the extractor JSON is an equivalent import-ready alternative for the classic extractor UI

Datadog

logforge_rule \{"eventVersion":"1\.09","eventTime":"%{date("yyyy-MM-dd'T'HH:mm:ssZ"):eventtime}","eventSource":"%{notSpace:eventsource}","eventName":"%{quotedString:eventname}","awsRegion":"%{quotedString:awsregion}","sourceIPAddress":"%{ipv4:sourceipaddress}","userIdentity":\{"type":"IAMUser","userName":"%{notSpace:useridentity_username}(?:"\},"requestParameters":\{"bucketName":"%{quotedString:requestparameters_bucketname})?(?:"\},"responseElements":\{"ConsoleLogin":"%{quotedString:responseelements_consolelogin})?

• note emitted rule name is "logforge_rule"; rename it to match your "aws-cloudtrail" convention if desired
• note json input — Datadog can parse JSON logs automatically; a Grok Parser is only needed for non-JSON message bodies
• note constant field "eventversion" embedded as literal anchor "1.09" (varying=false)
• note constant field "useridentity_type" embedded as literal anchor "IAMUser" (varying=false)
• note 2 optional field(s) wrapped in (?:…)? — Datadog Grok has no native optional matcher; a chain of optional columns may need a Helper Rule per shape
• note paste this line into a Grok Parser processor in a Datadog Log Pipeline; matchers are anchored left-to-right and rule whitespace matches log whitespace. Complex or multi-shape logs may need Helper Rules.

Fluent Bit

[PARSER]
    Name        aws-cloudtrail
    Format      regex
    Regex       ^(?=.*?"eventVersion":"1\.09")(?=.*?"eventTime":"(?<eventtime>[^"]*)")(?=.*?"eventSource":"(?<eventsource>[^"]*)")(?=.*?"eventName":"(?<eventname>[^"]*)")(?=.*?"awsRegion":"(?<awsregion>[^"]*)")(?=.*?"sourceIPAddress":"(?<sourceipaddress>[^"]*)")(?=.*?"type":"IAMUser")(?=.*?"userName":"(?<useridentity_username>[^"]*)")(?=.*?"bucketName":"(?<requestparameters_bucketname>[^"]*)"|)(?=.*?"ConsoleLogin":"(?<responseelements_consolelogin>[^"]*)"|).*$
    Time_Key    eventtime
    Time_Format %Y-%m-%dT%H:%M:%SZ
# Fluentd <parse> block:
#   <parse>
#     @type regexp
#     expression /^(?=.*?"eventVersion":"1\.09")(?=.*?"eventTime":"(?<eventtime>[^"]*)")(?=.*?"eventSource":"(?<eventsource>[^"]*)")(?=.*?"eventName":"(?<eventname>[^"]*)")(?=.*?"awsRegion":"(?<awsregion>[^"]*)")(?=.*?"sourceIPAddress":"(?<sourceipaddress>[^"]*)")(?=.*?"type":"IAMUser")(?=.*?"userName":"(?<useridentity_username>[^"]*)")(?=.*?"bucketName":"(?<requestparameters_bucketname>[^"]*)"|)(?=.*?"ConsoleLogin":"(?<responseelements_consolelogin>[^"]*)"|).*$/
#     time_key eventtime
#     time_format %Y-%m-%dT%H:%M:%SZ
#   </parse>

• note regex: input is JSON — use a JSON parser (jq, Logstash json filter, …) instead of a regex where possible
• note regex: a single linear template could not reproduce every input line — fields are captured with order-independent lookaheads instead
• note Time_Key set to "eventtime"; Time_Format "%Y-%m-%dT%H:%M:%SZ" is a best-effort strptime derived from the sample shape — verify it against your data (Fluent Bit uses %L for fractional seconds and %z for numeric offsets)

Vector

[transforms.aws_cloudtrail_parse]
type = "remap"
inputs = ["REPLACE_WITH_SOURCE"]
source = '''
. = parse_json!(.message)
'''

• note the reused regex needs lookahead, which the Rust regex crate behind parse_regex rejects; used parse_json! on .message (also the idiomatic Vector parser for JSON logs)

Loki

# promtail pipeline for "aws-cloudtrail" (generated by LogForge)
# Add these stages under a scrape_config in your promtail config:
#   scrape_configs:
#     - job_name: aws-cloudtrail
#       pipeline_stages:
# (the stages below are indented to sit under pipeline_stages)
pipeline_stages:
  - json:
      expressions:
        eventtime: 'eventTime'
        eventsource: 'eventSource'
        eventname: 'eventName'
        awsregion: 'awsRegion'
        sourceipaddress: 'sourceIPAddress'
        useridentity_username: 'userIdentity.userName'
        requestparameters_bucketname: 'requestParameters.bucketName'
        responseelements_consolelogin: 'responseElements.ConsoleLogin'

• note regex: input is JSON — use a JSON parser (jq, Logstash json filter, …) instead of a regex where possible
• note regex: a single linear template could not reproduce every input line — fields are captured with order-independent lookaheads instead
• note JSON input: the reused regex uses lookaheads (RE2 cannot compile them), so a native `- json:` stage is emitted instead — it extracts each field by key without a regex

syslog-ng

parser p_aws_cloudtrail {
    regexp-parser(
        prefix(".aws_cloudtrail.")
        patterns("(?=.*?\"eventVersion\":\"1\\.09\")(?=.*?\"eventTime\":\"(?<eventtime>[^\"]*)\")(?=.*?\"eventSource\":\"(?<eventsource>[^\"]*)\")(?=.*?\"eventName\":\"(?<eventname>[^\"]*)\")(?=.*?\"awsRegion\":\"(?<awsregion>[^\"]*)\")(?=.*?\"sourceIPAddress\":\"(?<sourceipaddress>[^\"]*)\")(?=.*?\"type\":\"IAMUser\")(?=.*?\"userName\":\"(?<useridentity_username>[^\"]*)\")(?=.*?\"bucketName\":\"(?<requestparameters_bucketname>[^\"]*)\"|)(?=.*?\"ConsoleLogin\":\"(?<responseelements_consolelogin>[^\"]*)\"|).*")
    );
};

• note regex: input is JSON — use a JSON parser (jq, Logstash json filter, …) instead of a regex where possible
• note regex: a single linear template could not reproduce every input line — fields are captured with order-independent lookaheads instead
• note captured fields are stored as name-value pairs under the prefix ".aws_cloudtrail." (e.g. a group (?<srcip>…) becomes ".aws_cloudtrail.srcip")
• note json structure: syslog-ng has a dedicated json-parser() that handles nested objects natively — consider json-parser(prefix(".logforge.")) instead of the emitted regexp-parser

FAQ

How are CloudTrail events delivered and what is the unit to parse?

CloudTrail writes gzipped JSON files to S3, and each file contains a {"Records":[…]} array with one object per API call. To parse, gunzip the object and iterate the Records array — the unit of work is a single Record, a nested JSON document, not a flat log line.

Where is the user identity in a CloudTrail record?

In the nested userIdentity object: userIdentity.type (IAMUser, AssumedRole, Root, AWSService), userIdentity.userName, userIdentity.arn, and userIdentity.accountId. For assumed roles the useful name is often in sessionContext. Extract these by JSON path — they are not top-level fields.

Why is sourceIPAddress sometimes not an IP address?

Because for actions initiated by an AWS service or the console on your behalf, CloudTrail records a service DNS name (like cloudtrail.amazonaws.com) in sourceIPAddress instead of a numeric address. A parser and any detection logic must tolerate a hostname there, not assume it is always an IPv4/IPv6 address.

Which CloudTrail fields matter most for threat detection?

eventName and eventSource (the action and service), userIdentity (who), sourceIPAddress (from where), and errorCode/errorMessage (failures). A burst of AccessDenied errorCodes from one principal signals permission probing; sensitive eventNames like CreateAccessKey, PutUserPolicy, DeleteTrail, or ConsoleLogin without MFA are high-priority.