You wrote a regex that works. You tested it on regex101, it matches, the capture groups are right. You paste it into a Vector parse_regex call or a promtail regex stage, and it either throws a compile error or, worse, matches nothing. The regex isn’t wrong. The engine is different.
Vector (Rust’s regex crate) and Loki’s promtail (Go’s regexp) are both RE2 engines. RE2 guarantees linear-time matching — it can’t catastrophically backtrack — and it buys that guarantee by dropping features that require backtracking: no lookahead, no lookbehind, no backreferences. It also spells named groups (?P<name>…), not (?<name>…). PCRE, Oniguruma (Logstash/Graylog grok), and JavaScript all have those features; RE2 doesn’t, on purpose.
Most of the time this never matters. A positional line — timestamp, then host, then message — parses fine in RE2. The trouble starts with one very common log shape.
Where a log parser actually needs lookahead
Key=value logs (FortiGate, a lot of app logs, anything logfmt-ish) don’t guarantee field order. Line one:
action=deny srcip=10.0.0.5 dstip=8.8.8.8
Line two, same source, fields in a different order and one extra:
srcip=10.0.0.6 proto=6 action=allow dstip=1.1.1.1
A single positional regex can’t capture both, because srcip isn’t always in the same place. The standard trick is order-independent capture with lookahead — one lookahead per field, each scanning the whole line for its key:
(?=.*?\bsrcip=(?<srcip>\S+))(?=.*?\baction=(?<action>\S+))(?=.*?\bdstip=(?<dstip>\S+))
That works beautifully in PCRE and JavaScript. Paste it into Vector’s parse_regex and Rust’s regex crate rejects it outright: look-around is not supported. Promtail does the same. The pattern that was the obvious answer is the one thing the engine can’t run.
The fix: stop forcing a regex
The mistake is trying to make RE2 do a job it’s structurally wrong for. Key=value and JSON logs have purpose-built parsers in both tools — use those instead of a heroic regex:
# Vector — don't regex key=value, parse it
. = parse_key_value!(.message)
# JSON logs
. = parse_json!(.message)
# or grok, which uses an Oniguruma engine and DOES support lookaround
. |= parse_grok!(.message, "%{DATA:action}...")
# promtail — logfmt for key=value, json for JSON
- logfmt:
mapping:
srcip:
action:
- json:
expressions:
srcip: srcip
parse_key_value and the logfmt stage handle reordered fields natively, in linear time, without any lookahead — because they’re actual parsers, not a regex pretending to be one. For genuinely positional lines where you do want a regex, just remember to write (?P<name>…) and keep it lookaround-free.
What a good generator does with this
This is one of the cases where generating the config beats hand-writing it, because the tool can see the mismatch. When LogForge builds a pattern for a key=value or JSON log and the pattern it would emit needs lookahead, it doesn’t hand you a Vector or Loki config that won’t compile. It detects the RE2 incompatibility and falls back — parse_key_value! / parse_json! for Vector, a logfmt/json stage for promtail, or grok — and leaves a note saying why. For positional patterns it converts (?<name>) to (?P<name>) automatically so you don’t ship a pattern that’s silently the wrong dialect. Everything runs in your browser; nothing is uploaded.
FAQ
How do I know if a regex uses lookahead? Look for (?=, (?!, (?<=, (?<!. If any appear, RE2 (Vector parse_regex, promtail regex, Go, Rust) will reject it. Backreferences (\1) and atomic groups ((?>) are out too.
Why does Vector’s parse_grok work when parse_regex doesn’t? They use different engines. parse_grok sits on an Oniguruma-based crate that supports lookaround; parse_regex is the Rust RE2 regex crate that doesn’t.
Is (?<name>) really invalid in Go/Rust? The named-group syntax is (?P<name>) there. (?<name>) is a parse error in RE2. Same feature, different spelling — and a common reason a copied pattern won’t compile.
Can’t I just enable lookahead in RE2? No — it’s a deliberate design choice that gives RE2 its linear-time guarantee. There’s no flag. Use a structured parser instead.