What it does. Perform structured log source onboarding into SIEM platforms (Splunk, Part of mukul975/Anthropic-Cybersecurity-Skills (817 security skills) (mukul975/Anthropic-Cybersecurity-Skills).
Install
npx skills add mukul975/Anthropic-Cybersecurity-Skills --skill performing-log-source-onboarding-in-siem, or copy the skill folder into ~/.claude/skills/performing-log-source-onboarding-in-siem/.- Raw file:
curl -sL https://raw.githubusercontent.com/mukul975/Anthropic-Cybersecurity-Skills/HEAD/skills/performing-log-source-onboarding-in-siem/SKILL.md
SKILL.md (verbatim)
name: performing-log-source-onboarding-in-siem
description: Perform structured log source onboarding into SIEM platforms (Splunk,
Elastic, Sentinel, QRadar, or similar) by prioritizing sources with a tiered value
framework, configuring collectors, building parsers, normalizing fields to a common
schema (e.g. CIM), and validating data quality. Use when planning or executing
the integration of new log sources — such as Active Directory, firewalls, EDR,
VPN, DNS, or cloud audit logs — into a SIEM for security monitoring and detection.
domain: cybersecurity
subdomain: soc-operations
tags:
- siem
- log-onboarding
- log-management
- data-ingestion
- parsing
- normalization
- soc
version: '1.0'
author: mahipal
license: Apache-2.0
nist_csf:
- DE.CM-01
- DE.AE-02
- RS.MA-01
- DE.AE-06
mitre_attack:
- T1078
- T1685.002
- T1685.005
- T1566
Performing Log Source Onboarding in SIEM
Overview
Log source onboarding is the systematic process of integrating new data sources into a SIEM platform to enable security monitoring and detection. Proper onboarding requires planning data sources, configuring collection agents, building parsers, normalizing fields to a common schema, and validating data quality. According to the UK NCSC, onboarding should prioritize log sources that provide the highest security value relative to their ingestion cost.
When to Use
- When conducting security assessments that involve performing log source onboarding in siem
- When following incident response procedures for related security events
- When performing scheduled security testing or auditing activities
- When validating security controls through hands-on testing
Prerequisites
- SIEM platform deployed (Splunk, Elastic, Sentinel, QRadar, or similar)
- Network access from source systems to SIEM collectors
- Administrative access on source systems for agent installation
- Common Information Model (CIM) or equivalent schema documentation
- Change management approval for production system modifications
Log Source Priority Framework
Tier 1 - Critical (Onboard First)
| Source |
Log Type |
Security Value |
| Active Directory |
Security Event Logs |
Authentication, privilege escalation |
| Firewalls |
Traffic logs |
Network access, C2 detection |
| EDR/AV |
Endpoint alerts |
Malware, process execution |
| VPN/Remote Access |
Connection logs |
Unauthorized access |
| DNS Servers |
Query logs |
C2 beaconing, data exfiltration |
| Email Gateway |
Email security logs |
Phishing, BEC |
Tier 2 - High Priority
| Source |
Log Type |
Security Value |
| Web Proxy |
HTTP/HTTPS logs |
Web-based attacks, data exfiltration |
| Cloud platforms (AWS/Azure/GCP) |
Audit logs |
Cloud security posture |
| Database servers |
Audit/query logs |
Data access, SQL injection |
| DHCP/IPAM |
Address allocation |
Asset tracking |
| File servers |
Access logs |
Data access monitoring |
Tier 3 - Standard
| Source |
Log Type |
Security Value |
| Application servers |
App logs |
Application-level attacks |
| Print servers |
Print logs |
Data loss prevention |
| Badge/physical access |
Access logs |
Physical security correlation |
| Network devices (switches/routers) |
Syslog |
Network anomalies |
Onboarding Process
Step 1: Discovery and Assessment
1. Identify the log source:
- System type and version
- Log format (syslog, CEF, JSON, Windows Events, etc.)
- Log volume estimate (EPS - events per second)
- Network location and firewall requirements
2. Assess security value:
- What threats can this source help detect?
- Which MITRE ATT&CK techniques does it cover?
- Is there an existing SIEM parser?
3. Estimate ingestion cost:
- Daily volume in GB
- License impact (per-GB or per-EPS pricing)
- Storage retention requirements
Syslog-Based Collection (Firewalls, Network Devices)
# rsyslog configuration for receiving syslog
# /etc/rsyslog.d/10-siem-collection.conf
# UDP reception
module(load="imudp")
input(type="imudp" port="514" ruleset="siem_forwarding")
# TCP reception
module(load="imtcp")
input(type="imtcp" port="514" ruleset="siem_forwarding")
# TLS reception
module(load="imtcp" StreamDriver.AuthMode="x509/name"
StreamDriver.Mode="1" StreamDriver.Name="gtls")
input(type="imtcp" port="6514" ruleset="siem_forwarding")
ruleset(name="siem_forwarding") {
# Forward to SIEM
action(type="omfwd" target="siem.company.com" port="9514"
protocol="tcp" queue.type="LinkedList"
queue.filename="siem_fwd" queue.maxdiskspace="1g"
queue.saveonshutdown="on" action.resumeRetryCount="-1")
}
Windows Event Log Collection (Splunk Universal Forwarder)
# inputs.conf on Splunk Universal Forwarder
[WinEventLog://Security]
disabled = 0
index = wineventlog
sourcetype = WinEventLog:Security
evt_resolve_ad_obj = 1
checkpointInterval = 5
[WinEventLog://System]
disabled = 0
index = wineventlog
sourcetype = WinEventLog:System
[WinEventLog://Microsoft-Windows-Sysmon/Operational]
disabled = 0
index = wineventlog
sourcetype = XmlWinEventLog:Microsoft-Windows-Sysmon/Operational
renderXml = true
[WinEventLog://Microsoft-Windows-PowerShell/Operational]
disabled = 0
index = wineventlog
sourcetype = XmlWinEventLog:Microsoft-Windows-PowerShell/Operational
Cloud Log Collection (AWS CloudTrail)
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"CloudTrailToSIEM": {
"Type": "AWS::CloudTrail::Trail",
"Properties": {
"TrailName": "siem-cloudtrail",
"S3BucketName": "company-cloudtrail-logs",
"IsLogging": true,
"IsMultiRegionTrail": true,
"IncludeGlobalServiceEvents": true,
"EnableLogFileValidation": true,
"EventSelectors": [
{
"ReadWriteType": "All",
"IncludeManagementEvents": true,
"DataResources": [
{
"Type": "AWS::S3::Object",
"Values": ["arn:aws:s3"]
}
]
}
]
}
}
}
}
Step 3: Parse and Normalize
Custom Parser Example (Splunk props.conf/transforms.conf)
# props.conf
[custom:firewall:logs]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%dT%H:%M:%S%z
MAX_TIMESTAMP_LOOKAHEAD = 30
TRANSFORMS-firewall = firewall_extract_fields
FIELDALIAS-src = src_addr AS src_ip
FIELDALIAS-dst = dst_addr AS dest_ip
EVAL-action = case(fw_action=="allow", "allowed", fw_action=="deny", "blocked", true(), "unknown")
EVAL-vendor_product = "Custom Firewall"
LOOKUP-geo = geo_ip_lookup ip AS dest_ip OUTPUT country, city, latitude, longitude
# transforms.conf
[firewall_extract_fields]
REGEX = ^(\S+)\s+(\S+)\s+action=(\w+)\s+src=(\S+):(\d+)\s+dst=(\S+):(\d+)\s+proto=(\w+)\s+bytes=(\d+)
FORMAT = timestamp::$1 hostname::$2 fw_action::$3 src_addr::$4 src_port::$5 dst_addr::$6 dst_port::$7 protocol::$8 bytes::$9
CIM Field Mapping
| Raw Field |
CIM Field |
Data Model |
| src_addr |
src_ip |
Network_Traffic |
| dst_addr |
dest_ip |
Network_Traffic |
| dst_port |
dest_port |
Network_Traffic |
| fw_action |
action |
Network_Traffic |
| bytes_sent + bytes_recv |
bytes |
Network_Traffic |
| user_name |
user |
Authentication |
| login_result |
action |
Authentication |
| process_path |
process |
Endpoint |
Step 4: Validate Data Quality
# Verify events are arriving
index=new_source earliest=-1h
| stats count by sourcetype, host, source
# Check field extraction quality
index=new_source earliest=-1h
| stats count(src_ip) as has_src count(dest_ip) as has_dest count(action) as has_action count by sourcetype
| eval src_coverage=round(has_src/count*100,1)
| eval dest_coverage=round(has_dest/count*100,1)
| eval action_coverage=round(has_action/count*100,1)
# Verify CIM compliance
| datamodel Network_Traffic search
| search sourcetype=new_sourcetype
| stats count by source, sourcetype
# Check for timestamp parsing issues
index=new_source earliest=-1h
| eval time_diff=abs(_time - _indextime)
| stats avg(time_diff) as avg_lag max(time_diff) as max_lag by host
| where avg_lag > 300
Step 5: Enable Detection Coverage
# Verify existing correlation searches work with new source
index=new_source sourcetype=new_sourcetype
| tstats count from datamodel=Authentication by _time span=1h
| timechart span=1h count
# Create source-specific detection rule
[New Source - Authentication Anomaly]
search = index=new_source sourcetype=new_sourcetype action=failure \
| stats count by src_ip, user \
| where count > 10
Onboarding Checklist
References
Other files in this skill
assets/template.md (verbatim)
Log Source Onboarding Request Template
| Field |
Value |
| Source Name |
|
| System Type |
|
| Log Format |
syslog / CEF / JSON / XML / CSV / Windows Events |
| Estimated EPS |
|
| Estimated Daily Volume (GB) |
|
| Security Tier |
Critical / High / Medium / Low |
| Collection Method |
Syslog / Agent / API / File / Cloud |
| Network Location |
|
| Owner/Contact |
|
Onboarding Checklist
CIM Field Mapping
| Raw Field |
CIM Field |
Data Model |
|
|
|
Validation Results
| Check |
Result |
Notes |
| Events arriving |
Pass/Fail |
|
| Field extraction |
Pass/Fail |
|
| CIM compliance |
Pass/Fail |
|
| Timestamp parsing |
Pass/Fail |
|
| Volume within budget |
Pass/Fail |
|
references/api-reference.md (verbatim)
API Reference — Performing Log Source Onboarding in SIEM
Libraries Used
- socket: Test syslog connectivity (UDP/TCP) to SIEM collectors
- re: Log format detection via pattern matching
- pathlib: Read log sample files
CLI Interface
python agent.py detect --file sample.log
python agent.py validate --host siem.corp.com [--port 514] [--protocol udp|tcp]
python agent.py parse-config --format syslog_rfc3164 --source-type firewall_logs
python agent.py checklist --source "Palo Alto FW" --format syslog_rfc3164 --siem-host siem.corp.com
Core Functions
Identifies: syslog RFC 3164/5424, CEF, LEEF, JSON, CSV, Windows Event, Apache combined.
validate_syslog_connectivity(host, port, protocol) — Test SIEM collector
Sends test syslog message via UDP or TCP. Validates port reachability.
Generates Splunk (props.conf/transforms.conf) and Elastic (Filebeat/Logstash) configs.
create_onboarding_checklist(...) — 10-step onboarding workflow
Covers: sample collection, format validation, connectivity, parsing, correlation rules, documentation.
| Format |
Pattern Indicator |
| syslog_rfc3164 |
<PRI>Mon DD HH:MM:SS |
| syslog_rfc5424 |
<PRI>VER YYYY-MM-DDT |
| CEF |
CEF:0| |
| LEEF |
LEEF:1.0| |
| JSON |
{...} |
| Apache combined |
IP - - [timestamp] "METHOD" |
Dependencies
No external packages — Python standard library only.
references/standards.md (verbatim)
Standards - Log Source Onboarding in SIEM
| SIEM Platform |
Schema |
Documentation |
| Splunk |
CIM (Common Information Model) |
docs.splunk.com |
| Elastic |
ECS (Elastic Common Schema) |
elastic.co/guide/en/ecs |
| Microsoft Sentinel |
ASIM (Azure Sentinel Information Model) |
learn.microsoft.com |
| Google Chronicle |
UDM (Unified Data Model) |
cloud.google.com/chronicle |
| Industry Standard |
OCSF (Open Cybersecurity Schema Framework) |
ocsf.io |
Log Collection Protocols
| Protocol |
Port |
Use Case |
Security |
| Syslog UDP |
514 |
Network devices, basic forwarding |
None |
| Syslog TCP |
514 |
Reliable delivery |
None |
| Syslog TLS |
6514 |
Encrypted syslog |
TLS 1.2+ |
| HTTP/S |
443/8088 |
REST API, HEC (Splunk) |
TLS |
| Windows WEF |
5985/5986 |
Windows Event Forwarding |
Kerberos/TLS |
| SNMP |
161/162 |
Network device monitoring |
SNMPv3 |
| S3/Blob |
N/A |
Cloud log storage |
IAM/SAS |
NIST SP 800-92 Log Management Guidelines
- Establish log management infrastructure
- Define log retention requirements
- Ensure log data integrity (tamper evidence)
- Configure time synchronization across all sources
- Implement log review and analysis procedures
references/workflows.md (verbatim)
Workflows - Log Source Onboarding in SIEM
Onboarding Workflow
1. Request Received (ticket/email)
|
v
2. Discovery & Assessment (1-2 days)
- Identify log format and volume
- Assess security value vs cost
- Check for existing parser
|
v
3. Planning (1 day)
- Determine collection method
- Plan network access
- Estimate storage impact
|
v
4. Implementation (2-5 days)
- Install/configure collector
- Build/customize parser
- Map to CIM fields
|
v
5. Validation (1-2 days)
- Verify data flow
- Check field extraction
- Confirm CIM compliance
- Test detection rules
|
v
6. Production Release (1 day)
- Enable detection rules
- Update dashboards
- Document in CMDB
- Notify SOC team
Daily Volume (GB) = EPS * Average Event Size (bytes) * 86400 / 1,073,741,824
Example:
EPS = 100
Avg Event Size = 500 bytes
Daily Volume = 100 * 500 * 86400 / 1,073,741,824 = 4.03 GB/day
Monthly Volume = 4.03 * 30 = 120.9 GB/month
Cost-Value Assessment Matrix
| Security Value |
Low Volume (<1GB/day) |
Medium (1-10GB) |
High (>10GB) |
| Critical |
Must have |
Must have |
Evaluate ROI |
| High |
Should have |
Should have |
Evaluate ROI |
| Medium |
Nice to have |
Evaluate ROI |
Defer |
| Low |
Defer |
Defer |
Reject |
Back to mukul975/Anthropic-Cybersecurity-Skills (817 security skills) or Agent skills.