Initial commit - 611 cybersecurity skills across all subdomains

2026-08-26 19:29:41 +03:00 · 2026-02-25 10:47:44 +01:00
commit 22a7ab1462
1765 changed files with 280648 additions and 0 deletions
@@ -0,0 +1,271 @@
+---
+name: performing-log-source-onboarding-in-siem
+description: Perform structured log source onboarding into SIEM platforms by configuring collectors, parsers, normalization, and validation for complete security visibility.
+domain: cybersecurity
+subdomain: soc-operations
+tags: [siem, log-onboarding, log-management, data-ingestion, parsing, normalization, soc]
+version: "1.0"
+author: mahipal
+license: MIT
+---
+
+# Performing Log Source Onboarding in SIEM
+
+## Overview
+
+Log source onboarding is the systematic process of integrating new data sources into a SIEM platform to enable security monitoring and detection. Proper onboarding requires planning data sources, configuring collection agents, building parsers, normalizing fields to a common schema, and validating data quality. According to the UK NCSC, onboarding should prioritize log sources that provide the highest security value relative to their ingestion cost.
+
+## Prerequisites
+
+- SIEM platform deployed (Splunk, Elastic, Sentinel, QRadar, or similar)
+- Network access from source systems to SIEM collectors
+- Administrative access on source systems for agent installation
+- Common Information Model (CIM) or equivalent schema documentation
+- Change management approval for production system modifications
+
+## Log Source Priority Framework
+
+### Tier 1 - Critical (Onboard First)
+
+| Source | Log Type | Security Value |
+|---|---|---|
+| Active Directory | Security Event Logs | Authentication, privilege escalation |
+| Firewalls | Traffic logs | Network access, C2 detection |
+| EDR/AV | Endpoint alerts | Malware, process execution |
+| VPN/Remote Access | Connection logs | Unauthorized access |
+| DNS Servers | Query logs | C2 beaconing, data exfiltration |
+| Email Gateway | Email security logs | Phishing, BEC |
+
+### Tier 2 - High Priority
+
+| Source | Log Type | Security Value |
+|---|---|---|
+| Web Proxy | HTTP/HTTPS logs | Web-based attacks, data exfiltration |
+| Cloud platforms (AWS/Azure/GCP) | Audit logs | Cloud security posture |
+| Database servers | Audit/query logs | Data access, SQL injection |
+| DHCP/IPAM | Address allocation | Asset tracking |
+| File servers | Access logs | Data access monitoring |
+
+### Tier 3 - Standard
+
+| Source | Log Type | Security Value |
+|---|---|---|
+| Application servers | App logs | Application-level attacks |
+| Print servers | Print logs | Data loss prevention |
+| Badge/physical access | Access logs | Physical security correlation |
+| Network devices (switches/routers) | Syslog | Network anomalies |
+
+## Onboarding Process
+
+### Step 1: Discovery and Assessment
+
+```
+1. Identify the log source:
+   - System type and version
+   - Log format (syslog, CEF, JSON, Windows Events, etc.)
+   - Log volume estimate (EPS - events per second)
+   - Network location and firewall requirements
+
+2. Assess security value:
+   - What threats can this source help detect?
+   - Which MITRE ATT&CK techniques does it cover?
+   - Is there an existing SIEM parser?
+
+3. Estimate ingestion cost:
+   - Daily volume in GB
+   - License impact (per-GB or per-EPS pricing)
+   - Storage retention requirements
+```
+
+### Step 2: Configure Log Collection
+
+#### Syslog-Based Collection (Firewalls, Network Devices)
+
+```conf
+# rsyslog configuration for receiving syslog
+# /etc/rsyslog.d/10-siem-collection.conf
+
+# UDP reception
+module(load="imudp")
+input(type="imudp" port="514" ruleset="siem_forwarding")
+
+# TCP reception
+module(load="imtcp")
+input(type="imtcp" port="514" ruleset="siem_forwarding")
+
+# TLS reception
+module(load="imtcp" StreamDriver.AuthMode="x509/name"
+       StreamDriver.Mode="1" StreamDriver.Name="gtls")
+input(type="imtcp" port="6514" ruleset="siem_forwarding")
+
+ruleset(name="siem_forwarding") {
+    # Forward to SIEM
+    action(type="omfwd" target="siem.company.com" port="9514"
+           protocol="tcp" queue.type="LinkedList"
+           queue.filename="siem_fwd" queue.maxdiskspace="1g"
+           queue.saveonshutdown="on" action.resumeRetryCount="-1")
+}
+```
+
+#### Windows Event Log Collection (Splunk Universal Forwarder)
+
+```conf
+# inputs.conf on Splunk Universal Forwarder
+[WinEventLog://Security]
+disabled = 0
+index = wineventlog
+sourcetype = WinEventLog:Security
+evt_resolve_ad_obj = 1
+checkpointInterval = 5
+
+[WinEventLog://System]
+disabled = 0
+index = wineventlog
+sourcetype = WinEventLog:System
+
+[WinEventLog://Microsoft-Windows-Sysmon/Operational]
+disabled = 0
+index = wineventlog
+sourcetype = XmlWinEventLog:Microsoft-Windows-Sysmon/Operational
+renderXml = true
+
+[WinEventLog://Microsoft-Windows-PowerShell/Operational]
+disabled = 0
+index = wineventlog
+sourcetype = XmlWinEventLog:Microsoft-Windows-PowerShell/Operational
+```
+
+#### Cloud Log Collection (AWS CloudTrail)
+
+```json
+{
+  "AWSTemplateFormatVersion": "2010-09-09",
+  "Resources": {
+    "CloudTrailToSIEM": {
+      "Type": "AWS::CloudTrail::Trail",
+      "Properties": {
+        "TrailName": "siem-cloudtrail",
+        "S3BucketName": "company-cloudtrail-logs",
+        "IsLogging": true,
+        "IsMultiRegionTrail": true,
+        "IncludeGlobalServiceEvents": true,
+        "EnableLogFileValidation": true,
+        "EventSelectors": [
+          {
+            "ReadWriteType": "All",
+            "IncludeManagementEvents": true,
+            "DataResources": [
+              {
+                "Type": "AWS::S3::Object",
+                "Values": ["arn:aws:s3"]
+              }
+            ]
+          }
+        ]
+      }
+    }
+  }
+}
+```
+
+### Step 3: Parse and Normalize
+
+#### Custom Parser Example (Splunk props.conf/transforms.conf)
+
+```conf
+# props.conf
+[custom:firewall:logs]
+SHOULD_LINEMERGE = false
+LINE_BREAKER = ([\r\n]+)
+TIME_PREFIX = ^
+TIME_FORMAT = %Y-%m-%dT%H:%M:%S%z
+MAX_TIMESTAMP_LOOKAHEAD = 30
+TRANSFORMS-firewall = firewall_extract_fields
+FIELDALIAS-src = src_addr AS src_ip
+FIELDALIAS-dst = dst_addr AS dest_ip
+EVAL-action = case(fw_action=="allow", "allowed", fw_action=="deny", "blocked", true(), "unknown")
+EVAL-vendor_product = "Custom Firewall"
+LOOKUP-geo = geo_ip_lookup ip AS dest_ip OUTPUT country, city, latitude, longitude
+
+# transforms.conf
+[firewall_extract_fields]
+REGEX = ^(\S+)\s+(\S+)\s+action=(\w+)\s+src=(\S+):(\d+)\s+dst=(\S+):(\d+)\s+proto=(\w+)\s+bytes=(\d+)
+FORMAT = timestamp::$1 hostname::$2 fw_action::$3 src_addr::$4 src_port::$5 dst_addr::$6 dst_port::$7 protocol::$8 bytes::$9
+```
+
+#### CIM Field Mapping
+
+| Raw Field | CIM Field | Data Model |
+|---|---|---|
+| src_addr | src_ip | Network_Traffic |
+| dst_addr | dest_ip | Network_Traffic |
+| dst_port | dest_port | Network_Traffic |
+| fw_action | action | Network_Traffic |
+| bytes_sent + bytes_recv | bytes | Network_Traffic |
+| user_name | user | Authentication |
+| login_result | action | Authentication |
+| process_path | process | Endpoint |
+
+### Step 4: Validate Data Quality
+
+```spl
+# Verify events are arriving
+index=new_source earliest=-1h
+| stats count by sourcetype, host, source
+
+# Check field extraction quality
+index=new_source earliest=-1h
+| stats count(src_ip) as has_src count(dest_ip) as has_dest count(action) as has_action count by sourcetype
+| eval src_coverage=round(has_src/count*100,1)
+| eval dest_coverage=round(has_dest/count*100,1)
+| eval action_coverage=round(has_action/count*100,1)
+
+# Verify CIM compliance
+| datamodel Network_Traffic search
+| search sourcetype=new_sourcetype
+| stats count by source, sourcetype
+
+# Check for timestamp parsing issues
+index=new_source earliest=-1h
+| eval time_diff=abs(_time - _indextime)
+| stats avg(time_diff) as avg_lag max(time_diff) as max_lag by host
+| where avg_lag > 300
+```
+
+### Step 5: Enable Detection Coverage
+
+```spl
+# Verify existing correlation searches work with new source
+index=new_source sourcetype=new_sourcetype
+| tstats count from datamodel=Authentication by _time span=1h
+| timechart span=1h count
+
+# Create source-specific detection rule
+[New Source - Authentication Anomaly]
+search = index=new_source sourcetype=new_sourcetype action=failure \
+| stats count by src_ip, user \
+| where count > 10
+```
+
+## Onboarding Checklist
+
+- [ ] Log source assessed and approved
+- [ ] Network connectivity verified
+- [ ] Collection agent/method configured
+- [ ] Log forwarding confirmed
+- [ ] Parser/field extraction configured
+- [ ] CIM compliance validated
+- [ ] Data model acceleration enabled
+- [ ] Volume within license budget
+- [ ] Retention policy configured
+- [ ] Detection rules enabled/created
+- [ ] Dashboard updated
+- [ ] Documentation completed
+- [ ] SOC team notified
+
+## References
+
+- [UK NCSC - Onboarding Systems and Log Sources](https://www.ncsc.gov.uk/collection/building-a-security-operations-centre/onboarding-systems-and-log-sources)
+- [Sumo Logic - Cloud SIEM Onboarding Checklist](https://help.sumologic.com/docs/cse/get-started-with-cloud-siem/onboarding-checklist-cse/)
+- [SIEM Logging Best Practices - Coralogix](https://coralogix.com/guides/siem/siem-logging/)
+- [Huntress - SIEM Implementation Guide](https://www.huntress.com/siem-guide/siem-implementation-guide)
@@ -0,0 +1,46 @@
+# Log Source Onboarding Request Template
+
+## Source Information
+
+| Field | Value |
+|---|---|
+| Source Name | |
+| System Type | |
+| Log Format | syslog / CEF / JSON / XML / CSV / Windows Events |
+| Estimated EPS | |
+| Estimated Daily Volume (GB) | |
+| Security Tier | Critical / High / Medium / Low |
+| Collection Method | Syslog / Agent / API / File / Cloud |
+| Network Location | |
+| Owner/Contact | |
+
+## Onboarding Checklist
+
+- [ ] Discovery and assessment complete
+- [ ] Planning approved
+- [ ] Network connectivity confirmed
+- [ ] Collection agent/method configured
+- [ ] Parser/field extraction built
+- [ ] CIM fields mapped
+- [ ] Data quality validated
+- [ ] Detection rules enabled
+- [ ] Dashboards updated
+- [ ] Documentation complete
+- [ ] SOC team notified
+- [ ] Production released
+
+## CIM Field Mapping
+
+| Raw Field | CIM Field | Data Model |
+|---|---|---|
+| | | |
+
+## Validation Results
+
+| Check | Result | Notes |
+|---|---|---|
+| Events arriving | Pass/Fail | |
+| Field extraction | Pass/Fail | |
+| CIM compliance | Pass/Fail | |
+| Timestamp parsing | Pass/Fail | |
+| Volume within budget | Pass/Fail | |
@@ -0,0 +1,31 @@
+# Standards - Log Source Onboarding in SIEM
+
+## Common Information Models
+
+| SIEM Platform | Schema | Documentation |
+|---|---|---|
+| Splunk | CIM (Common Information Model) | docs.splunk.com |
+| Elastic | ECS (Elastic Common Schema) | elastic.co/guide/en/ecs |
+| Microsoft Sentinel | ASIM (Azure Sentinel Information Model) | learn.microsoft.com |
+| Google Chronicle | UDM (Unified Data Model) | cloud.google.com/chronicle |
+| Industry Standard | OCSF (Open Cybersecurity Schema Framework) | ocsf.io |
+
+## Log Collection Protocols
+
+| Protocol | Port | Use Case | Security |
+|---|---|---|---|
+| Syslog UDP | 514 | Network devices, basic forwarding | None |
+| Syslog TCP | 514 | Reliable delivery | None |
+| Syslog TLS | 6514 | Encrypted syslog | TLS 1.2+ |
+| HTTP/S | 443/8088 | REST API, HEC (Splunk) | TLS |
+| Windows WEF | 5985/5986 | Windows Event Forwarding | Kerberos/TLS |
+| SNMP | 161/162 | Network device monitoring | SNMPv3 |
+| S3/Blob | N/A | Cloud log storage | IAM/SAS |
+
+## NIST SP 800-92 Log Management Guidelines
+
+- Establish log management infrastructure
+- Define log retention requirements
+- Ensure log data integrity (tamper evidence)
+- Configure time synchronization across all sources
+- Implement log review and analysis procedures
@@ -0,0 +1,60 @@
+# Workflows - Log Source Onboarding in SIEM
+
+## Onboarding Workflow
+
+```
+1. Request Received (ticket/email)
+   |
+   v
+2. Discovery & Assessment (1-2 days)
+   - Identify log format and volume
+   - Assess security value vs cost
+   - Check for existing parser
+   |
+   v
+3. Planning (1 day)
+   - Determine collection method
+   - Plan network access
+   - Estimate storage impact
+   |
+   v
+4. Implementation (2-5 days)
+   - Install/configure collector
+   - Build/customize parser
+   - Map to CIM fields
+   |
+   v
+5. Validation (1-2 days)
+   - Verify data flow
+   - Check field extraction
+   - Confirm CIM compliance
+   - Test detection rules
+   |
+   v
+6. Production Release (1 day)
+   - Enable detection rules
+   - Update dashboards
+   - Document in CMDB
+   - Notify SOC team
+```
+
+## Volume Estimation Formula
+
+```
+Daily Volume (GB) = EPS * Average Event Size (bytes) * 86400 / 1,073,741,824
+
+Example:
+  EPS = 100
+  Avg Event Size = 500 bytes
+  Daily Volume = 100 * 500 * 86400 / 1,073,741,824 = 4.03 GB/day
+  Monthly Volume = 4.03 * 30 = 120.9 GB/month
+```
+
+## Cost-Value Assessment Matrix
+
+| Security Value | Low Volume (<1GB/day) | Medium (1-10GB) | High (>10GB) |
+|---|---|---|---|
+| Critical | Must have | Must have | Evaluate ROI |
+| High | Should have | Should have | Evaluate ROI |
+| Medium | Nice to have | Evaluate ROI | Defer |
+| Low | Defer | Defer | Reject |
@@ -0,0 +1,197 @@
+#!/usr/bin/env python3
+"""
+SIEM Log Source Onboarding Manager
+
+Tracks log source onboarding progress, validates data quality,
+and generates configuration templates for common SIEM platforms.
+"""
+
+import json
+from datetime import datetime
+from typing import Optional
+
+
+class LogSource:
+    """Represents a log source to be onboarded into a SIEM."""
+
+    def __init__(self, name: str, source_type: str, log_format: str,
+                 estimated_eps: int, avg_event_size_bytes: int = 500,
+                 security_tier: str = "medium", collection_method: str = "syslog"):
+        self.name = name
+        self.source_type = source_type
+        self.log_format = log_format
+        self.estimated_eps = estimated_eps
+        self.avg_event_size_bytes = avg_event_size_bytes
+        self.security_tier = security_tier
+        self.collection_method = collection_method
+        self.status = "pending"
+        self.cim_fields_mapped = []
+        self.validation_results = {}
+
+    def estimate_daily_volume_gb(self) -> float:
+        return round(self.estimated_eps * self.avg_event_size_bytes * 86400 / 1_073_741_824, 2)
+
+    def estimate_monthly_volume_gb(self) -> float:
+        return round(self.estimate_daily_volume_gb() * 30, 2)
+
+    def validate_cim_compliance(self, required_fields: list) -> dict:
+        mapped = set(self.cim_fields_mapped)
+        required = set(required_fields)
+        missing = required - mapped
+        coverage = round(len(mapped & required) / max(1, len(required)) * 100, 1)
+        self.validation_results["cim_compliance"] = {
+            "coverage_pct": coverage,
+            "mapped_fields": list(mapped & required),
+            "missing_fields": list(missing),
+            "compliant": coverage >= 80,
+        }
+        return self.validation_results["cim_compliance"]
+
+    def generate_splunk_inputs_conf(self) -> str:
+        templates = {
+            "syslog": f"""[udp://514]
+connection_host = ip
+sourcetype = {self.source_type}
+index = main
+disabled = false""",
+            "windows_event": f"""[WinEventLog://Security]
+disabled = 0
+index = wineventlog
+sourcetype = {self.source_type}
+evt_resolve_ad_obj = 1
+checkpointInterval = 5""",
+            "file_monitor": f"""[monitor:///var/log/{self.name}/*.log]
+disabled = false
+sourcetype = {self.source_type}
+index = main
+crcSalt = <SOURCE>""",
+            "http_event_collector": f"""# Configure via Splunk HEC
+# POST to https://splunk:8088/services/collector/event
+# Headers: Authorization: Splunk <HEC_TOKEN>
+# Body: {{"event": "<log_data>", "sourcetype": "{self.source_type}", "index": "main"}}""",
+        }
+        return templates.get(self.collection_method, "# Unknown collection method")
+
+
+class OnboardingTracker:
+    """Tracks the onboarding status of multiple log sources."""
+
+    ONBOARDING_STEPS = [
+        "discovery",
+        "planning",
+        "collection_configured",
+        "parser_built",
+        "cim_mapped",
+        "validation_passed",
+        "detection_rules_enabled",
+        "documentation_complete",
+        "production_released",
+    ]
+
+    def __init__(self):
+        self.sources = []
+        self.step_completion = {}
+
+    def add_source(self, source: LogSource):
+        self.sources.append(source)
+        self.step_completion[source.name] = {step: False for step in self.ONBOARDING_STEPS}
+
+    def complete_step(self, source_name: str, step: str):
+        if source_name in self.step_completion and step in self.step_completion[source_name]:
+            self.step_completion[source_name][step] = True
+            # Update status
+            for src in self.sources:
+                if src.name == source_name:
+                    completed = sum(1 for v in self.step_completion[source_name].values() if v)
+                    total = len(self.ONBOARDING_STEPS)
+                    if completed == total:
+                        src.status = "completed"
+                    elif completed > 0:
+                        src.status = "in_progress"
+
+    def get_progress_report(self) -> dict:
+        report = {
+            "total_sources": len(self.sources),
+            "completed": sum(1 for s in self.sources if s.status == "completed"),
+            "in_progress": sum(1 for s in self.sources if s.status == "in_progress"),
+            "pending": sum(1 for s in self.sources if s.status == "pending"),
+            "total_daily_volume_gb": sum(s.estimate_daily_volume_gb() for s in self.sources),
+            "total_monthly_volume_gb": sum(s.estimate_monthly_volume_gb() for s in self.sources),
+            "sources": [],
+        }
+        for source in self.sources:
+            steps = self.step_completion.get(source.name, {})
+            completed_steps = sum(1 for v in steps.values() if v)
+            report["sources"].append({
+                "name": source.name,
+                "type": source.source_type,
+                "status": source.status,
+                "progress": f"{completed_steps}/{len(self.ONBOARDING_STEPS)}",
+                "daily_volume_gb": source.estimate_daily_volume_gb(),
+                "security_tier": source.security_tier,
+                "next_step": next((s for s, v in steps.items() if not v), "complete"),
+            })
+        return report
+
+
+CIM_REQUIRED_FIELDS = {
+    "Network_Traffic": ["src_ip", "dest_ip", "dest_port", "action", "bytes", "protocol", "transport"],
+    "Authentication": ["src_ip", "user", "action", "app", "dest"],
+    "Endpoint": ["dest", "process", "process_id", "user", "action"],
+    "Web": ["url", "http_method", "status", "src_ip", "dest_ip", "http_user_agent"],
+    "Email": ["src_user", "recipient", "subject", "action", "file_name"],
+}
+
+
+if __name__ == "__main__":
+    tracker = OnboardingTracker()
+
+    sources = [
+        LogSource("Palo Alto Firewall", "pan:traffic", "syslog-CEF", 500, 600, "critical", "syslog"),
+        LogSource("Windows Domain Controllers", "WinEventLog:Security", "windows-xml", 200, 800, "critical", "windows_event"),
+        LogSource("Squid Web Proxy", "squid:access", "squid-native", 1000, 400, "high", "file_monitor"),
+        LogSource("Custom App Server", "app:custom", "json", 50, 300, "medium", "http_event_collector"),
+    ]
+
+    for src in sources:
+        tracker.add_source(src)
+
+    # Simulate progress
+    tracker.complete_step("Palo Alto Firewall", "discovery")
+    tracker.complete_step("Palo Alto Firewall", "planning")
+    tracker.complete_step("Palo Alto Firewall", "collection_configured")
+    tracker.complete_step("Palo Alto Firewall", "parser_built")
+    tracker.complete_step("Palo Alto Firewall", "cim_mapped")
+
+    tracker.complete_step("Windows Domain Controllers", "discovery")
+    tracker.complete_step("Windows Domain Controllers", "planning")
+    tracker.complete_step("Windows Domain Controllers", "collection_configured")
+
+    tracker.complete_step("Squid Web Proxy", "discovery")
+
+    # CIM validation
+    sources[0].cim_fields_mapped = ["src_ip", "dest_ip", "dest_port", "action", "bytes", "protocol"]
+    cim_result = sources[0].validate_cim_compliance(CIM_REQUIRED_FIELDS["Network_Traffic"])
+
+    print("=" * 70)
+    print("SIEM LOG SOURCE ONBOARDING TRACKER")
+    print("=" * 70)
+
+    report = tracker.get_progress_report()
+    print(f"\nTotal Sources: {report['total_sources']}")
+    print(f"Completed: {report['completed']} | In Progress: {report['in_progress']} | Pending: {report['pending']}")
+    print(f"Total Daily Volume: {report['total_daily_volume_gb']} GB")
+    print(f"Total Monthly Volume: {report['total_monthly_volume_gb']} GB")
+
+    print(f"\n{'Source':<30} {'Status':<15} {'Progress':<10} {'Volume/Day':<12} {'Next Step'}")
+    print("-" * 85)
+    for s in report["sources"]:
+        print(f"{s['name']:<30} {s['status']:<15} {s['progress']:<10} {s['daily_volume_gb']:<12} {s['next_step']}")
+
+    print(f"\nCIM Compliance - Palo Alto Firewall:")
+    print(f"  Coverage: {cim_result['coverage_pct']}%")
+    print(f"  Compliant: {cim_result['compliant']}")
+    print(f"  Missing: {cim_result['missing_fields']}")
+
+    print(f"\nSample inputs.conf for Palo Alto Firewall:")
+    print(sources[0].generate_splunk_inputs_conf())