Initial commit - 611 cybersecurity skills across all subdomains

2026-08-13 05:43:19 +03:00 · 2026-02-25 10:47:44 +01:00
commit 22a7ab1462
1765 changed files with 280648 additions and 0 deletions
@@ -0,0 +1,295 @@
+---
+name: implementing-aws-macie-for-data-classification
+description: Implement Amazon Macie to automatically discover, classify, and protect sensitive data in S3 buckets using machine learning and pattern matching for PII, financial data, and credentials detection.
+domain: cybersecurity
+subdomain: cloud-security
+tags: [aws, macie, data-classification, s3, pii, sensitive-data, dlp, compliance]
+version: "1.0"
+author: mahipal
+license: MIT
+---
+
+# Implementing AWS Macie for Data Classification
+
+## Overview
+
+Amazon Macie is a fully managed data security and privacy service that uses machine learning and pattern matching to discover and protect sensitive data in Amazon S3. Macie automatically evaluates your S3 bucket inventory on a daily basis and identifies objects containing PII, financial information, credentials, and other sensitive data types. It provides two discovery approaches: automated sensitive data discovery for broad visibility and targeted discovery jobs for deep analysis.
+
+## Prerequisites
+
+- AWS account with S3 buckets containing data to classify
+- IAM permissions for Macie service configuration
+- AWS Organizations setup (for multi-account deployment)
+- S3 buckets in supported regions
+
+## Enable Macie
+
+### Via AWS CLI
+
+```bash
+# Enable Macie in the current account/region
+aws macie2 enable-macie
+
+# Verify Macie is enabled
+aws macie2 get-macie-session
+
+# Enable automated sensitive data discovery
+aws macie2 update-automated-discovery-configuration \
+  --status ENABLED
+```
+
+### Via Terraform
+
+```hcl
+resource "aws_macie2_account" "main" {}
+
+resource "aws_macie2_classification_export_configuration" "main" {
+  depends_on = [aws_macie2_account.main]
+
+  s3_destination {
+    bucket_name = aws_s3_bucket.macie_results.id
+    key_prefix  = "macie-findings/"
+    kms_key_arn = aws_kms_key.macie.arn
+  }
+}
+```
+
+## Configure Discovery Jobs
+
+### Create a classification job for specific buckets
+
+```bash
+aws macie2 create-classification-job \
+  --job-type ONE_TIME \
+  --name "pii-scan-production-buckets" \
+  --s3-job-definition '{
+    "bucketDefinitions": [{
+      "accountId": "123456789012",
+      "buckets": [
+        "production-data-bucket",
+        "customer-records-bucket"
+      ]
+    }]
+  }' \
+  --managed-data-identifier-selector ALL
+```
+
+### Create a scheduled recurring job
+
+```bash
+aws macie2 create-classification-job \
+  --job-type SCHEDULED \
+  --name "weekly-sensitive-data-scan" \
+  --schedule-frequency-details '{
+    "weekly": {
+      "dayOfWeek": "MONDAY"
+    }
+  }' \
+  --s3-job-definition '{
+    "bucketDefinitions": [{
+      "accountId": "123456789012",
+      "buckets": ["all-data-bucket"]
+    }],
+    "scoping": {
+      "includes": {
+        "and": [{
+          "simpleScopeTerm": {
+            "comparator": "STARTS_WITH",
+            "key": "OBJECT_KEY",
+            "values": ["uploads/", "documents/"]
+          }
+        }]
+      }
+    }
+  }'
+```
+
+## Custom Data Identifiers
+
+### Create a custom identifier for internal IDs
+
+```bash
+aws macie2 create-custom-data-identifier \
+  --name "internal-employee-id" \
+  --description "Matches internal employee ID format EMP-XXXXXX" \
+  --regex "EMP-[0-9]{6}" \
+  --severity-levels '[
+    {"occurrencesThreshold": 1, "severity": "LOW"},
+    {"occurrencesThreshold": 10, "severity": "MEDIUM"},
+    {"occurrencesThreshold": 50, "severity": "HIGH"}
+  ]'
+```
+
+### Create identifier for project codes
+
+```bash
+aws macie2 create-custom-data-identifier \
+  --name "project-code-identifier" \
+  --description "Matches project codes in format PRJ-XXXX-XX" \
+  --regex "PRJ-[A-Z]{4}-[0-9]{2}" \
+  --keywords '["project", "code", "initiative"]' \
+  --maximum-match-distance 50
+```
+
+## Allow Lists
+
+### Create an allow list to suppress false positives
+
+```bash
+aws macie2 create-allow-list \
+  --name "test-data-exclusions" \
+  --description "Exclude known test data patterns" \
+  --criteria '{
+    "regex": "TEST-[0-9]{4}-[0-9]{4}-[0-9]{4}-[0-9]{4}"
+  }'
+```
+
+## Managed Data Identifiers
+
+Macie provides 300+ managed data identifiers covering:
+
+| Category | Examples |
+|----------|---------|
+| **PII** | SSN, passport numbers, driver's license, date of birth, names, addresses |
+| **Financial** | Credit card numbers, bank account numbers, SWIFT codes |
+| **Credentials** | AWS secret keys, API keys, SSH private keys, OAuth tokens |
+| **Health** | HIPAA identifiers, health insurance claim numbers |
+| **Legal** | Tax identification numbers, national ID numbers |
+
+## Findings Management
+
+### List findings
+
+```bash
+# Get sensitive data findings
+aws macie2 list-findings \
+  --finding-criteria '{
+    "criterion": {
+      "severity.description": {
+        "eq": ["High"]
+      },
+      "category": {
+        "eq": ["CLASSIFICATION"]
+      }
+    }
+  }' \
+  --sort-criteria '{"attributeName": "updatedAt", "orderBy": "DESC"}' \
+  --max-results 25
+```
+
+### Get finding details
+
+```bash
+aws macie2 get-findings \
+  --finding-ids '["finding-id-1", "finding-id-2"]'
+```
+
+### Export findings to Security Hub
+
+```bash
+# Macie automatically publishes findings to Security Hub
+# Verify integration:
+aws macie2 get-macie-session --query 'findingPublishingFrequency'
+```
+
+## EventBridge Integration for Automated Response
+
+```json
+{
+  "source": ["aws.macie"],
+  "detail-type": ["Macie Finding"],
+  "detail": {
+    "severity": {
+      "description": ["High", "Critical"]
+    }
+  }
+}
+```
+
+### Lambda function for automated remediation
+
+```python
+import boto3
+import json
+
+s3 = boto3.client('s3')
+sns = boto3.client('sns')
+
+def lambda_handler(event, context):
+    finding = event['detail']
+    severity = finding['severity']['description']
+    bucket = finding['resourcesAffected']['s3Bucket']['name']
+    key = finding['resourcesAffected']['s3Object']['key']
+    sensitive_types = [d['type'] for d in finding.get('classificationDetails', {}).get('result', {}).get('sensitiveData', [])]
+
+    if severity in ['High', 'Critical']:
+        # Tag the object for review
+        s3.put_object_tagging(
+            Bucket=bucket,
+            Key=key,
+            Tagging={
+                'TagSet': [
+                    {'Key': 'macie-finding', 'Value': severity},
+                    {'Key': 'sensitive-data', 'Value': ','.join(sensitive_types)},
+                    {'Key': 'requires-review', 'Value': 'true'}
+                ]
+            }
+        )
+
+        # Notify security team
+        sns.publish(
+            TopicArn='arn:aws:sns:us-east-1:123456789012:security-alerts',
+            Subject=f'Macie {severity} Finding: {bucket}/{key}',
+            Message=json.dumps({
+                'bucket': bucket,
+                'key': key,
+                'severity': severity,
+                'sensitive_data_types': sensitive_types,
+                'finding_id': finding['id']
+            }, indent=2)
+        )
+
+    return {'statusCode': 200}
+```
+
+## Multi-Account Deployment
+
+### Designate Macie administrator account
+
+```bash
+# From the management account
+aws macie2 enable-organization-admin-account \
+  --admin-account-id 111111111111
+```
+
+### Add member accounts
+
+```bash
+# From the administrator account
+aws macie2 create-member \
+  --account '{"accountId": "222222222222", "email": "security@example.com"}'
+```
+
+## Monitoring Macie Operations
+
+### Usage statistics
+
+```bash
+aws macie2 get-usage-statistics \
+  --filter-by '[{"comparator": "GT", "key": "accountId", "values": []}]' \
+  --sort-by '{"key": "accountId", "orderBy": "ASC"}'
+```
+
+### Classification job status
+
+```bash
+aws macie2 list-classification-jobs \
+  --filter-criteria '{"includes": [{"comparator": "EQ", "key": "jobStatus", "values": ["RUNNING"]}]}'
+```
+
+## References
+
+- AWS Macie Documentation: https://docs.aws.amazon.com/macie/
+- AWS Macie Pricing
+- Supported File Types for Macie Analysis
+- GDPR and CCPA Compliance with Macie
@@ -0,0 +1,32 @@
+# AWS Macie Data Classification Template
+
+## Configuration
+
+| Setting | Value |
+|---------|-------|
+| Administrator Account | |
+| Regions Enabled | |
+| Automated Discovery | Enabled / Disabled |
+| Finding Publication | 15min / 1hr / 6hr |
+| Export Bucket | |
+
+## S3 Bucket Coverage
+
+| Bucket | Classification Job | Sensitive Data Found | Action Taken |
+|--------|-------------------|---------------------|-------------|
+| | | | |
+
+## Custom Data Identifiers
+
+| Name | Regex Pattern | Severity Levels | Purpose |
+|------|--------------|-----------------|---------|
+| | | | |
+
+## Findings Summary
+
+| Severity | Count | Top Data Type | Remediation Status |
+|----------|-------|--------------|-------------------|
+| Critical | | | |
+| High | | | |
+| Medium | | | |
+| Low | | | |
@@ -0,0 +1,18 @@
+# Standards - AWS Macie for Data Classification
+
+## Compliance Frameworks Supported
+- GDPR Article 30: Records of processing activities
+- CCPA: California Consumer Privacy Act data discovery
+- HIPAA: Protected health information identification
+- PCI DSS 4.0: Cardholder data discovery (Requirement 3)
+- SOC 2: Data classification and protection controls
+
+## AWS Well-Architected Security Pillar
+- SEC 8: Protect data at rest
+- SEC 10: Prepare for security events
+
+## NIST 800-53 Controls
+- RA-5: Vulnerability Monitoring and Scanning
+- SC-28: Protection of Information at Rest
+- SI-4: System Monitoring
+- MP-4: Media Storage
@@ -0,0 +1,22 @@
+# Workflows - AWS Macie Data Classification
+
+## Implementation Workflow
+```
+1. Enable Macie → Configure administrator account
+2. Bucket Inventory → Review automated S3 inventory
+3. Discovery Jobs → Create targeted classification jobs
+4. Custom Identifiers → Add organization-specific patterns
+5. Allow Lists → Suppress known false positives
+6. Automation → EventBridge + Lambda for response
+7. Reporting → Dashboard and Security Hub integration
+```
+
+## Remediation Workflow
+```
+1. Finding Generated → Macie detects sensitive data
+2. Triage → Security team reviews severity and data type
+3. Classify → Determine data classification level
+4. Protect → Apply encryption, access controls, or relocate
+5. Validate → Re-scan to confirm remediation
+6. Document → Update data classification inventory
+```
@@ -0,0 +1,162 @@
+#!/usr/bin/env python3
+"""
+AWS Macie Data Classification Management Script
+
+Automates Macie configuration, job creation, and findings analysis.
+"""
+
+import boto3
+import json
+import sys
+from datetime import datetime
+
+
+def enable_macie(session):
+    """Enable Macie in the current account."""
+    client = session.client('macie2')
+    try:
+        client.enable_macie(status='ENABLED')
+        print("[+] Macie enabled successfully")
+    except client.exceptions.ConflictException:
+        print("[*] Macie is already enabled")
+
+    # Enable automated discovery
+    try:
+        client.update_automated_discovery_configuration(status='ENABLED')
+        print("[+] Automated discovery enabled")
+    except Exception as e:
+        print(f"[!] Could not enable automated discovery: {e}")
+
+
+def create_classification_job(session, bucket_names, account_id, job_name=None):
+    """Create a one-time classification job for specified buckets."""
+    client = session.client('macie2')
+
+    if not job_name:
+        job_name = f"scan-{datetime.now().strftime('%Y%m%d-%H%M%S')}"
+
+    response = client.create_classification_job(
+        jobType='ONE_TIME',
+        name=job_name,
+        s3JobDefinition={
+            'bucketDefinitions': [{
+                'accountId': account_id,
+                'buckets': bucket_names
+            }]
+        },
+        managedDataIdentifierSelector='ALL'
+    )
+
+    job_id = response['jobId']
+    print(f"[+] Classification job created: {job_id}")
+    return job_id
+
+
+def list_findings(session, severity=None, max_results=25):
+    """List Macie findings with optional severity filter."""
+    client = session.client('macie2')
+
+    criteria = {}
+    if severity:
+        criteria['severity.description'] = {'eq': [severity]}
+
+    response = client.list_findings(
+        findingCriteria={'criterion': criteria} if criteria else {},
+        sortCriteria={'attributeName': 'updatedAt', 'orderBy': 'DESC'},
+        maxResults=max_results
+    )
+
+    finding_ids = response.get('findingIds', [])
+    print(f"[+] Found {len(finding_ids)} findings")
+
+    if finding_ids:
+        details = client.get_findings(findingIds=finding_ids[:20])
+        for finding in details.get('findings', []):
+            severity_desc = finding['severity']['description']
+            category = finding.get('category', 'N/A')
+            title = finding.get('title', 'N/A')
+            bucket = finding.get('resourcesAffected', {}).get('s3Bucket', {}).get('name', 'N/A')
+            print(f"  [{severity_desc}] {title}")
+            print(f"    Bucket: {bucket} | Category: {category}")
+
+    return finding_ids
+
+
+def get_bucket_statistics(session):
+    """Get Macie statistics for all monitored S3 buckets."""
+    client = session.client('macie2')
+
+    response = client.describe_buckets(
+        criteria={},
+        maxResults=50
+    )
+
+    buckets = response.get('buckets', [])
+    print(f"\n[+] Monitoring {len(buckets)} S3 buckets:")
+    print(f"{'Bucket':<40} {'Encryption':<15} {'Public':<10} {'Shared':<10} {'Objects':<10}")
+    print("-" * 85)
+
+    for bucket in buckets:
+        name = bucket.get('bucketName', 'N/A')
+        encryption = bucket.get('serverSideEncryption', {}).get('type', 'NONE')
+        public_access = bucket.get('publicAccess', {}).get('effectivePermission', 'NOT_PUBLIC')
+        shared = bucket.get('sharedAccess', 'NOT_SHARED')
+        obj_count = bucket.get('objectCount', 0)
+
+        public_flag = "YES" if public_access == "PUBLIC" else "NO"
+        shared_flag = "YES" if shared != "NOT_SHARED" else "NO"
+
+        print(f"{name:<40} {encryption:<15} {public_flag:<10} {shared_flag:<10} {obj_count:<10}")
+
+    return buckets
+
+
+def get_usage_stats(session):
+    """Get Macie usage statistics."""
+    client = session.client('macie2')
+
+    response = client.get_usage_totals()
+    usage = response.get('usageTotals', [])
+
+    print("\n[+] Macie Usage Statistics:")
+    for item in usage:
+        usage_type = item.get('type', 'N/A')
+        amount = item.get('estimatedCost', 0)
+        currency = item.get('currency', 'USD')
+        print(f"  {usage_type}: ${amount:.2f} {currency}")
+
+
+if __name__ == "__main__":
+    import argparse
+
+    parser = argparse.ArgumentParser(description="AWS Macie Management")
+    parser.add_argument("--enable", action="store_true", help="Enable Macie")
+    parser.add_argument("--scan", nargs="+", help="Create scan job for buckets")
+    parser.add_argument("--account-id", type=str, help="AWS account ID for scan")
+    parser.add_argument("--findings", action="store_true", help="List findings")
+    parser.add_argument("--severity", type=str, choices=["Low", "Medium", "High"], help="Filter by severity")
+    parser.add_argument("--buckets", action="store_true", help="Show bucket statistics")
+    parser.add_argument("--usage", action="store_true", help="Show usage statistics")
+    parser.add_argument("--region", type=str, default="us-east-1", help="AWS region")
+    parser.add_argument("--profile", type=str, help="AWS profile name")
+
+    args = parser.parse_args()
+
+    session_kwargs = {"region_name": args.region}
+    if args.profile:
+        session_kwargs["profile_name"] = args.profile
+    session = boto3.Session(**session_kwargs)
+
+    if args.enable:
+        enable_macie(session)
+    if args.scan:
+        if not args.account_id:
+            sts = session.client('sts')
+            args.account_id = sts.get_caller_identity()['Account']
+        create_classification_job(session, args.scan, args.account_id)
+    if args.findings:
+        list_findings(session, severity=args.severity)
+    if args.buckets:
+        get_bucket_statistics(session)
+    if args.usage:
+        get_usage_stats(session)