Initial commit - 611 cybersecurity skills across all subdomains

This commit is contained in:
mukul975
2026-02-25 10:47:44 +01:00
commit 22a7ab1462
1765 changed files with 280648 additions and 0 deletions
@@ -0,0 +1,69 @@
---
name: implementing-envelope-encryption-with-aws-kms
description: Envelope encryption is a strategy where data is encrypted with a data encryption key (DEK), and the DEK itself is encrypted with a master key (KEK) managed by AWS KMS. This approach allows encrypting
domain: cybersecurity
subdomain: cryptography
tags: [cryptography, encryption, aws, kms, envelope-encryption, key-management]
version: "1.0"
author: mahipal
license: MIT
---
# Implementing Envelope Encryption with AWS KMS
## Overview
Envelope encryption is a strategy where data is encrypted with a data encryption key (DEK), and the DEK itself is encrypted with a master key (KEK) managed by AWS KMS. This approach allows encrypting large volumes of data locally while keeping the master key secure in a hardware security module (HSM) managed by AWS. This skill covers implementing envelope encryption using AWS KMS GenerateDataKey API.
## Objectives
- Understand the envelope encryption pattern and its advantages
- Generate data encryption keys using AWS KMS GenerateDataKey
- Encrypt/decrypt data locally using DEKs
- Store encrypted DEK alongside ciphertext
- Implement key caching to reduce KMS API calls
- Handle key rotation with automatic re-encryption
- Implement multi-region encryption for disaster recovery
## Key Concepts
### Envelope Encryption Flow
1. Call `kms:GenerateDataKey` to get plaintext DEK + encrypted DEK
2. Use plaintext DEK to encrypt data locally (AES-256-GCM)
3. Store encrypted DEK alongside ciphertext
4. Discard plaintext DEK from memory
5. For decryption: call `kms:Decrypt` on encrypted DEK, then decrypt data
### Advantages Over Direct KMS Encryption
| Aspect | Direct KMS | Envelope Encryption |
|--------|-----------|-------------------|
| Max data size | 4 KB | Unlimited |
| Latency | Network round-trip per operation | Local encryption |
| Cost | $0.03/10,000 requests | Fewer KMS requests |
| Offline | Not possible | Yes (with cached DEKs) |
### KMS Key Types
- **AWS Managed**: AWS creates and manages (`aws/s3`, `aws/ebs`)
- **Customer Managed**: You create and manage policies
- **Custom Key Store**: Backed by CloudHSM cluster
## Security Considerations
- Never store plaintext DEK; only keep encrypted DEK
- Use key policies to restrict who can call GenerateDataKey and Decrypt
- Enable AWS CloudTrail logging for all KMS API calls
- Implement key rotation (automatic annual rotation for CMKs)
- Use encryption context for authenticated encryption metadata
- Handle KMS throttling with exponential backoff
## Validation Criteria
- [ ] GenerateDataKey returns plaintext and encrypted DEK
- [ ] Data encrypts correctly with plaintext DEK using AES-256-GCM
- [ ] Encrypted DEK can be decrypted via KMS Decrypt API
- [ ] Decrypted DEK recovers the original data
- [ ] Plaintext DEK is wiped from memory after use
- [ ] Encryption context is validated during decryption
- [ ] Key rotation re-encrypts DEKs with new master key
@@ -0,0 +1,90 @@
# Envelope Encryption with AWS KMS Template
## Prerequisites Checklist
- [ ] AWS account with KMS access
- [ ] IAM policy allows kms:GenerateDataKey, kms:Decrypt, kms:ReEncrypt
- [ ] KMS Customer Managed Key (CMK) created
- [ ] CloudTrail logging enabled for KMS events
- [ ] boto3 and cryptography Python libraries installed
## IAM Policy Template
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kms:GenerateDataKey",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:DescribeKey"
],
"Resource": "arn:aws:kms:us-east-1:123456789012:key/your-key-id"
}
]
}
```
## KMS Key Policy Template
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowKeyAdministration",
"Effect": "Allow",
"Principal": {"AWS": "arn:aws:iam::123456789012:role/KeyAdmin"},
"Action": [
"kms:Create*", "kms:Describe*", "kms:Enable*", "kms:List*",
"kms:Put*", "kms:Update*", "kms:Revoke*", "kms:Disable*",
"kms:Get*", "kms:Delete*", "kms:ScheduleKeyDeletion",
"kms:CancelKeyDeletion"
],
"Resource": "*"
},
{
"Sid": "AllowKeyUsage",
"Effect": "Allow",
"Principal": {"AWS": "arn:aws:iam::123456789012:role/AppRole"},
"Action": ["kms:Decrypt", "kms:GenerateDataKey", "kms:ReEncrypt*"],
"Resource": "*"
}
]
}
```
## Quick Reference
```python
import boto3
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
import os
kms = boto3.client('kms')
# Encrypt
resp = kms.generate_data_key(KeyId='alias/my-key', KeySpec='AES_256')
plaintext_key = resp['Plaintext']
encrypted_key = resp['CiphertextBlob']
nonce = os.urandom(12)
ciphertext = AESGCM(plaintext_key).encrypt(nonce, data, None)
# Store: encrypted_key + nonce + ciphertext
# Decrypt
resp = kms.decrypt(CiphertextBlob=encrypted_key)
plaintext_key = resp['Plaintext']
data = AESGCM(plaintext_key).decrypt(nonce, ciphertext, None)
```
## Cost Estimation
| Operation | Price | Notes |
|-----------|-------|-------|
| KMS API requests | $0.03 per 10,000 | GenerateDataKey, Decrypt |
| CMK storage | $1.00 per month | Per customer managed key |
| Key rotation | Free | Automatic annual rotation |
@@ -0,0 +1,53 @@
# Standards and References - Envelope Encryption with AWS KMS
## AWS Documentation
### AWS KMS Developer Guide
- **URL**: https://docs.aws.amazon.com/kms/latest/developerguide/
- **Envelope Encryption**: https://docs.aws.amazon.com/kms/latest/developerguide/concepts.html#enveloping
### AWS KMS API Reference
- **GenerateDataKey**: https://docs.aws.amazon.com/kms/latest/APIReference/API_GenerateDataKey.html
- **Decrypt**: https://docs.aws.amazon.com/kms/latest/APIReference/API_Decrypt.html
- **ReEncrypt**: https://docs.aws.amazon.com/kms/latest/APIReference/API_ReEncrypt.html
### AWS Encryption SDK
- **URL**: https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/
- **Python**: https://aws-encryption-sdk-python.readthedocs.io/
## Cryptographic Standards
### NIST SP 800-57 Part 1 - Key Management
- **URL**: https://csrc.nist.gov/publications/detail/sp/800-57-part-1/rev-5/final
- **Relevance**: Key hierarchy and key wrapping concepts
### NIST SP 800-38F - Key Wrap
- **URL**: https://csrc.nist.gov/publications/detail/sp/800-38f/final
- **Description**: AES Key Wrap specification used by KMS internally
### FIPS 140-2 Level 2 (KMS HSMs)
- **Description**: KMS HSMs are validated at FIPS 140-2 Level 2 (Level 3 for CloudHSM)
## Compliance Frameworks
### PCI DSS v4.0 Requirement 3
- Key management with separation of DEK and KEK
- KMS satisfies key management requirements
### SOC 2 Type II
- AWS KMS is SOC 2 compliant
- Encryption controls map to CC6.1 (logical access controls)
### HIPAA
- KMS encryption satisfies encryption requirements for ePHI
- BAA required with AWS
## Python Libraries
### boto3 (AWS SDK for Python)
- **URL**: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/kms.html
- **PyPI**: https://pypi.org/project/boto3/
### aws-encryption-sdk
- **URL**: https://pypi.org/project/aws-encryption-sdk/
- **Description**: High-level envelope encryption with caching
@@ -0,0 +1,76 @@
# Workflows - Envelope Encryption with AWS KMS
## Workflow 1: Encrypt Data with Envelope Encryption
```
[Application]
|
[Call KMS GenerateDataKey]
(KeyId=CMK ARN, KeySpec=AES_256)
|
[KMS Returns]:
- Plaintext DEK (32 bytes)
- Encrypted DEK (ciphertext blob)
|
[Encrypt Data Locally]
(AES-256-GCM with plaintext DEK)
|
[Store]:
- Encrypted DEK (ciphertext blob)
- Encrypted data (nonce + ciphertext + tag)
- Encryption context metadata
|
[Wipe Plaintext DEK from Memory]
```
## Workflow 2: Decrypt Data
```
[Read Stored Data]
- Encrypted DEK
- Encrypted data
- Encryption context
|
[Call KMS Decrypt]
(CiphertextBlob=Encrypted DEK, EncryptionContext)
|
[KMS Returns Plaintext DEK]
|
[Decrypt Data Locally]
(AES-256-GCM with plaintext DEK)
|
[Return Plaintext Data]
|
[Wipe Plaintext DEK from Memory]
```
## Workflow 3: Key Rotation
```
[Enable Automatic Key Rotation on CMK]
(KMS rotates backing key annually)
|
[New GenerateDataKey calls use new backing key]
|
[Old encrypted DEKs still decrypt]
(KMS tracks all backing key versions)
|
[Optional: Re-encrypt old DEKs]
[Call KMS ReEncrypt to update DEK encryption]
```
## Workflow 4: Multi-Region Encryption
```
[Primary Region (us-east-1)]
|
[Create Multi-Region CMK]
[Replicate to us-west-2, eu-west-1]
|
[Encrypt with Regional Endpoint]
|
[Secondary Region (us-west-2)]
|
[Same Key ID works for Decrypt]
[No cross-region API calls needed]
```
@@ -0,0 +1,345 @@
#!/usr/bin/env python3
"""
Envelope Encryption with AWS KMS
Implements envelope encryption pattern using AWS KMS for master key management
and local AES-256-GCM for data encryption.
Requirements:
pip install boto3 cryptography
Usage:
python process.py encrypt --key-id alias/my-key --input data.json --output data.json.enc
python process.py decrypt --input data.json.enc --output data.json
python process.py encrypt-file --key-id alias/my-key --input largefile.zip --output largefile.zip.enc
python process.py re-encrypt --key-id alias/new-key --input data.json.enc --output data.json.reenc
Environment:
AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_DEFAULT_REGION
or ~/.aws/credentials
"""
import os
import sys
import json
import struct
import argparse
import logging
import base64
import ctypes
from pathlib import Path
from typing import Dict, Optional, Tuple
import boto3
from botocore.exceptions import ClientError
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)
NONCE_LENGTH = 12
MAGIC_BYTES = b"ENVKMS01"
def secure_wipe(data: bytearray):
"""Attempt to securely wipe sensitive data from memory."""
if isinstance(data, (bytearray, memoryview)):
for i in range(len(data)):
data[i] = 0
def generate_data_key(
kms_client,
key_id: str,
encryption_context: Optional[Dict[str, str]] = None,
) -> Tuple[bytes, bytes]:
"""
Generate a data encryption key (DEK) using AWS KMS.
Returns:
Tuple of (plaintext_key, encrypted_key)
"""
params = {
"KeyId": key_id,
"KeySpec": "AES_256",
}
if encryption_context:
params["EncryptionContext"] = encryption_context
try:
response = kms_client.generate_data_key(**params)
plaintext_key = response["Plaintext"]
encrypted_key = response["CiphertextBlob"]
logger.info(f"Generated data key using KMS key: {key_id}")
return plaintext_key, encrypted_key
except ClientError as e:
logger.error(f"KMS GenerateDataKey failed: {e}")
raise
def decrypt_data_key(
kms_client,
encrypted_key: bytes,
encryption_context: Optional[Dict[str, str]] = None,
) -> bytes:
"""Decrypt an encrypted data key using AWS KMS."""
params = {"CiphertextBlob": encrypted_key}
if encryption_context:
params["EncryptionContext"] = encryption_context
try:
response = kms_client.decrypt(**params)
return response["Plaintext"]
except ClientError as e:
logger.error(f"KMS Decrypt failed: {e}")
raise
def re_encrypt_data_key(
kms_client,
encrypted_key: bytes,
new_key_id: str,
source_context: Optional[Dict[str, str]] = None,
dest_context: Optional[Dict[str, str]] = None,
) -> bytes:
"""Re-encrypt a data key with a new master key without exposing plaintext."""
params = {
"CiphertextBlob": encrypted_key,
"DestinationKeyId": new_key_id,
}
if source_context:
params["SourceEncryptionContext"] = source_context
if dest_context:
params["DestinationEncryptionContext"] = dest_context
try:
response = kms_client.re_encrypt(**params)
return response["CiphertextBlob"]
except ClientError as e:
logger.error(f"KMS ReEncrypt failed: {e}")
raise
def envelope_encrypt(
data: bytes,
kms_client,
key_id: str,
encryption_context: Optional[Dict[str, str]] = None,
) -> bytes:
"""
Encrypt data using envelope encryption.
Output format:
MAGIC (8 bytes) || encrypted_key_len (4 bytes, big-endian) ||
encrypted_key (variable) || context_len (4 bytes) || context_json (variable) ||
nonce (12 bytes) || ciphertext+tag (variable)
"""
plaintext_key, encrypted_key = generate_data_key(kms_client, key_id, encryption_context)
try:
nonce = os.urandom(NONCE_LENGTH)
aesgcm = AESGCM(plaintext_key)
ciphertext = aesgcm.encrypt(nonce, data, associated_data=None)
finally:
# Wipe plaintext key
if isinstance(plaintext_key, bytes):
plaintext_key = bytearray(plaintext_key)
secure_wipe(plaintext_key)
context_json = json.dumps(encryption_context or {}).encode()
output = bytearray()
output.extend(MAGIC_BYTES)
output.extend(struct.pack(">I", len(encrypted_key)))
output.extend(encrypted_key)
output.extend(struct.pack(">I", len(context_json)))
output.extend(context_json)
output.extend(nonce)
output.extend(ciphertext)
return bytes(output)
def envelope_decrypt(
data: bytes,
kms_client,
) -> Tuple[bytes, Dict]:
"""
Decrypt envelope-encrypted data.
Returns:
Tuple of (plaintext, encryption_context)
"""
if not data.startswith(MAGIC_BYTES):
raise ValueError("Invalid envelope encryption format")
offset = len(MAGIC_BYTES)
enc_key_len = struct.unpack(">I", data[offset : offset + 4])[0]
offset += 4
encrypted_key = data[offset : offset + enc_key_len]
offset += enc_key_len
ctx_len = struct.unpack(">I", data[offset : offset + 4])[0]
offset += 4
context_json = data[offset : offset + ctx_len]
offset += ctx_len
encryption_context = json.loads(context_json.decode())
nonce = data[offset : offset + NONCE_LENGTH]
offset += NONCE_LENGTH
ciphertext = data[offset:]
plaintext_key = decrypt_data_key(
kms_client,
encrypted_key,
encryption_context if encryption_context else None,
)
try:
aesgcm = AESGCM(plaintext_key)
plaintext = aesgcm.decrypt(nonce, ciphertext, associated_data=None)
finally:
if isinstance(plaintext_key, bytes):
plaintext_key = bytearray(plaintext_key)
secure_wipe(plaintext_key)
return plaintext, encryption_context
def envelope_re_encrypt(
data: bytes,
kms_client,
new_key_id: str,
) -> bytes:
"""Re-encrypt an envelope-encrypted file with a new KMS key."""
if not data.startswith(MAGIC_BYTES):
raise ValueError("Invalid envelope encryption format")
offset = len(MAGIC_BYTES)
enc_key_len = struct.unpack(">I", data[offset : offset + 4])[0]
offset += 4
encrypted_key = data[offset : offset + enc_key_len]
offset += enc_key_len
ctx_len = struct.unpack(">I", data[offset : offset + 4])[0]
ctx_start = offset + 4
context_json = data[ctx_start : ctx_start + ctx_len]
encryption_context = json.loads(context_json.decode())
remainder = data[ctx_start + ctx_len :]
new_encrypted_key = re_encrypt_data_key(
kms_client,
encrypted_key,
new_key_id,
source_context=encryption_context if encryption_context else None,
dest_context=encryption_context if encryption_context else None,
)
output = bytearray()
output.extend(MAGIC_BYTES)
output.extend(struct.pack(">I", len(new_encrypted_key)))
output.extend(new_encrypted_key)
output.extend(struct.pack(">I", ctx_len))
output.extend(context_json)
output.extend(remainder)
logger.info(f"Re-encrypted data key with new KMS key: {new_key_id}")
return bytes(output)
def encrypt_file(
input_path: str,
output_path: str,
kms_client,
key_id: str,
encryption_context: Optional[Dict[str, str]] = None,
) -> Dict:
"""Encrypt a file using envelope encryption."""
plaintext = Path(input_path).read_bytes()
if encryption_context is None:
encryption_context = {"filename": Path(input_path).name}
encrypted = envelope_encrypt(plaintext, kms_client, key_id, encryption_context)
Path(output_path).write_bytes(encrypted)
logger.info(f"Encrypted {input_path} -> {output_path}")
return {
"input": input_path,
"output": output_path,
"original_size": len(plaintext),
"encrypted_size": len(encrypted),
"kms_key_id": key_id,
"encryption_context": encryption_context,
}
def decrypt_file(input_path: str, output_path: str, kms_client) -> Dict:
"""Decrypt an envelope-encrypted file."""
data = Path(input_path).read_bytes()
plaintext, context = envelope_decrypt(data, kms_client)
Path(output_path).write_bytes(plaintext)
logger.info(f"Decrypted {input_path} -> {output_path}")
return {
"input": input_path,
"output": output_path,
"decrypted_size": len(plaintext),
"encryption_context": context,
}
def main():
parser = argparse.ArgumentParser(description="Envelope Encryption with AWS KMS")
parser.add_argument("--region", default=None, help="AWS region")
parser.add_argument("--profile", default=None, help="AWS profile")
subparsers = parser.add_subparsers(dest="command")
enc = subparsers.add_parser("encrypt", help="Encrypt a file")
enc.add_argument("--key-id", required=True, help="KMS key ID or alias")
enc.add_argument("--input", "-i", required=True, help="Input file")
enc.add_argument("--output", "-o", required=True, help="Output file")
enc.add_argument("--context", help="Encryption context as JSON string")
dec = subparsers.add_parser("decrypt", help="Decrypt a file")
dec.add_argument("--input", "-i", required=True, help="Encrypted input file")
dec.add_argument("--output", "-o", required=True, help="Output file")
reenc = subparsers.add_parser("re-encrypt", help="Re-encrypt with new key")
reenc.add_argument("--key-id", required=True, help="New KMS key ID or alias")
reenc.add_argument("--input", "-i", required=True, help="Encrypted input file")
reenc.add_argument("--output", "-o", required=True, help="Output file")
args = parser.parse_args()
session_kwargs = {}
if args.region:
session_kwargs["region_name"] = args.region
if args.profile:
session_kwargs["profile_name"] = args.profile
session = boto3.Session(**session_kwargs)
kms_client = session.client("kms")
if args.command == "encrypt":
context = json.loads(args.context) if args.context else None
result = encrypt_file(args.input, args.output, kms_client, args.key_id, context)
print(json.dumps(result, indent=2))
elif args.command == "decrypt":
result = decrypt_file(args.input, args.output, kms_client)
print(json.dumps(result, indent=2))
elif args.command == "re-encrypt":
data = Path(args.input).read_bytes()
result = envelope_re_encrypt(data, kms_client, args.key_id)
Path(args.output).write_bytes(result)
print(json.dumps({"input": args.input, "output": args.output, "new_key_id": args.key_id}))
else:
parser.print_help()
if __name__ == "__main__":
main()