Initial commit - 611 cybersecurity skills across all subdomains

This commit is contained in:
mukul975
2026-02-25 10:47:44 +01:00
commit 22a7ab1462
1765 changed files with 280648 additions and 0 deletions
@@ -0,0 +1,82 @@
---
name: implementing-aes-encryption-for-data-at-rest
description: AES (Advanced Encryption Standard) is a symmetric block cipher standardized by NIST (FIPS 197) used to protect classified and sensitive data. This skill covers implementing AES-256 encryption in GCM m
domain: cybersecurity
subdomain: cryptography
tags: [cryptography, encryption, aes, data-at-rest, symmetric-encryption]
version: "1.0"
author: mahipal
license: MIT
---
# Implementing AES Encryption for Data at Rest
## Overview
AES (Advanced Encryption Standard) is a symmetric block cipher standardized by NIST (FIPS 197) used to protect classified and sensitive data. This skill covers implementing AES-256 encryption in GCM mode for encrypting files and data stores at rest, including proper key derivation, IV/nonce management, and authenticated encryption.
## Objectives
- Implement AES-256-GCM encryption and decryption for files
- Derive encryption keys from passwords using PBKDF2 and Argon2
- Manage initialization vectors (IVs) and nonces securely
- Encrypt and decrypt entire directory trees
- Implement authenticated encryption to detect tampering
- Handle large files with streaming encryption
## Key Concepts
### AES Modes of Operation
| Mode | Authentication | Parallelizable | Use Case |
|------|---------------|----------------|----------|
| GCM | Yes (AEAD) | Yes | Network data, file encryption |
| CBC | No | Decrypt only | Legacy systems, disk encryption |
| CTR | No | Yes | Streaming encryption |
| CCM | Yes (AEAD) | No | IoT, constrained environments |
### Key Derivation
Never use raw passwords as encryption keys. Always derive keys using:
- **PBKDF2**: NIST-approved, widely supported (minimum 600,000 iterations as of 2024)
- **Argon2id**: Winner of Password Hashing Competition, memory-hard
- **scrypt**: Memory-hard, good alternative to Argon2
### Nonce/IV Management
- GCM requires a 96-bit (12-byte) nonce that must NEVER be reused with the same key
- Generate nonces using `os.urandom()` (CSPRNG)
- Store nonce alongside ciphertext (it is not secret)
## Implementation Steps
1. Install the `cryptography` library: `pip install cryptography`
2. Generate or derive an encryption key
3. Create a random nonce for each encryption operation
4. Encrypt data using AES-256-GCM with the key and nonce
5. Store nonce + ciphertext + authentication tag together
6. For decryption, extract nonce, verify tag, and decrypt
## Encrypted File Format
```
[salt: 16 bytes][nonce: 12 bytes][ciphertext: variable][tag: 16 bytes]
```
## Security Considerations
- Always use authenticated encryption (GCM, CCM) to prevent tampering
- Never reuse a nonce with the same key (catastrophic in GCM)
- Use at least 256-bit keys for long-term data protection
- Securely wipe keys from memory after use when possible
- Rotate encryption keys periodically per organizational policy
- For disk-level encryption, consider XTS mode (AES-XTS)
## Validation Criteria
- [ ] AES-256-GCM encryption produces valid ciphertext
- [ ] Decryption recovers original plaintext exactly
- [ ] Authentication tag detects any ciphertext modification
- [ ] Key derivation uses sufficient iterations/parameters
- [ ] Nonces are never reused for the same key
- [ ] Large files (>1GB) can be processed via streaming
- [ ] Encrypted file format includes all necessary metadata
@@ -0,0 +1,92 @@
# AES Encryption Implementation Template
## Pre-Implementation Checklist
- [ ] Identify data classification level and regulatory requirements
- [ ] Determine key management strategy (local, HSM, KMS)
- [ ] Select AES mode (GCM recommended for authenticated encryption)
- [ ] Define key derivation parameters (algorithm, iterations)
- [ ] Plan nonce/IV generation strategy
- [ ] Determine encrypted file format and metadata storage
- [ ] Review compliance requirements (PCI-DSS, HIPAA, GDPR)
## Configuration Parameters
```yaml
encryption:
algorithm: AES-256-GCM
key_length: 256
nonce_length: 96 # bits
tag_length: 128 # bits
key_derivation:
algorithm: PBKDF2-SHA256
iterations: 600000
salt_length: 128 # bits
file_format:
magic_bytes: "AES256GCM"
version: 1
header: "magic || version || salt || nonce"
body: "ciphertext || tag"
```
## Integration Code Template
```python
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC
from cryptography.hazmat.primitives import hashes
import os
def encrypt_data(plaintext: bytes, password: str) -> bytes:
"""Encrypt data with AES-256-GCM."""
salt = os.urandom(16)
kdf = PBKDF2HMAC(
algorithm=hashes.SHA256(),
length=32,
salt=salt,
iterations=600_000,
)
key = kdf.derive(password.encode())
nonce = os.urandom(12)
aesgcm = AESGCM(key)
ciphertext = aesgcm.encrypt(nonce, plaintext, None)
return salt + nonce + ciphertext
def decrypt_data(data: bytes, password: str) -> bytes:
"""Decrypt AES-256-GCM encrypted data."""
salt = data[:16]
nonce = data[16:28]
ciphertext = data[28:]
kdf = PBKDF2HMAC(
algorithm=hashes.SHA256(),
length=32,
salt=salt,
iterations=600_000,
)
key = kdf.derive(password.encode())
aesgcm = AESGCM(key)
return aesgcm.decrypt(nonce, ciphertext, None)
```
## Testing Checklist
- [ ] Encrypt and decrypt a small text file
- [ ] Encrypt and decrypt a large binary file (>100MB)
- [ ] Verify wrong password raises authentication error
- [ ] Verify tampered ciphertext raises authentication error
- [ ] Verify nonce uniqueness across multiple encryptions
- [ ] Measure encryption throughput (MB/s)
- [ ] Test with empty files and edge cases
## Common Pitfalls
| Pitfall | Impact | Mitigation |
|---------|--------|------------|
| Nonce reuse with same key | Complete loss of confidentiality in GCM | Always generate random nonce per encryption |
| Low PBKDF2 iterations | Brute-force password attacks | Use minimum 600,000 iterations |
| ECB mode usage | Pattern leakage in ciphertext | Always use GCM or CBC (never ECB) |
| No authentication | Undetected ciphertext modification | Use AEAD modes (GCM, CCM) |
| Hardcoded keys | Key compromise | Use KMS, HSM, or environment variables |
| No key rotation | Extended exposure window | Implement periodic key rotation policy |
@@ -0,0 +1,76 @@
# Standards and References - AES Encryption for Data at Rest
## Primary Standards
### NIST FIPS 197 - Advanced Encryption Standard (AES)
- **URL**: https://csrc.nist.gov/publications/detail/fips/197/final
- **Description**: Defines the AES algorithm (Rijndael) with key sizes of 128, 192, and 256 bits
- **Block size**: 128 bits (16 bytes)
- **Key sizes**: 128, 192, or 256 bits
- **Rounds**: 10 (128-bit), 12 (192-bit), 14 (256-bit)
### NIST SP 800-38D - Recommendation for Block Cipher Modes: GCM and GMAC
- **URL**: https://csrc.nist.gov/publications/detail/sp/800-38d/final
- **Description**: Specifies Galois/Counter Mode (GCM) for authenticated encryption
- **IV length**: 96 bits recommended for GCM
- **Tag length**: 128 bits recommended (minimum 96 bits)
- **Max plaintext**: 2^39 - 256 bits per invocation
### NIST SP 800-132 - Recommendation for Password-Based Key Derivation
- **URL**: https://csrc.nist.gov/publications/detail/sp/800-132/final
- **Description**: Covers PBKDF2 for deriving cryptographic keys from passwords
- **Minimum iterations**: 600,000 (OWASP 2024 recommendation for PBKDF2-SHA256)
- **Salt length**: Minimum 128 bits (16 bytes)
### NIST SP 800-38A - Recommendation for Block Cipher Modes of Operation
- **URL**: https://csrc.nist.gov/publications/detail/sp/800-38a/final
- **Description**: Defines ECB, CBC, CFB, OFB, and CTR modes
### NIST SP 800-57 Part 1 Rev. 5 - Key Management
- **URL**: https://csrc.nist.gov/publications/detail/sp/800-57-part-1/rev-5/final
- **Description**: Recommendations for cryptographic key lengths and algorithms
- **AES-256 security strength**: 256 bits
- **Recommended until**: Beyond 2031
## RFC Standards
### RFC 5116 - An Interface and Algorithms for Authenticated Encryption
- **URL**: https://www.rfc-editor.org/rfc/rfc5116
- **Description**: Defines AEAD interface including AES-GCM
### RFC 5869 - HMAC-based Extract-and-Expand Key Derivation Function (HKDF)
- **URL**: https://www.rfc-editor.org/rfc/rfc5869
- **Description**: Key derivation from existing key material (not passwords)
### RFC 9106 - Argon2 Memory-Hard Function
- **URL**: https://www.rfc-editor.org/rfc/rfc9106
- **Description**: Argon2 password hashing / key derivation specification
- **Recommended variant**: Argon2id (hybrid of Argon2i and Argon2d)
## Compliance Frameworks
### PCI DSS v4.0 - Requirement 3
- Encrypt stored cardholder data with strong cryptography
- AES-256 meets the strong cryptography requirement
- Key management procedures required
### HIPAA Security Rule - 45 CFR 164.312(a)(2)(iv)
- Encryption of ePHI at rest is an addressable implementation specification
- AES-256 is an acceptable encryption method
### GDPR Article 32 - Security of Processing
- Encryption is listed as an appropriate technical measure
- AES-256 satisfies encryption requirements for personal data protection
## Python Library References
### cryptography (pyca/cryptography)
- **URL**: https://cryptography.io/en/latest/
- **PyPI**: https://pypi.org/project/cryptography/
- **AES-GCM**: `cryptography.hazmat.primitives.ciphers.aead.AESGCM`
- **PBKDF2**: `cryptography.hazmat.primitives.kdf.pbkdf2.PBKDF2HMAC`
### PyCryptodome
- **URL**: https://pycryptodome.readthedocs.io/
- **PyPI**: https://pypi.org/project/pycryptodome/
- **AES-GCM**: `Crypto.Cipher.AES` with `MODE_GCM`
@@ -0,0 +1,122 @@
# Workflows - AES Encryption for Data at Rest
## Workflow 1: Single File Encryption
```
[Input File] --> [Read File Bytes]
|
[Derive Key from Password]
(PBKDF2 / Argon2id + random salt)
|
[Generate Random Nonce]
(12 bytes from CSPRNG)
|
[AES-256-GCM Encrypt]
(key + nonce + plaintext --> ciphertext + tag)
|
[Write Encrypted File]
(salt || nonce || ciphertext || tag)
```
## Workflow 2: Single File Decryption
```
[Encrypted File] --> [Parse Header]
(extract salt, nonce)
|
[Derive Key from Password]
(same PBKDF2 / Argon2id params + extracted salt)
|
[AES-256-GCM Decrypt]
(key + nonce + ciphertext + tag)
|
[Verify Authentication Tag]
(reject if tag invalid)
|
[Write Decrypted File]
```
## Workflow 3: Streaming Encryption for Large Files
```
[Large Input File]
|
[Read in Chunks] (e.g., 64KB chunks)
|
[For Each Chunk]:
- [Encrypt chunk with AES-256-CTR]
- [Update HMAC with ciphertext chunk]
- [Write encrypted chunk to output]
|
[Finalize HMAC]
[Append HMAC tag to output]
```
## Workflow 4: Directory Tree Encryption
```
[Source Directory]
|
[Walk Directory Tree]
|
[For Each File]:
- [Derive unique file key from master key + file path]
- [Generate random nonce]
- [AES-256-GCM encrypt file]
- [Write encrypted file preserving directory structure]
|
[Create Manifest File]
(maps original paths to encrypted paths with metadata)
```
## Workflow 5: Key Derivation Pipeline
```
[User Password]
|
[Generate Random Salt] (16 bytes)
|
[PBKDF2-SHA256]
- iterations: 600,000+
- dkLen: 32 bytes (256 bits)
|
[Derived Key (256-bit)]
|
[Optional: HKDF Expand]
- Derive multiple subkeys from single derived key
- info="encryption" --> encryption key
- info="authentication" --> HMAC key
```
## Workflow 6: Envelope Encryption Pattern
```
[Master Key] (stored in HSM/KMS)
|
[Generate Random Data Encryption Key (DEK)]
(32 bytes from CSPRNG)
|
[Encrypt DEK with Master Key] --> [Encrypted DEK]
|
[Encrypt Data with DEK] --> [Ciphertext]
|
[Store: Encrypted DEK + Ciphertext]
[Securely Wipe DEK from Memory]
```
## Error Handling Workflow
```
[Decryption Attempt]
|
[Parse Header] --FAIL--> [Return: Corrupt/invalid file format]
|
[Derive Key] --FAIL--> [Return: KDF parameter error]
|
[Decrypt + Verify Tag]
|
[Tag Valid?]
YES --> [Return plaintext]
NO --> [Return: Authentication failed - data tampered]
[DO NOT return partial plaintext]
```
@@ -0,0 +1,330 @@
#!/usr/bin/env python3
"""
AES-256-GCM Encryption for Data at Rest
Implements file and directory encryption using AES-256-GCM with
PBKDF2 key derivation. Supports single file, streaming large file,
and directory tree encryption.
Requirements:
pip install cryptography
Usage:
python process.py encrypt --input secret.pdf --output secret.pdf.enc --password "MySecurePass"
python process.py decrypt --input secret.pdf.enc --output secret.pdf --password "MySecurePass"
python process.py encrypt-dir --input ./sensitive/ --output ./encrypted/ --password "MySecurePass"
"""
import os
import sys
import json
import struct
import hashlib
import argparse
import logging
from pathlib import Path
from typing import Optional, Tuple
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.backends import default_backend
logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s")
logger = logging.getLogger(__name__)
# Constants
SALT_LENGTH = 16 # 128-bit salt
NONCE_LENGTH = 12 # 96-bit nonce (recommended for GCM)
TAG_LENGTH = 16 # 128-bit authentication tag
KEY_LENGTH = 32 # 256-bit key
PBKDF2_ITERATIONS = 600_000 # OWASP 2024 recommendation
CHUNK_SIZE = 64 * 1024 # 64KB chunks for streaming
MAGIC_BYTES = b"AES256GCM" # File format identifier
VERSION = 1
def derive_key(password: str, salt: bytes, iterations: int = PBKDF2_ITERATIONS) -> bytes:
"""Derive a 256-bit encryption key from a password using PBKDF2-SHA256."""
kdf = PBKDF2HMAC(
algorithm=hashes.SHA256(),
length=KEY_LENGTH,
salt=salt,
iterations=iterations,
backend=default_backend(),
)
return kdf.derive(password.encode("utf-8"))
def encrypt_bytes(plaintext: bytes, password: str) -> bytes:
"""
Encrypt plaintext bytes using AES-256-GCM with PBKDF2 key derivation.
Output format:
MAGIC (9 bytes) || VERSION (1 byte) || SALT (16 bytes) || NONCE (12 bytes) || CIPHERTEXT+TAG (variable)
The authentication tag is appended to ciphertext by AESGCM.
"""
salt = os.urandom(SALT_LENGTH)
nonce = os.urandom(NONCE_LENGTH)
key = derive_key(password, salt)
aesgcm = AESGCM(key)
ciphertext = aesgcm.encrypt(nonce, plaintext, associated_data=None)
header = MAGIC_BYTES + struct.pack("B", VERSION)
return header + salt + nonce + ciphertext
def decrypt_bytes(data: bytes, password: str) -> bytes:
"""
Decrypt AES-256-GCM encrypted data.
Raises:
ValueError: If file format is invalid or authentication fails.
"""
magic_len = len(MAGIC_BYTES)
min_length = magic_len + 1 + SALT_LENGTH + NONCE_LENGTH + TAG_LENGTH
if len(data) < min_length:
raise ValueError("Data too short to be a valid encrypted file")
magic = data[:magic_len]
if magic != MAGIC_BYTES:
raise ValueError(f"Invalid file format: expected magic bytes {MAGIC_BYTES!r}, got {magic!r}")
version = struct.unpack("B", data[magic_len : magic_len + 1])[0]
if version != VERSION:
raise ValueError(f"Unsupported version: {version}")
offset = magic_len + 1
salt = data[offset : offset + SALT_LENGTH]
offset += SALT_LENGTH
nonce = data[offset : offset + NONCE_LENGTH]
offset += NONCE_LENGTH
ciphertext = data[offset:]
key = derive_key(password, salt)
aesgcm = AESGCM(key)
try:
plaintext = aesgcm.decrypt(nonce, ciphertext, associated_data=None)
except Exception as e:
raise ValueError(
"Decryption failed: authentication tag verification failed. "
"Either the password is wrong or the data has been tampered with."
) from e
return plaintext
def encrypt_file(input_path: str, output_path: str, password: str) -> dict:
"""Encrypt a single file."""
input_file = Path(input_path)
if not input_file.exists():
raise FileNotFoundError(f"Input file not found: {input_path}")
plaintext = input_file.read_bytes()
original_size = len(plaintext)
original_hash = hashlib.sha256(plaintext).hexdigest()
ciphertext = encrypt_bytes(plaintext, password)
output_file = Path(output_path)
output_file.parent.mkdir(parents=True, exist_ok=True)
output_file.write_bytes(ciphertext)
encrypted_size = len(ciphertext)
logger.info(f"Encrypted {input_path} -> {output_path} ({original_size} -> {encrypted_size} bytes)")
return {
"input": str(input_path),
"output": str(output_path),
"original_size": original_size,
"encrypted_size": encrypted_size,
"original_sha256": original_hash,
"algorithm": "AES-256-GCM",
"kdf": "PBKDF2-SHA256",
"kdf_iterations": PBKDF2_ITERATIONS,
}
def decrypt_file(input_path: str, output_path: str, password: str) -> dict:
"""Decrypt a single file."""
input_file = Path(input_path)
if not input_file.exists():
raise FileNotFoundError(f"Input file not found: {input_path}")
data = input_file.read_bytes()
plaintext = decrypt_bytes(data, password)
output_file = Path(output_path)
output_file.parent.mkdir(parents=True, exist_ok=True)
output_file.write_bytes(plaintext)
recovered_hash = hashlib.sha256(plaintext).hexdigest()
logger.info(f"Decrypted {input_path} -> {output_path} ({len(plaintext)} bytes)")
return {
"input": str(input_path),
"output": str(output_path),
"decrypted_size": len(plaintext),
"recovered_sha256": recovered_hash,
}
def encrypt_directory(input_dir: str, output_dir: str, password: str) -> dict:
"""Encrypt all files in a directory tree, preserving structure."""
input_path = Path(input_dir)
output_path = Path(output_dir)
if not input_path.is_dir():
raise NotADirectoryError(f"Input is not a directory: {input_dir}")
output_path.mkdir(parents=True, exist_ok=True)
manifest = {
"algorithm": "AES-256-GCM",
"kdf": "PBKDF2-SHA256",
"kdf_iterations": PBKDF2_ITERATIONS,
"files": [],
}
file_count = 0
total_original = 0
total_encrypted = 0
for file in sorted(input_path.rglob("*")):
if file.is_file():
relative = file.relative_to(input_path)
encrypted_name = str(relative) + ".enc"
dest = output_path / encrypted_name
result = encrypt_file(str(file), str(dest), password)
manifest["files"].append({
"original_path": str(relative),
"encrypted_path": encrypted_name,
"original_sha256": result["original_sha256"],
"original_size": result["original_size"],
})
file_count += 1
total_original += result["original_size"]
total_encrypted += result["encrypted_size"]
manifest_path = output_path / "manifest.json"
manifest_encrypted = encrypt_bytes(json.dumps(manifest, indent=2).encode(), password)
(output_path / "manifest.json.enc").write_bytes(manifest_encrypted)
logger.info(
f"Encrypted {file_count} files from {input_dir} -> {output_dir} "
f"({total_original} -> {total_encrypted} bytes)"
)
return {
"files_encrypted": file_count,
"total_original_bytes": total_original,
"total_encrypted_bytes": total_encrypted,
"output_directory": str(output_path),
}
def decrypt_directory(input_dir: str, output_dir: str, password: str) -> dict:
"""Decrypt all .enc files in a directory tree."""
input_path = Path(input_dir)
output_path = Path(output_dir)
if not input_path.is_dir():
raise NotADirectoryError(f"Input is not a directory: {input_dir}")
output_path.mkdir(parents=True, exist_ok=True)
file_count = 0
total_decrypted = 0
for file in sorted(input_path.rglob("*.enc")):
if file.name == "manifest.json.enc":
continue
if file.is_file():
relative = file.relative_to(input_path)
decrypted_name = str(relative).removesuffix(".enc")
dest = output_path / decrypted_name
result = decrypt_file(str(file), str(dest), password)
file_count += 1
total_decrypted += result["decrypted_size"]
logger.info(f"Decrypted {file_count} files from {input_dir} -> {output_dir}")
return {
"files_decrypted": file_count,
"total_decrypted_bytes": total_decrypted,
"output_directory": str(output_path),
}
def verify_roundtrip(test_data: bytes, password: str) -> bool:
"""Verify encryption/decryption roundtrip integrity."""
encrypted = encrypt_bytes(test_data, password)
decrypted = decrypt_bytes(encrypted, password)
return decrypted == test_data
def main():
parser = argparse.ArgumentParser(description="AES-256-GCM File Encryption Tool")
subparsers = parser.add_subparsers(dest="command", help="Command to execute")
# Encrypt command
enc = subparsers.add_parser("encrypt", help="Encrypt a file")
enc.add_argument("--input", "-i", required=True, help="Input file path")
enc.add_argument("--output", "-o", required=True, help="Output file path")
enc.add_argument("--password", "-p", required=True, help="Encryption password")
# Decrypt command
dec = subparsers.add_parser("decrypt", help="Decrypt a file")
dec.add_argument("--input", "-i", required=True, help="Input file path")
dec.add_argument("--output", "-o", required=True, help="Output file path")
dec.add_argument("--password", "-p", required=True, help="Decryption password")
# Encrypt directory command
encdir = subparsers.add_parser("encrypt-dir", help="Encrypt a directory")
encdir.add_argument("--input", "-i", required=True, help="Input directory path")
encdir.add_argument("--output", "-o", required=True, help="Output directory path")
encdir.add_argument("--password", "-p", required=True, help="Encryption password")
# Decrypt directory command
decdir = subparsers.add_parser("decrypt-dir", help="Decrypt a directory")
decdir.add_argument("--input", "-i", required=True, help="Input directory path")
decdir.add_argument("--output", "-o", required=True, help="Output directory path")
decdir.add_argument("--password", "-p", required=True, help="Decryption password")
# Verify command
subparsers.add_parser("verify", help="Run roundtrip verification test")
args = parser.parse_args()
if args.command == "encrypt":
result = encrypt_file(args.input, args.output, args.password)
print(json.dumps(result, indent=2))
elif args.command == "decrypt":
result = decrypt_file(args.input, args.output, args.password)
print(json.dumps(result, indent=2))
elif args.command == "encrypt-dir":
result = encrypt_directory(args.input, args.output, args.password)
print(json.dumps(result, indent=2))
elif args.command == "decrypt-dir":
result = decrypt_directory(args.input, args.output, args.password)
print(json.dumps(result, indent=2))
elif args.command == "verify":
test_data = b"The quick brown fox jumps over the lazy dog. " * 100
password = "test_password_123!"
success = verify_roundtrip(test_data, password)
print(f"Roundtrip verification: {'PASSED' if success else 'FAILED'}")
if not success:
sys.exit(1)
else:
parser.print_help()
if __name__ == "__main__":
main()