Cloudgrep - Blackhat Aresenal - cloudgrep searches cloud storage

cloudgrep
cloudgrep searches cloud storage.

ci License

It currently supports searching log files, optionally compressed with gzip (.gz) or zip (.zip), in AWS S3, Azure Storage or Google Cloud Storage.

Diagram

Why?
Directly searching cloud storage, without indexing logs into a SIEM or Log Analysis tool, can be faster and cheaper.
There is no need to wait for logs to be ingested, indexed, and made available for searching.
It searches files in parallel for speed.
This may be of use when debugging applications, or investigating a security incident.
Example
Simple example:

./cloudgrep --bucket test-s3-access-logs --query 9RXXKPREHHTFQD77
python3 cloudgrep.py -b test-s3-access-logs -q 9RXXKPREHHTFQD77
Simple Azure example:

python3 cloudgrep.py -an some_account -cn some_container -q my_search
Simple Google example:

python3 cloudgrep.py -gb my-gcp-bucket -q my_search
Simple CloudTrail log example, outputting results as JSON:

python3 cloudgrep.py -b test-s3-access-logs -q 9RXXKPREHHTFQD77 -lt cloudtrail -jo
Simple custom log example:

python3 cloudgrep.py -b test-s3-access-logs -q 9RXXKPREHHTFQD77 -lf json -lp Records
More complicated example:

python3 cloudgrep.py -b test-s3-access-logs --prefix "logs/" --filename ".log" -q 9RXXKPREHHTFQD77 -s "2023-01-09 20:30:00" -e "2023-01-09 20:45:00" --file_size 10000 --debug
Saving the output to a file:

python3 cloudgrep.py -b test-s3-access-logs -q 9RXXKPREHHTFQD77 --hide_filenames > matching_events.log
Example output:

[2023-11-30 13:37:12,416] - Bucket is in region: us-east-2 : Search from the same region to avoid egress charges.
[2023-11-30 13:37:12,417] - Searching 11 files in test-s3-access-logs for 9RXXKPREHHTFQD77...
{"key_name": "access2023-01-09-20-34-20-EAC533CB93B4ACBE", "line": "abbd82b5ad5dc5d024cd1841d19c0cf2fd7472c47a1501ececde37fe91adc510 bucket-72561-s3bucketalt-1my9piwesfim7 [09/Jan/2023:19:20:00 +0000] 1.125.222.333 arn:aws:sts::000011110470:assumed-role/bucket-72561-myResponseRole-1WP2IOKDV7B4Y/1673265251.340187 9RXXKPREHHTFQD77 REST.GET.BUCKET - \"GET /?list-type=2&prefix=-collector%2Fproject-&start-after=&encoding-type=url HTTP/1.1\" 200 - 946 - 33 32 \"-\" \"Boto3/1.21.24 Python/3.9.2 Linux/5.10.0-10-cloud-amd64 Botocore/1.24.46\" - aNPuHKw== SigV4 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader bucket-72561-s3bucketalt-1my9piwesfim7.s3.us-east-2.amazonaws.com TLSv1.2 - -"}
Arguments
usage: cloudgrep.py [-h] [-b BUCKET] [-an ACCOUNT_NAME] [-cn CONTAINER_NAME] [-gb GOOGLE_BUCKET] [-q QUERY]
[-v FILE] [-y YARA] [-p PREFIX] [-f FILENAME] [-s START_DATE] [-e END_DATE]
[-fs FILE_SIZE] [-pr PROFILE] [-d] [-hf] [-lt LOG_TYPE] [-lf LOG_FORMAT]
[-lp LOG_PROPERTIES] [-jo JSON_OUTPUT]

CloudGrep searches is grep for cloud storage like S3 and Azure Storage. Version: 1.0.5

options:
-h, --help show this help message and exit
-b BUCKET, --bucket BUCKET
AWS S3 Bucket to search. E.g. my-bucket
-an ACCOUNT_N

1. #BHUSA   @BlackHatEvents
cloudgrep
cloudgrep is grep for cloud storage


https://github.com/cado-security/cloudgrep
https://x.com/cadosecurity

2. # BHUSA   @BlackHatEvents
Introducing clougrep

3. # BHUSA   @BlackHatEvents
So you’ve got an incident in the cloud…
Cloud
(Control Plane)
Forensics
Host Forensics

Log Forensics Network Forensics

4. # BHUSA   @BlackHatEvents
DFIR in the Cloud == Log Analysis?

5. # BHUSA   @BlackHatEvents
Time
●Do you need to ship the logs first?
●Do you need to wait for the logs to be indexed?
●Do you know where the logs are stored?

Cost
●Can you pay to index and store all the data?
●Hot search vs cold data
●Egress fees


Log Analysis in The Cloud -
Challenges

6. # BHUSA   @BlackHatEvents
If the data is already in your SIEM and
you're in a position to do this - that's
fantastic. You can stop now :)

But they can also be expensive, and
you need to get the data there.
Modern SIEM’s often use a cheaper
“indexless” approach to search larger
amounts of data at less cost.
Or they even keep logs “in place” to
reduce the cost of moving them.

SIEM Systems

7. # BHUSA   @BlackHatEvents
Athena for parsing and searching logs in
S3
CREATE EXTERNAL TABLE IF NOT EXISTS elb_logs_raw_native (
request_timestamp string,
request_ip string,
request_port int,
backend_ip string,
backend_port int, …)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
'serialization.format' = '1','input.regex' = '([^ ]*) ... ([A-Za-z0-9.-]*)$' )
LOCATION 's3://athena-examples/elb/raw/' ;
SELECT * FROM elb_logs_raw_native WHERE elb_response_code = '200' LIMIT 100;
See also: Automated First-Response in AWS using Sigma and
Athena Invictus Incident Response Invictus Incident Response

8. # BHUSA   @BlackHatEvents
●CloudWatch
●CloudSearch
●AWS OpenSearch
●… more
A million ways to search logs in AWS

9. # BHUSA   @BlackHatEvents
I just want to grep my cloud files…
grep is a command-line utility for searching plaintext datasets for lines that match a regular
expression. Its name comes from the ed command g/re/p
Mountpoint
https://aws.amazon.com/s3/features/mountpoint/
aws s3 cp s3://bucket/log-folder/ .
--recursive
zgrep "evil" *
mount-s3 DOC-EXAMPLE-BUCKET /path/to/mount
AWS CLI

10. # BHUSA   @BlackHatEvents
Introducing clougrep

11. # BHUSA   @BlackHatEvents
Overview of cloudgrep

12. # BHUSA   @BlackHatEvents
●Simple installation and operation
●No indexing required
●Operates across aws, azure and gcp
●Files are downloaded and searched in parallel for speed
●Low cost operation (see later)
●Able to run detection rules via regex or Yara for file
content
●Can search compressed files
●Can search binary files
Why cloudgrep
MZ@!L!This program cannot be run in DOS mode.


$PEL8Q"
0 @
~`NOH
H.text `.rsrcH@@.reloc

13. # BHUSA   @BlackHatEvents
git clone https://github.com/cado-security/cloudgrep.git
pip3 install -r requirements.txt

AWS
~/.aws/credentials or instance profile
Use a VPC endpoint to S3 on an EC2 to avoid egress charges

Azure
az login

GCP
export
GOOGLE_APPLICATION_CREDENTIALS="/Users/creds.json"

Setup

14. # BHUSA   @BlackHatEvents
Demo

15. # BHUSA   @BlackHatEvents
Demo
# Basic search
python3 cloudgrep.py -b blackhat-cado-demo -q "ssm.amazonaws.com"
# Debug mode
python3 cloudgrep.py -b blackhat-cado-demo -q "ssm.amazonaws.com" --debug
# Azure
python3 cloudgrep.py -an some_account -cn some_container -q my_search
# GCP
python3 cloudgrep.py -gb my-gcp-bucket -q my_search
# Json parsing and output
python3 cloudgrep.py -b blackhat-cado-demo -q "2020-07-31T23:58:37Z" -lt cloudtrail -jo
# --prefix is a *fast* filter
# Start and end date are a *fast* filter
# --filename is a *slow* filter
python3 cloudgrep.py -b blackhat-cado-demo -q "2024-01-02T02:51:00Z" -s "2023-01-09 20:30:00" -e
"2025-01-09 20:45:00" --file_size 100
# Search a number of queries
python3 cloudgrep.py -b blackhat-cado-demo --file queries.txt
# Search files with a Yara rule
python3 cloudgrep.py -b blackhat-cado-demo --yara yara.rule

16. # BHUSA   @BlackHatEvents
Thank-you for all the community contributions!

Particular shout out to Daniel Bohannon, Andi
Ahmeti and the rest of the team @ Permiso for:
-Log parsing support
-JSON output

Also see their tool Cloud Grappler for threat
detection against logs in cloud storage, using
cloudgrep as the search library.
Community Contributions

17. #BHUSA   @BlackHatEvents
cloudgrep
cloudgrep is grep for cloud storage


https://github.com/cado-security/cloudgrep
https://x.com/cadosecurity