Void Banshee Exploits Zombie Internet Explorer in Zero-Day Attacks
July 19, 2024
How to setup a IoC local cache with Maltiverse & Redis
Threat Intelligence (TI) is a vital component in cybersecurity operations, allowing organizations to detect, analyze, and mitigate threats. Indicators of Compromise (IoCs) such as malicious IPs, Hostnames, URLs, or file hashes play a central role in identifying ongoing cyberattacks. However, efficiently managing and processing these large IoC datasets can be challenging, especially in environments that demand high-frequency querying and real-time access.
To address these challenges, Maltiverse offers a powerful integration with Redis—a high-performance, in-memory data store ideal for caching IoCs in local environments. In this post, we’ll guide you through setting up Redis to cache IoCs from Maltiverse, helping you optimize your cybersecurity infrastructure for handling massive volumes of requests.
Redis (Remote Dictionary Server) is an open-source, in-memory key-value store known for its blazing-fast speed and scalability. By using Redis as a caching database for threat intelligence data, organizations can drastically reduce query times while ensuring data remains fresh through Time-to-Live (TTL) functionality, which automatically purges outdated IoCs.
With Maltiverse’s Redis connector, you can easily sync IoC feeds directly to your Redis instance, benefiting from:
- TTL-based data rotation: Automatically cleans up outdated IoCs after 30 days.
- Unlimited local requests: Ideal for platforms and systems that need to process high-frequency CTI requests.
- Performance optimization: Redis stores data in memory, enabling rapid response times for query-heavy environments.
Why Use Redis for Threat Intelligence Caching?
Step 1: Install Redis
Redis can be installed either through a traditional installation on Linux or using Docker, depending on your setup and preference. Below, we outline both methods.
Option 1: Installing Redis via Docker
Follow the instructions provided here https://redis.io/kb/doc/1hcec8xg9w/how-can-i-install-redis-on-docker
Option 2: Installing Redis on Linux
Follow the instructions provided here https://redis.io/docs/latest/operate/oss_and_stack/install/install-redis/install-redis-on-linux/
Step 2: Download the Maltiverse Redis Connector
First, download the Maltiverse Redis Connector script from the official Maltiverse GitHub repository. This script will help you load and synchronize IoC feeds with your Redis instance and can be placed on the same machine where Redis is running:
git clone https://github.com/maltiverse/maltiverse-feed-redis-connector
Once downloaded, place the script in a location on your local environment where it has network reachability to your Redis instance.
Step 3: Execute Initial Sync
With the Redis Connector script in place, the next step is to sync the IoC feed to Redis. You can perform an initial sync by running the script with the appropriate parameters.
Here’s a list ofcommands that syncs the entire content of the most important feeds in Maltiverse to your Redis instance:
python3 maltiverse-redis.py --redis_host localhost --redis_port 6379 --maltiverse_email EMAIL --maltiverse_password PASSWORD --feed uYxZknEB8jmkCY9eQoUJ
python3 maltiverse-redis.py --redis_host localhost --redis_port 6379 --maltiverse_email EMAIL --maltiverse_password PASSWORD --feed xKWKangBN4Q8MD8oRYd-
python3 maltiverse-redis.py --redis_host localhost --redis_port 6379 --maltiverse_email EMAIL --maltiverse_password PASSWORD --feed H4yrknEB8jmkCY9eb4aN
python3 maltiverse-redis.py --redis_host localhost --redis_port 6379 --maltiverse_email EMAIL --maltiverse_password PASSWORD --feed EIAO4HAB8jmkCY9e8HoL
python3 maltiverse-redis.py --redis_host localhost --redis_port 6379 --maltiverse_email EMAIL --maltiverse_password PASSWORD --feed WZ0XJHIB8jmkCY9eLpr0
Make sure to:
- Replace
localhostand6379with your Redis hostname and port if they differ. - Input your Maltiverse credentials (
EMAILandPASSWORD). - Specify the feed ID (
VdhZV34B4jHUXfKt_gDiin this example) for the data source.
Note: The Redis TTL is set to 30 days by default, meaning that old data will automatically be removed after this period.
Step 4: Set Up Hourly Incremental Updates
To ensure your IoC data remains up-to-date, it’s crucial to synchronize the Redis database with incremental updates from Maltiverse. You can achieve this by scheduling hourly updates using a cron job (in Linux) or an equivalent scheduler on your operating system.
Here’s how to set it up on a Linux machine:
Create a Shell Script
Create a shell script in the directory /etc/cron.hourly named maltiverse-redis.sh that runs the Redis sync script with the --range parameter, ensuring only the last hour’s updates are fetched:
#!/bin/bash
/usr/bin/python3 /path/to/your/script/maltiverse-redis.py --redis_host localhost --redis_port 6379 --maltiverse_email EMAIL --maltiverse_password PASSWORD --range now-1h --feed VdhZV34B4jHUXfKt_gDi
/usr/bin/python3 /path/to/your/script/maltiverse-redis.py --redis_host localhost --redis_port 6379 --maltiverse_email EMAIL --maltiverse_password PASSWORD --range now-1h --feed xKWKangBN4Q8MD8oRYd-
/usr/bin/python3 /path/to/your/script/maltiverse-redis.py --redis_host localhost --redis_port 6379 --maltiverse_email EMAIL --maltiverse_password PASSWORD --range now-1h --feed H4yrknEB8jmkCY9eb4aN
/usr/bin/python3 /path/to/your/script/maltiverse-redis.py --redis_host localhost --redis_port 6379 --maltiverse_email EMAIL --maltiverse_password PASSWORD --range now-1h --feed EIAO4HAB8jmkCY9e8HoL
/usr/bin/python3 /path/to/your/script/maltiverse-redis.py --redis_host localhost --redis_port 6379 --maltiverse_email EMAIL --maltiverse_password PASSWORD --range now-1h --feed WZ0XJHIB8jmkCY9eLpr0
Make sure to:
- Replace
/path/to/your/scriptto the rigth path. - Replace
localhostand6379with your Redis hostname and port if they differ. - Input your Maltiverse credentials (
EMAILandPASSWORD). - Specify the feed ID (
VdhZV34B4jHUXfKt_gDiin this example) for the data source.
Make the script executable
Once you’ve created the shell script, give it executable permissions:
sudo chmod +x /etc/cron.hourly/maltiverse-redis.sh
Schedule the Script to Run Hourly
Move the script to the /etc/cron.hourly/ directory, ensuring it runs once every hour to keep the IoC data in Redis up-to-date.
Step 5: Use the cache
Once your Redis cache is set up and synchronized with the IoC feeds from Maltiverse, you can easily query the data using any coding language that can connect to Redis with a library. Redis stores different types of IoCs in different databases:
- DB 0: IP addresses
- DB 1: Hostnames
- DB 2: URLs (stored as SHA-256 hashes)
- DB 4: File hashes
The IoC itself is stored as the key, while the value is a string containing the IoC details in JSON format. In this step, we will create a Python class to query any type of IoC and convert the JSON string into a Python dictionary.
Example Python Script to Query Redis for an IP Address
Here you can find a Python3 class that encapsulates methods to request of IoCs in our local cache. This class will provide methods to retrieve IP addresses, hostnames, URLs (using SHA-256 hashes), and file hashes from their respective Redis databases.
import ast
import redis
import json
import hashlib
class RedisIoCQuery:
def __init__(self, host='localhost', port=6379):
"""Initialize the Redis connection"""
self.redis_ip = redis.StrictRedis(host=host, port=port, db=0, decode_responses=True)
self.redis_hostname = redis.StrictRedis(host=host, port=port, db=1, decode_responses=True)
self.redis_url = redis.StrictRedis(host=host, port=port, db=2, decode_responses=True)
self.redis_hash = redis.StrictRedis(host=host, port=port, db=4, decode_responses=True)
def query_ip(self, ip):
"""Query Redis for an IP address stored in DB 0 and return it as a dictionary"""
return self._query_redis(self.redis_ip, ip)
def query_hostname(self, hostname):
"""Query Redis for a hostname stored in DB 1 and return it as a dictionary"""
return self._query_redis(self.redis_hostname, hostname)
def query_url(self, url):
"""Query Redis for a URL stored in DB 2 by its SHA-256 hash and return it as a dictionary"""
# Hash the URL using SHA-256
url_hash = hashlib.sha256(url.encode('utf-8')).hexdigest()
return self._query_redis(self.redis_url, url_hash)
def query_file_hash(self, file_hash):
"""Query Redis for a file hash stored in DB 4 and return it as a dictionary"""
return self._query_redis(self.redis_hash, file_hash)
def _query_redis(self, redis_db, key):
"""Helper method to query Redis and convert the result to a dictionary"""
ioc_data = redis_db.get(key)
if ioc_data:
return ast.literal_eval(ioc_data)
else:
return None
# Example Usage:
if __name__ == "__main__":
redis_ioc_query = RedisIoCQuery()
# Example IP
ip_address = "217.77.220.234"
ip_result = redis_ioc_query.query_ip(ip_address)
if ip_result:
print("IoC Data for IP", ip_address, ":", json.dumps(ip_result, indent=4))
else:
print(f"No data found for IP {ip_address}.")
# Example Hostname
hostname = "example.com"
hostname_result = redis_ioc_query.query_hostname(hostname)
if hostname_result:
print("IoC Data for Hostname", hostname, ":", json.dumps(hostname_result, indent=4))
else:
print(f"No data found for hostname {hostname}.")
# Example URL
url = "http://rigagrindex.club/"
url_result = redis_ioc_query.query_url(url)
if url_result:
print("IoC Data for URL", url, ":", json.dumps(url_result, indent=4))
else:
print(f"No data found for URL {url}.")
# Example File Hash
file_hash = "275a021bbfb6489e54d471899f7db9d1663fc695ec2fe2a2c4538aabf651fd0f"
file_hash_result = redis_ioc_query.query_file_hash(file_hash)
if file_hash_result:
print("IoC Data for File Hash", file_hash, ":", json.dumps(file_hash_result, indent=4))
else:
print(f"No data found for File Hash {file_hash}.")
Explanation of the Class:
-
__init__(self, host='localhost', port=6379'): Initializes connections to four separate Redis databases (db=0for IPs,db=1for hostnames,db=2for URLs, anddb=4for file hashes). You can modifyhostandportas needed. -
query_ip(self, ip): Queries the IP address in DB 0 and returns the JSON data as a dictionary. -
query_hostname(self, hostname): Queries the hostname in DB 1 and returns the result as a dictionary. -
query_url(self, url): First, it hashes the URL using SHA-256 and then queries the corresponding hash in DB 2 to get the data. -
query_file_hash(self, file_hash): Queries the file hash (e.g., MD5, SHA-1, SHA-256) in DB 3 and returns the JSON data as a dictionary. -
_query_redis(self, redis_db, key): A helper method that queries the specified Redis database for a given key (IoC) and converts the resulting JSON string to a Python dictionary. It’s used by all other query methods.
Understand the data model
To understand the different fields in the JSON response please check the API Specification where data models are explained
Conclusion
Setting up a Redis cache for IoCs with Maltiverse enables your CTI systems to handle massive amounts of threat intelligence requests with maximum efficiency. With Redis’ TTL capabilities, you can be assured that outdated IoCs are automatically rotated out, while the Maltiverse connector keeps your data synchronized in real-time.
By following the steps outlined in this article, you’ll have a robust caching solution in place, ready to support the demands of modern cybersecurity operations. Don’t hesitate to explore other use cases where Redis can bring value to your threat intelligence stack!
