Threat Intelligence (TI) is a vital component in cybersecurity operations, allowing organizations to detect, analyze, and mitigate threats. Indicators of Compromise (IoCs) such as malicious IPs, Hostnames, URLs, or file hashes play a central role in identifying ongoing cyberattacks. However, efficiently managing and processing these large IoC datasets can be challenging, especially in environments that demand high-frequency querying and real-time access.
To address these challenges, Maltiverse offers a powerful integration with Redis—a high-performance, in-memory data store ideal for caching IoCs in local environments. In this post, we’ll guide you through setting up Redis to cache IoCs from Maltiverse, helping you optimize your cybersecurity infrastructure for handling massive volumes of requests.
Redis (Remote Dictionary Server) is an open-source, in-memory key-value store known for its blazing-fast speed and scalability. By using Redis as a caching database for threat intelligence data, organizations can drastically reduce query times while ensuring data remains fresh through Time-to-Live (TTL) functionality, which automatically purges outdated IoCs.
With Maltiverse’s Redis connector, you can easily sync IoC feeds directly to your Redis instance, benefiting from:
Redis can be installed either through a traditional installation on Linux or using Docker, depending on your setup and preference. Below, we outline both methods.
Follow the instructions provided here https://redis.io/kb/doc/1hcec8xg9w/how-can-i-install-redis-on-docker
Follow the instructions provided here https://redis.io/docs/latest/operate/oss_and_stack/install/install-redis/install-redis-on-linux/
First, download the Maltiverse Redis Connector script from the official Maltiverse GitHub repository. This script will help you load and synchronize IoC feeds with your Redis instance and can be placed on the same machine where Redis is running:
git clone https://github.com/maltiverse/maltiverse-feed-redis-connector
Once downloaded, place the script in a location on your local environment where it has network reachability to your Redis instance.
With the Redis Connector script in place, the next step is to sync the IoC feed to Redis. You can perform an initial sync by running the script with the appropriate parameters.
Here’s a list ofcommands that syncs the entire content of the most important feeds in Maltiverse to your Redis instance:
python3 maltiverse-redis.py --redis_host localhost --redis_port 6379 --maltiverse_email EMAIL --maltiverse_password PASSWORD --feed uYxZknEB8jmkCY9eQoUJ
python3 maltiverse-redis.py --redis_host localhost --redis_port 6379 --maltiverse_email EMAIL --maltiverse_password PASSWORD --feed xKWKangBN4Q8MD8oRYd-
python3 maltiverse-redis.py --redis_host localhost --redis_port 6379 --maltiverse_email EMAIL --maltiverse_password PASSWORD --feed H4yrknEB8jmkCY9eb4aN
python3 maltiverse-redis.py --redis_host localhost --redis_port 6379 --maltiverse_email EMAIL --maltiverse_password PASSWORD --feed EIAO4HAB8jmkCY9e8HoL
python3 maltiverse-redis.py --redis_host localhost --redis_port 6379 --maltiverse_email EMAIL --maltiverse_password PASSWORD --feed WZ0XJHIB8jmkCY9eLpr0
Make sure to:
localhost
and 6379
with your Redis hostname and port if they differ.EMAIL
and PASSWORD
).VdhZV34B4jHUXfKt_gDi
in this example) for the data source.Note: The Redis TTL is set to 30 days by default, meaning that old data will automatically be removed after this period.
To ensure your IoC data remains up-to-date, it’s crucial to synchronize the Redis database with incremental updates from Maltiverse. You can achieve this by scheduling hourly updates using a cron job (in Linux) or an equivalent scheduler on your operating system.
Create a shell script in the directory /etc/cron.hourly
named maltiverse-redis.sh
that runs the Redis sync script with the --range
parameter, ensuring only the last hour’s updates are fetched:
#!/bin/bash
/usr/bin/python3 /path/to/your/script/maltiverse-redis.py --redis_host localhost --redis_port 6379 --maltiverse_email EMAIL --maltiverse_password PASSWORD --range now-1h --feed VdhZV34B4jHUXfKt_gDi
/usr/bin/python3 /path/to/your/script/maltiverse-redis.py --redis_host localhost --redis_port 6379 --maltiverse_email EMAIL --maltiverse_password PASSWORD --range now-1h --feed xKWKangBN4Q8MD8oRYd-
/usr/bin/python3 /path/to/your/script/maltiverse-redis.py --redis_host localhost --redis_port 6379 --maltiverse_email EMAIL --maltiverse_password PASSWORD --range now-1h --feed H4yrknEB8jmkCY9eb4aN
/usr/bin/python3 /path/to/your/script/maltiverse-redis.py --redis_host localhost --redis_port 6379 --maltiverse_email EMAIL --maltiverse_password PASSWORD --range now-1h --feed EIAO4HAB8jmkCY9e8HoL
/usr/bin/python3 /path/to/your/script/maltiverse-redis.py --redis_host localhost --redis_port 6379 --maltiverse_email EMAIL --maltiverse_password PASSWORD --range now-1h --feed WZ0XJHIB8jmkCY9eLpr0
Make sure to:
/path/to/your/script
to the rigth path.localhost
and 6379
with your Redis hostname and port if they differ.EMAIL
and PASSWORD
).VdhZV34B4jHUXfKt_gDi
in this example) for the data source.Once you’ve created the shell script, give it executable permissions:
sudo chmod +x /etc/cron.hourly/maltiverse-redis.sh
Move the script to the /etc/cron.hourly/
directory, ensuring it runs once every hour to keep the IoC data in Redis up-to-date.
Once your Redis cache is set up and synchronized with the IoC feeds from Maltiverse, you can easily query the data using any coding language that can connect to Redis with a library. Redis stores different types of IoCs in different databases:
The IoC itself is stored as the key, while the value is a string containing the IoC details in JSON format. In this step, we will create a Python class to query any type of IoC and convert the JSON string into a Python dictionary.
Here you can find a Python3 class that encapsulates methods to request of IoCs in our local cache. This class will provide methods to retrieve IP addresses, hostnames, URLs (using SHA-256 hashes), and file hashes from their respective Redis databases.
import ast
import redis
import json
import hashlib
class RedisIoCQuery:
def __init__(self, host='localhost', port=6379):
"""Initialize the Redis connection"""
self.redis_ip = redis.StrictRedis(host=host, port=port, db=0, decode_responses=True)
self.redis_hostname = redis.StrictRedis(host=host, port=port, db=1, decode_responses=True)
self.redis_url = redis.StrictRedis(host=host, port=port, db=2, decode_responses=True)
self.redis_hash = redis.StrictRedis(host=host, port=port, db=4, decode_responses=True)
def query_ip(self, ip):
"""Query Redis for an IP address stored in DB 0 and return it as a dictionary"""
return self._query_redis(self.redis_ip, ip)
def query_hostname(self, hostname):
"""Query Redis for a hostname stored in DB 1 and return it as a dictionary"""
return self._query_redis(self.redis_hostname, hostname)
def query_url(self, url):
"""Query Redis for a URL stored in DB 2 by its SHA-256 hash and return it as a dictionary"""
# Hash the URL using SHA-256
url_hash = hashlib.sha256(url.encode('utf-8')).hexdigest()
return self._query_redis(self.redis_url, url_hash)
def query_file_hash(self, file_hash):
"""Query Redis for a file hash stored in DB 4 and return it as a dictionary"""
return self._query_redis(self.redis_hash, file_hash)
def _query_redis(self, redis_db, key):
"""Helper method to query Redis and convert the result to a dictionary"""
ioc_data = redis_db.get(key)
if ioc_data:
return ast.literal_eval(ioc_data)
else:
return None
# Example Usage:
if __name__ == "__main__":
redis_ioc_query = RedisIoCQuery()
# Example IP
ip_address = "217.77.220.234"
ip_result = redis_ioc_query.query_ip(ip_address)
if ip_result:
print("IoC Data for IP", ip_address, ":", json.dumps(ip_result, indent=4))
else:
print(f"No data found for IP {ip_address}.")
# Example Hostname
hostname = "example.com"
hostname_result = redis_ioc_query.query_hostname(hostname)
if hostname_result:
print("IoC Data for Hostname", hostname, ":", json.dumps(hostname_result, indent=4))
else:
print(f"No data found for hostname {hostname}.")
# Example URL
url = "http://rigagrindex.club/"
url_result = redis_ioc_query.query_url(url)
if url_result:
print("IoC Data for URL", url, ":", json.dumps(url_result, indent=4))
else:
print(f"No data found for URL {url}.")
# Example File Hash
file_hash = "275a021bbfb6489e54d471899f7db9d1663fc695ec2fe2a2c4538aabf651fd0f"
file_hash_result = redis_ioc_query.query_file_hash(file_hash)
if file_hash_result:
print("IoC Data for File Hash", file_hash, ":", json.dumps(file_hash_result, indent=4))
else:
print(f"No data found for File Hash {file_hash}.")
__init__(self, host='localhost', port=6379')
: Initializes connections to four separate Redis databases (db=0
for IPs, db=1
for hostnames, db=2
for URLs, and db=4
for file hashes). You can modify host
and port
as needed.
query_ip(self, ip)
: Queries the IP address in DB 0 and returns the JSON data as a dictionary.
query_hostname(self, hostname)
: Queries the hostname in DB 1 and returns the result as a dictionary.
query_url(self, url)
: First, it hashes the URL using SHA-256 and then queries the corresponding hash in DB 2 to get the data.
query_file_hash(self, file_hash)
: Queries the file hash (e.g., MD5, SHA-1, SHA-256) in DB 4 and returns the JSON data as a dictionary.
_query_redis(self, redis_db, key)
: A helper method that queries the specified Redis database for a given key (IoC) and converts the resulting JSON string to a Python dictionary. It’s used by all other query methods.
To understand the different fields in the JSON response please check the API Specification where data models are explained
Setting up a Redis cache for IoCs with Maltiverse enables your CTI systems to handle massive amounts of threat intelligence requests with maximum efficiency. With Redis’ TTL capabilities, you can be assured that outdated IoCs are automatically rotated out, while the Maltiverse connector keeps your data synchronized in real-time.
By following the steps outlined in this article, you’ll have a robust caching solution in place, ready to support the demands of modern cybersecurity operations. Don’t hesitate to explore other use cases where Redis can bring value to your threat intelligence stack!
Cookie | Duration | Description |
---|---|---|
cookielawinfo-checkbox-analytics | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Analytics". |
cookielawinfo-checkbox-functional | 11 months | The cookie is set by GDPR cookie consent to record the user consent for the cookies in the category "Functional". |
cookielawinfo-checkbox-necessary | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookies is used to store the user consent for the cookies in the category "Necessary". |
cookielawinfo-checkbox-others | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Other. |
cookielawinfo-checkbox-performance | 11 months | This cookie is set by GDPR Cookie Consent plugin. The cookie is used to store the user consent for the cookies in the category "Performance". |
viewed_cookie_policy | 11 months | The cookie is set by the GDPR Cookie Consent plugin and is used to store whether or not user has consented to the use of cookies. It does not store any personal data. |