Substantial Upgrades to Crawling History, Artifact Collection

Substantial Upgrades to Crawling History, Artifact Collection

At Validin, we believe cybersecurity threat intelligence teams, including threat hunters, analysts, and researchers, deserve every edge against continuously evolving adversaries.

One capability our customers consistently mention is the importance of HTTP/S banners and host response features in identifying adversary behaviors. Based on this feedback, we’ve accelerated our roadmap to offer advanced HTTPS banner analysis and historical response data, bolstering your threat hunting and incident response workflows.

Today, we’re excited to roll out two significant upgrades to support your cybersecurity threat intel operations:

• 8+ months of historical HTTPS banners and virtual host responses: Over 100 billion banner responses indexed and searchable. That’s access to well over 100 billion HTTP/S banners collected on virtual hosts (using SNI for certificate negotiation and a Host: header in the HTTP request), complete with feature pivoting across the entire dataset, going back to early November 2024 as of today.

• On-demand artifact access: We provide complete HTTP responses, favicon, and certificate artifacts we observe while crawling.

With these enhancements, Validin enables analysts to:

• Monitor changes in malicious domain/IP behavior across time
• Observe pre- and post-weaponization of malicious infrastructure
• Track behaviors
• Identify operational security (OPSEC) mistakes that can tie clusters of indicators together
• Pivot across the full context of responses for reporting and further analysis

Greatly Expanded Host Response History

A detailed host response history provides a variety of possibilities for forensic investigation and retrospective threat hunting. With a comprehensive history of virtual host responses, threat hunters can understand the changes and lifecycle of adversarial infrastructure, helping to answer common questions like:

• When was a domain first active?
• Was there a delay between activation and weaponization?
• How long was a domain weaponized?
• Are there unique patterns in the setup stages, timing, and content that can identify similar domains and IPs at other points in time?

These insights are invaluable for cybersecurity threat intelligence professionals performing malware infrastructure analysis, phishing domain tracking, and advanced digital forensics.

We understand these insights are very important for our customers, and so Validin continues its investment and commitment to bring them forward for research and hunting purposes.

Unique Challenges

At Validin, we process over 500,000 DNS RDATA records and around 10,000 host responses every second of every day on average (nearly 1 billion host responses per day).

We resolve over 5 billion fully qualified domain names (FQDNs), resolving DNS records for each of them at least daily, and actively requesting HTTP host responses on every domain name with a valid A record at least once every 2 weeks.

This volume of data introduces unique challenges around collecting, processing, maintaining, indexing, storing, and querying at scale.

To handle this volume of data, we:

• Created highly parallelized, bespoke data pipelines spanning five cloud service providers
• Developed a patented, cloud-agnostic database purpose-built for storing this volume of data (averaging tens of TB of inbound data each day) while maintaining point-in-time data fidelity
• Added over 200 TB of additional capacity to the low-latency database cluster for increased stability and reliability

Combined with our innovative cloud provider partners, we are in a unique position to provide the most complete internet infrastructure data and insights platform available on the market today.

What Validin Delivers

You can now search host response history for more than 8 months using:

• Domains, subdomains, and IP addresses
• HTTPS banner hashes and header values
• Favicon hashes and SSL certificate fingerprints
• Body content hashes and HTML meta tags

With more than 8 months of history, you can learn how indicators were configured and behaving well before many of them were publicly identified, uncovering new fingerprints and detections that might otherwise have been hidden.

To demonstrate this, we’ll walk you through an example highlighting new indicators that can be easily tracked with this new capability.

Use Case: More Indicators Related to Bybit Heist

On March 11, 2025, we used host response history in Validin to investigate the Bybit heist, attributed by the FBI to North Korea’s Lazarus Group as “TraderTraitor.”

Now, with expanded host response history, a new search reveals previously unknown infrastructure:

We noted that the header hash f4407a84d90c5ecc1025 combined with the HTML title tag value “404 Not Found” identified a highly-specific cluster of domain names and IP addresses that were likely used by this threat actor based on a number of features, including:

• Domain
• Subdomain
• host server
• host response
• registration similarities

However, at the time, we could only look back 3 months to December 2024.

Performing this same search today in the Validin Enterprise Edition platform, with detailed virtual host data to early November 2024, we’re able to identify an additional, previously unreported domain name and IP address that was likely associated with the TraderTraitor activity:

• en.stocksitem[.]org
• 2.56.10[.]90

On-Demand Full Response Artifacts

Our approach to saving content of host responses, extracting dozens of categories of features from responses, indexing them, and storing summarized output provides significant value from host responses by identifying features that can be compared and correlated across billions of responses.

As of July 1, 2025 for virtual host responses, and July 15, 2025 for IPv4 responses, Validin stores and serves full host response artifacts for both virtual hosts and IPv4 responses. Including:

• HTML responses
• Favicon files
• SSL/TLS certificates

We’ve made these artifacts available to Enterprise customers as both in-page and downloadable artifacts via API, indexed by the artifact hash. Additionally, we are not planning to age these artifacts out of our database, enabling retention and historic search capabilities.

Our patented database technology made this possible while also enabling a clear path for us to rapidly expand the scope and scale of the artifacts we collect.

See it in Action

On the Enterprise platform now, when opening the slideout, you’re presented with three new options:

• View Full HTML
• View Favicon
• View Full Certificate

Clicking the “View Full HTML” button loads a pop-over that displays the syntax-highlighted HTML response, the ability to copy the full response to your clipboard, and a download option.

Clicking the “View Favicon” option loads the favicon content in-place and provides an option to download the favicon file.

Clicking “View Full Certificate” opens a pop-over that shows a human-readable certificate, and provides options for copying and downloading both the human-readable certificate details and the raw PEM.

Conclusion

With these upgrades, Validin continues building the most robust cyber threat intelligence platform (TIP) for analysts. We want threat hunters to feel like they have an unfair advantage against cyber adversaries. These improvements provide a significant stepping stone toward our vision to provide the best possible solution and database for public infrastructure threat hunting.

Ready to elevate your threat hunting, threat attribution, and incident response efforts? Whether you’re an individual analyst or part of a larger enterprise team, Validin offers solutions that meet your needs. Individual users can create a free account and self-upgrade to access more advanced features and data.

Part of a team? Contact us today to explore our enterprise options and discover how Validin can provide your teams with powerful tools unparalleled data. Let Validin help you work smarter, faster, and more effectively in the fight against cyber threats.