Why Bots Target wlwmanifest.xml and How to Mitigate Unauthorized Access

The internet is a vast ecosystem where legitimate users and malicious actors coexist. One common issue that website administrators face is malicious crawling—automated bots scanning websites for vulnerabilities. A frequent target of these bots is WordPress installations, given its widespread use. Even if you don’t use WordPress, your server might still experience unusual HTTP requests targeting WordPress directories.

This article delves into why this happens and provides a comprehensive guide on how to mitigate such malicious activities using Cloudflare’s Super Bot Fight Mode.

The Issue: Unusual HTTP Requests Targeting WordPress Files

Example HTTP Logs

You might notice entries in your server logs similar to the following:

<span class="hljs-number">127.0.0.1</span> - - <span class="hljs-string">[02/Jun/2021 15:28:00]</span> <span class="hljs-string">"<span class="hljs-keyword">GET</span> //wp-includes/wlwmanifest.xml HTTP/1.0"</span> <span class="hljs-number">404</span> -
<span class="hljs-number">127.0.0.1</span> - - <span class="hljs-string">[02/Jun/2021 15:28:00]</span> <span class="hljs-string">"<span class="hljs-keyword">GET</span> //xmlrpc.php?rsd HTTP/1.0"</span> <span class="hljs-number">404</span> -
<span class="hljs-number">127.0.0.1</span> - - <span class="hljs-string">[02/Jun/2021 15:28:00]</span> <span class="hljs-string">"<span class="hljs-keyword">GET</span> / HTTP/1.0"</span> <span class="hljs-number">200</span> -
... (multiple similar requests)

These logs indicate that someone (or something) is attempting to access WordPress-specific files like wlwmanifest.xml, xmlrpc.php, and directories like /wp-includes/.

Understanding the Requests

• wlwmanifest.xml: Used by Windows Live Writer and other clients to interact with WordPress sites.
• xmlrpc.php: Enables remote connections to WordPress, often exploited for DDoS attacks or brute-force login attempts.
• Various WordPress Directories: Bots scan common paths to locate WordPress installations.

Why Is This Happening?

Prevalence of Malicious Bots

• Automated Scanning: Bots routinely scan IP addresses and domains for known vulnerabilities.
• WordPress Popularity: As WordPress powers over 40% of websites globally, it’s a prime target.
• Outdated Exploits: Some bots use outdated scripts, attempting exploits that may no longer be effective but are still tried en masse.

Goals of Malicious Actors

• Identify Vulnerabilities: Find outdated software versions to exploit known security flaws.
• Gain Unauthorized Access: Use vulnerabilities to control the server or website.
• Resource Exploitation: Leverage compromised servers for spamming, data theft, or further attacks.

Potential Risks Even Without WordPress Installed

• Server Load: Excessive requests can strain server resources, affecting performance.
• False Positives: Security systems might misinterpret these requests, triggering unnecessary alerts.
• Exposure of Other Vulnerabilities: While targeting WordPress files, bots might stumble upon other weaknesses.

Mitigating Malicious Crawling with Cloudflare’s Super Bot Fight Mode

Introducing Super Bot Fight Mode

Cloudflare’s Super Bot Fight Mode is a premium feature designed to protect websites from malicious bots by:

• Identifying Bot Traffic: Uses patterns and signatures to detect bots.
• Challenging or Blocking Bots: Automatically mitigates harmful requests.
• Protecting Static Resources: Shields assets like images, scripts, and stylesheets.
• Providing Analytics: Offers insights into bot traffic patterns.

Step-by-Step Guide to Enabling Super Bot Fight Mode

Prerequisites

• Cloudflare Account: Ensure you have a Pro subscription or higher.
• Access to Cloudflare Dashboard: Administrative privileges to modify security settings.

Enabling Super Bot Fight Mode

1. Log In to Cloudflare
   • Visit the Cloudflare Dashboard and log in to your account.
2. Select Your Domain
   • From your account overview, choose the website you wish to protect.
3. Navigate to Security Settings
   • In the dashboard, go to Security > Bots.
4. Configure Super Bot Fight Mode
   • Click on Configure Super Bot Fight Mode.
   • You will see options to set actions for different categories of traffic.
5. Set Actions for Traffic Categories
   • Definitely Automated: Set to Block or Challenge to prevent malicious bots.
     • Warning: If you use services like Cloudflare Tunnel, set this to Allow to avoid disruptions.
   • Verified Bots: Typically set to Allow to permit legitimate bots (e.g., search engine crawlers).
   • Static Resource Protection: Enable to protect static files from automated scraping.
   • JavaScript Detections: Turn on to inject invisible code that helps detect bots.
6. Enable Additional Protections
   • Block AI Bots: Toggle this option to prevent AI crawlers and scrapers from accessing content for training language models.
7. Save Your Configuration
   • Confirm your settings and save the changes.

Fine-Tuning with WAF Custom Rules

If certain parts of your site require bot traffic (e.g., APIs or webhooks), you can create custom rules:

1. Access WAF Custom Rules
   • In the Cloudflare dashboard, navigate to Security > WAF.
2. Create a New Rule
   • Click on Create a Firewall Rule.
3. Define the Rule Criteria
   • Set conditions to Skip Super Bot Fight Mode for specific URLs or parameters.
4. Deploy the Rule
   • Save and deploy the rule to apply the exceptions.

Verifying Your Setup

1. Monitor Security Events
   • Go to Security > Events to view recent actions taken against bot traffic.
   • Look for entries labeled Super Bot Fight Mode.
2. Review Bot Analytics
   • In Security > Bots, review the Bot Report to understand traffic patterns.

Additional Security Recommendations

• Regularly Update Software: Keep all server software and applications up to date.
• Use Strong Authentication: Implement multi-factor authentication for administrative access.
• Employ Rate Limiting: Limit the number of requests allowed from a single IP address.
• Implement a Web Application Firewall (WAF): Adds an additional layer of security against web exploits.

Conclusion

Malicious bots probing for vulnerabilities are an unfortunate reality of managing a website today. Understanding these threats is the first step toward mitigation. By leveraging Cloudflare’s Super Bot Fight Mode, you can substantially reduce unwanted bot traffic, safeguarding your server’s resources and enhancing security. Regular monitoring and proactive security measures will help ensure your website remains resilient against ongoing and evolving threats.

Frequently Asked Questions

Q1: Will Super Bot Fight Mode affect legitimate traffic to my site?

A: When configured correctly, Super Bot Fight Mode distinguishes between malicious bots and legitimate users or good bots (like search engine crawlers). Ensure that Verified Bots are set to Allow to prevent blocking beneficial traffic.

Q2: Do I need to make changes to my server after enabling Super Bot Fight Mode?

A: No server-side changes are necessary. Super Bot Fight Mode works at the Cloudflare edge network, filtering traffic before it reaches your server.

Q3: Can I use Super Bot Fight Mode on the free Cloudflare plan?

A: Super Bot Fight Mode with advanced features is available on Pro and higher-tier plans. The free plan includes basic bot mitigation, but for comprehensive protection, consider upgrading.

By proactively addressing malicious crawling attempts, you not only protect your website but also contribute to a safer internet ecosystem. Stay vigilant and make use of the tools available to you for optimal security.