GHSA-23j4-mw76-5v7h: 
Python vulnerability analysis and mitigation

Scrapy versions prior to 2.11.2 contained a vulnerability where the application followed redirects regardless of the URL protocol, affecting URLs using data://, file://, ftp://, s3://, and other schemes defined in the DOWNLOAD_HANDLERS setting. The vulnerability was discovered and reported by @mvsantos, and was patched in version 2.11.2. The issue has been assigned identifier GHSA-23j4-mw76-5v7h with a CVSS score of 6.5 (Moderate severity) (GitHub Advisory).

The vulnerability stems from Scrapy's redirect handling mechanism where HTTP redirects were not properly restricted to only http:// and https:// schemes. The issue affects both the RedirectMiddleware and MetaRefreshMiddleware components. The vulnerability has been assigned a CVSS v3.1 base score of 6.5 with vector string CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:N/A:N, indicating network attack vector, low attack complexity, low privileges required, no user interaction needed, unchanged scope, and high impact on confidentiality (GitHub Advisory).

A malicious actor with write access to spider start requests (e.g., ability to define start_urls) and read access to spider output could exploit this vulnerability to: read contents of local files via file:// scheme redirects, obtain FTP credentials by redirecting to malicious FTP servers, and access S3 content using configured credentials through s3:// redirects. The severity of data exposure depends on how the spider implements its parsing - spiders that output entire response contents are more vulnerable than those that extract specific fragments (GitHub Advisory).

The vulnerability requires an attacker to have write access to the spider's start requests and read access to the spider's output. The attack can be executed remotely with low complexity and requires low privileges. No user interaction is needed for exploitation (GitHub Advisory).

The primary mitigation is to upgrade to Scrapy version 2.11.2 or later. For users unable to upgrade immediately, a workaround is available by replacing the built-in retry middlewares (RedirectMiddleware and MetaRefreshMiddleware) with custom implementations that incorporate the fix from Scrapy 2.11.2. Users implementing this workaround should verify that their custom implementation works as intended (GitHub Advisory).