While using Scrapy, there are certain common issues that are hard to detect. For example, a typo in the name of a setting.

Build a list of common issues in code using Scrapy that could be detected using static code analysis, and build a tool or extend an existing tool to detect those.

HTTP responses should include a Content-Type header that indicates the MIME type of the response body. However, responses do not always include such a header, and sometimes they include it but the specified MIME type does not really match the response body.

Create a Python library that implements the complete MIME Sniffing Standard.

Integrate the resulting library into Scrapy.

This is a collection of small improvements that have been asked over time for scrapy’s FEEDS delivery.

Improve scrapy’s FEED delivery capabilities.

Currently scrapy doesn’t handle 429s properly. So, whenever we get 429 response code, we should update throttling configs and concurrency to adapt to the new rate.

A new middleware/extension that will handle 429 response codes and adjust request rates properly.

We believe there is much room for improvement in the performance of Dateparser. Moreover, the current implementation is not thread-safe.

Profile and optimize the performance of the library.

Make the library thread-safe.

Currently language detection is rudimentary and often causes incorrect interpretation of dates.

Improve how language detection works. Plugging-in an optional language detection library is an option.

There is a long list of issues that affect the search_dates function of Dateparser.

Make a plan to refactor and improve the function, fixing some of those issues.

Fix even more of those issues.

When you inspect a website element in a web browser, you get a DOM-based HTML tree that is different from the actual, underlying HTML tree. This makes it difficult to translate what you find in a web browser into an XPath or CSS expression that can work in Parsel. More so when the underlying HTML is actually broken.

Extend Parsel to support different HTML parsers, and add support for additional HTML parsers.

There is a W3C working draft for additional CSS selectors that adds many features

Extend cssselect to support additional CSS Selectors Level 4 that can be translated into XPath 1.0.

number-parser should allow to improve the capabilities of Dateparser and price-parser

Integrate number-parser into Dateparser and price-parser as a dependency that brings support for natural language numbers to those libraries

Extend the features of number-parser.