docs: Add guide "HttpCrawler with custom parser" #1622
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1622 +/- ##
==========================================
- Coverage 92.50% 92.49% -0.01%
==========================================
Files 157 157
Lines 10437 10439 +2
==========================================
+ Hits 9655 9656 +1
- Misses 782 783 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
Thanks for the guide Max. While the code contained in there works, the idea was to use AbstractHttpCrawler along with a custom implementation of AbstractHttpParser - similarly to how BeautifulSoupCrawler and company are implemented.
The cool thing about this is that you can then use the parser class in AdaptivePlaywrightCrawler as well.
Got it. What do you think about expanding this guide with another section on implementation based on |
Well, just adding it to the bottom won't cut it, but you can make the guide into two parts - quick and dirty solution and the "native" way. You can also explain the benefits of each approach. |
vdusek
left a comment
vdusek
left a comment
There was a problem hiding this comment.
Looks great!
I'm also updating the description to note that this resolves #702 as well, due to saxonche.
However, I would say this content would fit better in the HTTP crawlers guide. I'd suggest merging it there.
"Using HttpCrawler with a custom parser" could be the next section after "HttpCrawler", and "Creating a custom crawler" is essentially the same as "Creating a custom HTTP crawler".
docs/guides/code_examples/crawler_custom_parser/selectolax_adaptive_run.py
Show resolved
Hide resolved
…ptive_run.py Co-authored-by: Jan Buchar <Teyras@gmail.com>
vdusek
left a comment
vdusek
left a comment
There was a problem hiding this comment.
This is really good, thanks!
Just a few minor things... Mostly regarding more concise titles and consistent grammatical forms.
| 'AdaptivePlaywrightCrawler', | ||
| 'AdaptivePlaywrightCrawlingContext', | ||
| 'AdaptivePlaywrightPreNavCrawlingContext', | ||
| 'AdaptivePlaywrightCrawlerStatisticState', |
There was a problem hiding this comment.
If this was private until now, we should expose it in a separate "feat:" PR, as it extends the public interface.
Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
Co-authored-by: Vlada Dusek <v.dusek96@gmail.com>
3 participants
Description
Issues