Skip to content

Develop an example using the HTTP client as a web crawler #108

Open

Description

@glynos

A good example of the HTTP client would be as a web crawler. This could also be a good demonstration of the flexibility of the URI class.

Activity

  1. deanberris commented on Apr 25, 2012

    @deanberris
    Member

    This sounds like a good idea. However I'm afraid of the effort that would need to go into something like this. Parsing HTML is scary stuff and finding the URL's and interpreting rel="nofollow" in a DOM along with the myriad other link elements in a document is a tall order.

    That said, I'd be willing to accept contributions to this effect.

  2. glynos commented on Apr 26, 2012

    @glynos
    MemberAuthor

    My motivation for opening this issue is to start generating more complex and interesting examples for the HTTP client. At this stage I am not worried about the complexity of the HTML parsing.

  3. tex commented on Jan 3, 2018

    @tex

    Perhaps I'm rising a zombie, but nonetheless, I created this example
    https://github.com/tex/cpp-netlib-example
    Would you consider it as a good example of using cpp-netlib as a simple web crawler?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions