Linking to javadoc.io using Javadoc -link option

Question

I am trying to link into some Javadocs hosted at javadoc.io (specifically, PowerMock's Javadocs) using the @link option. I have tried to add the URL to PowerMock's Javadocs to my -link flag, but can't get Javadoc to recognize it. I am using external links to other Javadocs just fine (e.g. Guava, Java SE 7) with Gradle as my build system. I have tried the following options:

-link http://static.javadoc.io/org.powermock/powermock-core/1.6.3/

^ I have confirmed that there is a package-list file in this directory

-link http://static.javadoc.io/org.powermock/powermock-core/

-link http://javadoc.io/doc/org.powermock/powermock-core/1.6.3/

-link http://javadoc.io/doc/org.powermock/powermock-core/

All of these result in the following error (URL changed accordingly):

javadoc: warning - Error fetching URL: http://static.javadoc.io/org.powermock/powermock-core/1.6.3/

Does anyone have advice on how to make this work?

As far as I can tell this is some sort of javadoc.io specific problem, though likely a usage issue on my end - for example I am currently using -link http://junit.org/javadoc/latest/ without issue, but -link http://static.javadoc.io/junit/junit/4.12/ doesn't work.

Maybe running javadoc with the -verbose Option will give a hint, whats wrong. — hinneLinks
– hinneLinks, Commented Nov 23, 2015 at 9:59
Unfortunately I still got nothing beyond: [ERROR] [system.err] javadoc: warning - Error fetching URL: http://static.javadoc.io/org.powermock/powermock-core/1.6.3/ — xkrogen
– xkrogen, Commented Nov 23, 2015 at 17:58
Accessing those URLs (static.javadoc.io versions) in a browser results in an access denied error. Javadoc probably has the same result. — erickson
– erickson, Commented Nov 23, 2015 at 18:55
Does Javadoc actually try to access the base URL? You can access the package list file, package summaries, and class files just fine. — xkrogen
– xkrogen, Commented Nov 23, 2015 at 19:15
I've emailed the javadoc.io contact people as well, was just hoping someone here would have experience since (AFAIK) javadoc.io is pretty standard. — xkrogen
– xkrogen, Commented Nov 23, 2015 at 19:19

Chris Povirk · Accepted Answer · 2017-12-20 20:52:50Z

8

From the command line, use an argument like -J-Dhttp.agent=javadoc.

In Maven, use something like:

<additionalJOption>-J-Dhttp.agent=maven-javadoc-plugin-${pom‌.name}</additionalJO‌ption>

The background: As Danilo Pianini suggests in another answer, the problem is the User-Agent header. However, the problem isn't an empty User-Agent; it's the default Java User-Agent, which looks something like "Java/1.8.0_112":

$ URL=https://static.javadoc.io/org.checkerframework/checker-qual/2.2.2/package-list

# default Java User-Agent:
$ wget -U Java/1.8.0_112 "$URL" 2>&1 | grep response
HTTP request sent, awaiting response... 403 Forbidden

# no User-Agent:
$ wget -U '' "$URL" 2>&1 | grep response
HTTP request sent, awaiting response... 200 OK

# custom User-Agent:
$ wget -U javadoc "$URL" 2>&1 | grep response
HTTP request sent, awaiting response... 200 OK

So the fix is to tell Javadoc to use a different User-Agent. Java won't let you omit the User-Agent, so you'll have to provide a value, which Java will prepend to its default agent.

As best I can tell, the blocking of Javadoc isn't intentional: Javadoc just (probably unwisely) uses the default Java User-Agent, and the content delivery network that javadoc.io uses blocks that by default.

(One more note about Maven: Everything works fine with -link. It also works fine with -linkoffline if you download the package-list file and tell Javadoc to read it from disk. However, if you use -linkoffline but tell Javadoc to fetch package-list from the javadoc.io URL (this is an unusual thing to do), it may fail. The problem: Maven tries to pre-validate the package-list file but, under some versions of Java, fails because it rejects the SSL certificate of javadoc.io, a certificate that Javadoc itself accepts.)

(Oh, and it appears to be important to use a URL specifically from static.javadoc.io, not javadoc.io. Also, I would recommend https, not http, in case http://static.javadoc.io someday starts issuing redirects to https://static.javadoc.io, as Javadoc currently doesn't handle such redirects. Also, https is a good thing :))

edited Dec 20, 2017 at 20:52

answered Dec 19, 2017 at 16:39

Chris Povirk

4,0603 gold badges34 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Laird Nelson Over a year ago

A quick note to indicate that this mechanism of setting the User-Agent for javadoc to use no longer works in JDK 11. I'm not sure why, but I'm guessing that javadoc probably no longer uses the java.net.* classes.

Laird Nelson Over a year ago

Slightly related bugs: bugs.openjdk.java.net/browse/JDK-8184969, bugs.openjdk.java.net/browse/JDK-8190312

ST-DDT Over a year ago

For gradle 5 you need: options.jFlags('-Dhttp.agent=gradle-javadoc') (The single quotes are somehow important...)

Radio Rogal · Accepted Answer · 2016-06-08 12:25:12Z

It's strange: I could see in the browser e.g. http://static.javadoc.io/org.pegdown/pegdown/1.6.0/package-list but when I add http://static.javadoc.io/org.pegdown/pegdown/1.6.0 as javadoc's link option it says

Error fetching URL: http://static.javadoc.io/org.pegdown/pegdown/1.6.0/package-list

I use next workaround:

With maven-dependency-plugin unapack the javadoc of desired dependency.

Link it with linkoffline option.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <version>2.8</version>
    <executions>
        <execution>
            <id>unpack-javadoc</id>
            <phase>package</phase>
            <goals>
                <goal>unpack</goal>
            </goals>
            <configuration>
                <artifactItems>
                    <artifactItem>
                        <groupId>org.pegdown</groupId>
                        <artifactId>pegdown</artifactId>
                        <classifier>javadoc</classifier>
                        <version>${pegdownVersion}</version>
                        <overWrite>false</overWrite>
                        <outputDirectory>${project.build.directory}/pegdown-javadoc</outputDirectory>
                    </artifactItem>
                </artifactItems>
            </configuration>
        </execution>
    </executions>
</plugin>
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-javadoc-plugin</artifactId>
    <configuration>
        <links>
            <link>http://www.slf4j.org/apidocs/</link>
        </links>
        <offlineLinks>
            <offlineLink>
                <url>http://static.javadoc.io/org.pegdown/pegdown/${pegdownVersion}</url>
                <location>${project.build.directory}/pegdown-javadoc</location>
            </offlineLink>
        </offlineLinks>
    </configuration>
</plugin>

Danilo Pianini · Accepted Answer · 2017-01-26 13:46:20Z

4

I have investigated the problem, the issue here is that a user agent must be set (an empty string is ok) in order for the connection to javadoc.io to complete successfully.

I worked the problem around and wrote a Gradle plugin that may be of help for those who rely on that build system.

Unfortunately, the work around can not get ported to the regular javadoc -link command invocation.

edited Jan 26, 2017 at 13:46

answered Nov 9, 2016 at 18:22

Danilo Pianini

1,11613 silver badges21 bronze badges

4 Comments

xkrogen Over a year ago

That's awesome! Great work figuring out the root cause and working around it, Danilo.

Danilo Pianini Over a year ago

Thanks! By the way, I forgot a not in the last sentence: the workaround can NOT get ported to the regular command line invocation.

Chris Povirk Over a year ago

I got this to work with the default plugin by setting the system property that Java uses for User-Agent on the javadoc process. Note that this works only with -link, not -linkoffline, as Maven pre-validates -linkoffline package-list files, and javadoc.io appears to be rejecting Maven's requests, too -- though seemingly based on something more than the User-Agent check it's using to reject javadoc itself. Anyway, the magic is <additionalJOption>-J-Dhttp.agent=maven-javadoc-plugin-${pom.name}</additionalJOption> (or whatever agent string you want to use).

Chris Povirk Over a year ago

Also, I should note that I don't think javadoc.io is intentionally blocking javadoc -link. This Eclipse bug suggests that blocking Java/* user agents is the default behavior for some content delivery networks, and javadoc.io and javadoc are just accidentally affected. Maybe javadoc itself should set a User-Agent....

Max · Accepted Answer · 2019-04-08 13:09:16Z

4

I am running javadoc.io.

This is reported as this github issue and it's solved just now. There is no need to override user agent string any more.

Feel free to re-open github issue if things are still not working. This thread is not actively monitored.

curl -I -A "Java/1.6.0_14" https://static.javadoc.io/org.checkerframework/checker-qual/2.2.2/package-list
HTTP/1.1 200 OK
Date: Mon, 08 Apr 2019 13:06:04 GMT
Content-Type: text/plain

answered Apr 8, 2019 at 13:09

Max

2,10524 silver badges20 bronze badges

Comments

xkrogen · Accepted Answer · 2015-11-24 01:23:33Z

2

I ended up just using -linkoffline to get around this issue, which I suppose has the nice property of not needing internet connectivity at build time, though if anyone has further thoughts on how to make this work with -link I'm all ears.

answered Nov 24, 2015 at 1:23

xkrogen

6546 silver badges15 bronze badges

Collectives™ on Stack Overflow

Linking to javadoc.io using Javadoc -link option

5 Answers 5

3 Comments

Comments

4 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

3 Comments

Comments

4 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related