You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Infinite or very long loop in URL validation URL #376
Using ESAPI to validate URL with the default regex in the properties file causes some URLs to loop for a very long time, while hitting high, e.g. 30% CPU usage.
Call validator.isValidInput with the following URL:
I have verified that the validation.properties file in ESAPI tag 2.1.0.1 is identical to the one used in esapi tag 2.1.0. The problem is related to the custom regex. Will close as this problem isn't due to ESAPI, but in the regex used.
Below regular expression fails to validate the URL http://core-jenkins.scansafe.cisco.com/佐贺诺伦-^ńörén.jpg appropriately. Returns FALSE after around 30911 ms (OS -X El Capitan 10.11.5,core i7 2.2 GHz ) ^(ht|f)tp(s?)\\:\\/\\/[0-9a-zA-Z]([-.\\w]*[0-9a-zA-Z])*(:(0-9)*)*(\\/?)([a-zA-Z0-9\\-\\.\\?\\,\\:\\'\\/\\\\\\+=&%\\$#_]*)?$
This is the default one from validation.properties.
In local or regional contexts and with improving technology, users
might benefit from being able to use a wider range of characters;
such use is not defined by this specification. Percent-encoded
octets (Section 2.1) may be used within a URI to represent characters
outside the range of the US-ASCII coded character set if this
representation is allowed by the scheme or by the protocol element in
which the URI is referenced. Such a definition should specify the
character encoding used to map those characters to octets prior to
being percent-encoded for the URI.
The URI you've provided uses non-ascii characters and is therefore outside the bounds of RFC-3986. This isn't something we will support.
If you percent-encoded the URL, like this: http%3A%2F%2Fcore-jenkins.scansafe.cisco.com%2F%E4%BD%90%E8%B4%BA%E8%AF%BA%E4%BC%A6-%5E%C5%84%C3%B6r%C3%A9n.jpg
To underline that the use case is unsupported, run the following program:
package org.owasp;
import java.net.URI;
public class URITest {
public static void main(String[] args) throws Exception {
// TODO Auto-generated method stub
URI uri = new URI("http://core-jenkins.scansafe.cisco.com/佐贺诺伦-^ńörén.jpg");
}
}
You should get the following exception:
Exception in thread "main" java.net.URISyntaxException: Illegal character in path at index 44: http://core-jenkins.scansafe.cisco.com/佐贺诺伦-^ńörén.jpg
at java.net.URI$Parser.fail(URI.java:2848)
at java.net.URI$Parser.checkChars(URI.java:3021)
at java.net.URI$Parser.parseHierarchical(URI.java:3105)
at java.net.URI$Parser.parse(URI.java:3053)
at java.net.URI.<init>(URI.java:588)
at org.owasp.URITest.main(URITest.java:9)
My question is, if I get everything scanned through enc.canonicalize(), what all the possibilities(threats) do I eliminate ? I presume, I'd still need to run the individual canonicalized parts through some REGEX validation .. your thoughts on this please ?
enc.canonicalize(String s); buys you a scanner to protect against mixed encoding and multiple encoding attacks. However, canonicalizing any input twice like you do here:
Doesn't buy you anything. If you're doing that to decode the string, you lose the chance to detect multiple encodings.
Also, your input:
URI testUri = new URI("http%3A%2F%2Fcore- jenkins.scansafe.cisco.com%2F%E4%BD%90%E8%B4%BA%E8%AF%BA%E4%BC%A6-%5E%C5%84%C3%B6r%C3%A9n.jpg");
Is invalid in the first place. When I run that in a test program I get:
Exception in thread "main" java.net.URISyntaxException: Illegal character in path at index 18: http%3A%2F%2Fcore- jenkins.scansafe.cisco.com%2F%E4%BD%90%E8%B4%BA%E8%AF%BA%E4%BC%A6-%5E%C5%84%C3%B6r%C3%A9n.jpg
at java.net.URI$Parser.fail(URI.java:2848)
at java.net.URI$Parser.checkChars(URI.java:3021)
at java.net.URI$Parser.parseHierarchical(URI.java:3105)
at java.net.URI$Parser.parse(URI.java:3063)
at java.net.URI.<init>(URI.java:588)
at org.owasp.URITest.main(URITest.java:9)
But even if it were to pass URI parsing, the spaces make it a doubly-encoded input, because you've got percent-encoded parts in it.
The main danger with what you're doing is that if your URI has a query, canonicalize will for example transform ?¶m1=foo¶m2=bar into ¶m1=foo¶m2=bar which contains invalid URI characters. That's why in my implementation I treated the query separately.
As for,
"But even if it were to pass URI parsing, the spaces make it a doubly-encoded input, because you've got percent-encoded parts in it."
I think the spaces got introduced while I pasted the code from eclipse into the textarea here (hence the failure at char 18).
"The main danger with what you're doing is that if your URI has a query, canonicalize will for example transform ?¶m1=foo¶m2=bar into ¶m1=foo¶m2=bar which contains invalid URI characters. That's why in my implementation I treated the query separately."
Yes I faced this as well.. is it possible to share your implementation or some reference code please ?
Check the commits to this baseline I did yesterday. The changes haven't been reviewed by @kwwall yet, and I suspect I might have some cleanup to do, but the code is available.
_What if I escape the URL parts as below -
{PROTOCOL=http,
HOST=core-jenkins.scansafe.cisco.com,
PORT=-1,
AUTHORITY=core-jenkins.scansafe.cisco.com,
QUERY=,
PATH=/佐贺诺伦-^ńörén.jpg}
Is this a safe way to render non US-ASCII chars ?
Please note that my intention here is not to validate a URL but to render a URL containing non US-ASCII safely on a browser (view only).
If your goal is to safely render a URL for a browser, you should be escaping the output, it should (sort of ) look like this:
//If the target is an attribute:
<a href="${uri}"/>
then escape it like this:
encoder.escapeForHtmlAttribute(uri);
if the target output is link text:
<a href="foo">${uri}</a>
//escape with
encoder.escapeForHTML(uri);
That's all you actually need to do. Ideally, perform the escaping as close to the page as possible, meaning if you're in a JSP use the esapi taglibs or the taglibs provided by the Encoder project.
@xeno6696
Thanks for your response. However, it's not even a HREF .. let's say I just want to put it on the page within a < p > < /p >
To be precise, what I wanted to understand is, if "urlPart" in the below snippet can be considered safe for rendering like plain text ..
urlPart = encoder.encodeForHTML("/佐贺诺伦-^ńörén.jpg");
For the record, it were an 'href' attribute for , a 'src' attribute
for , etc. ... any attribute that behaves as a URL, then you want to
use URL encoding, not just simply HTML attribute output encoding. Otherwise
XSS possibilities still exist. Bit since you plan on just displaying it in
normal HTML context (e.g., between HTML tags, not in Javascript or CSS,
etc.), then you can just use
ESAPI.encoder().encodeForHtml(yourUriHere)
to do the output encoding.
@xeno6696 will try to review your code this wkend.
-kevin
Sent from my Droid; please excuse typos.
On Jun 24, 2016 11:36 AM, "sdey1977" notifications@github.com wrote:
@xeno6696https://github.com/xeno6696
Thanks for your response. However, it's not even a HREF .. let's say I
just want to put it on the page within a
...
To be precise, what I wanted to understand is, if "urlPart" in the below
snippet can be considered safe for rendering like plain text ..
urlPart = encoder.encodeForHTML("/佐贺诺伦-^ńörén.jpg");
Release 2.2.0.0 is planned RSN. Just having some maven struggles and doing some final code cleanup. Unfortunately, Dependency Check now seems to be broken. AFAICT, it's nothing that I did; I plan up updating ESAPI to use the latest version of Dep Ck. We're using 4.0.1. I'm going to try 5.0.0-M2 and if that doesn't fix it, I will contact the Dep Ck team. Join the new OWASP ESAPI Project Users Google group that was recently advertised as we will probably announce it first there.
-kevin
P.S.- In the meantime, if you are desperate, download it from GitHub, build it, and use the latest from the 'develop' branch where it is fixed.
Activity
xeno6696 commentedon May 18, 2016
This is why you don't want to use regex to "validate" URLs. I think you've discovered a worst-case input.
xeno6696 commentedon May 18, 2016
Also, what version of ESAPI, and could you please give us the full method call as well as the exact regex rule backed by validation.properties?
sshort commentedon May 18, 2016
Agreed - I wouldn't choose to use a regex for this.
Using ESAPI 2.1.0.
Regex:
^(?:(?:https?|ftp):\\/\\/)(?:\\S+(?::\\S*)?@)?(?:(?!(?:)(?:\\.\\d{1,3}){3})(?!(?:169\\.254|192\\.168)(?:\\.\\d{1,3}){2})(?!172\\.(?:1[6-9]|2\\d|3[0-1])(?:\\.\\d{1,3}){2})(?:[1-9]\\d?|1\\d\\d|2[01]\\d|22[0-3])(?:\\.(?:1?\\d{1,2}|2[0-4]\\d|25[0-5])){2}(?:\\.(?:[1-9]\\d?|1\\d\\d|2[0-4]\\d|25[0-4]))|(?:(?:[a-z\\u00a1-\\uffff0-9]-*)*[a-z\\u00a1-\\uffff0-9]+)(?:\\.(?:[a-z\\u00a1-\\uffff0-9]-*)*[a-z\\u00a1-\\uffff0-9]+)*(?:\\.(?:[a-z\\u00a1-\\uffff]{2,})))(?::\\d{2,5})?(?:\\/\\S*)?$xeno6696 commentedon May 18, 2016
I cannot reproduce this, @kwwall I ran the following unit-test in ValidatorTest:
assertTrue(instance.isValidInput("test", "http://mvcrzaufyh.youtube.com.hwq19b2dycja1red.youtube.com.25sq0l9.1cslt00rcg49.youtube.com.84116406.controversialtopic.net.//6js73/Q151E3//...//Q151E3//96093547/9863552/684833.bvmi?cCsRPsbcs2zDcvjJhczcvScwcdN6jcdMt" , "URL", 2030, false));All tests run in less than a second.
xeno6696 commentedon May 18, 2016
The regex backing in the default
validation.propertiesfile is this:^(ht|f)tp(s?)\\:\\/\\/[0-9a-zA-Z]([-.\\w]*[0-9a-zA-Z])*(:(0-9)*)*(\\/?)([a-zA-Z0-9\\-\\.\\?\\,\\:\\'\\/\\\\\\+=&%\\$#_]*)?$xeno6696 commentedon May 18, 2016
Yeah @sshort the problem is the regex. Where did you get it from?
xeno6696 commentedon May 18, 2016
I have verified that the
validation.propertiesfile in ESAPI tag 2.1.0.1 is identical to the one used in esapi tag 2.1.0. The problem is related to the custom regex. Will close as this problem isn't due to ESAPI, but in the regex used.sshort commentedon May 18, 2016
Not sure - I inherited this project. I thought I tracked it back to ESAPI itself but I shall try it out with the default regex you mention above.
Thanks for the fast responses.
sdey1977 commentedon Jun 3, 2016
Below regular expression fails to validate the URL
http://core-jenkins.scansafe.cisco.com/佐贺诺伦-^ńörén.jpgappropriately. Returns FALSE after around 30911 ms (OS -X El Capitan 10.11.5,core i7 2.2 GHz )^(ht|f)tp(s?)\\:\\/\\/[0-9a-zA-Z]([-.\\w]*[0-9a-zA-Z])*(:(0-9)*)*(\\/?)([a-zA-Z0-9\\-\\.\\?\\,\\:\\'\\/\\\\\\+=&%\\$#_]*)?$This is the default one from validation.properties.
Please advice.
xeno6696 commentedon Jun 3, 2016
Per the URI specification:
The URI you've provided uses non-ascii characters and is therefore outside the bounds of RFC-3986. This isn't something we will support.
If you percent-encoded the URL, like this:
http%3A%2F%2Fcore-jenkins.scansafe.cisco.com%2F%E4%BD%90%E8%B4%BA%E8%AF%BA%E4%BC%A6-%5E%C5%84%C3%B6r%C3%A9n.jpgThis would meet RFC-3986, and would be supported.
xeno6696 commentedon Jun 3, 2016
To underline that the use case is unsupported, run the following program:
You should get the following exception:
xeno6696 commentedon Jun 3, 2016
I should learn to read before posting. I missed this part @sdey1977 and I'm sorry.
The problem isn't that we're not supporting an expanded character set, it's that the URI provided is a worst-case input for the regex engine.
xeno6696 commentedon Jun 3, 2016
Added the following test case to ValidatorTest
Current execution time:

23 remaining items
xeno6696 commentedon Jun 13, 2016
enc.canonicalize(String s);buys you a scanner to protect against mixed encoding and multiple encoding attacks. However, canonicalizing any input twice like you do here:uriData.path(enc.canonicalize(enc.canonicalize(testUri.getAuthority() + testUri.getPath())));Doesn't buy you anything. If you're doing that to decode the string, you lose the chance to detect multiple encodings.
Also, your input:
URI testUri = new URI("http%3A%2F%2Fcore- jenkins.scansafe.cisco.com%2F%E4%BD%90%E8%B4%BA%E8%AF%BA%E4%BC%A6-%5E%C5%84%C3%B6r%C3%A9n.jpg");Is invalid in the first place. When I run that in a test program I get:
But even if it were to pass URI parsing, the spaces make it a doubly-encoded input, because you've got percent-encoded parts in it.
The main danger with what you're doing is that if your URI has a query, canonicalize will for example transform
?¶m1=foo¶m2=barinto¶m1=foo¶m2=barwhich contains invalid URI characters. That's why in my implementation I treated the query separately.xeno6696 commentedon Jun 13, 2016
I missed part of your question:
Yes, there should be some kind of further restriction on the URL. What that is however, is going to be highly context-dependent on your application.
sdey1977 commentedon Jun 13, 2016
As for,
"But even if it were to pass URI parsing, the spaces make it a doubly-encoded input, because you've got percent-encoded parts in it."
I think the spaces got introduced while I pasted the code from eclipse into the textarea here (hence the failure at char 18).
"The main danger with what you're doing is that if your URI has a query, canonicalize will for example transform ?¶m1=foo¶m2=bar into ¶m1=foo¶m2=bar which contains invalid URI characters. That's why in my implementation I treated the query separately."
Yes I faced this as well.. is it possible to share your implementation or some reference code please ?
xeno6696 commentedon Jun 13, 2016
Check the commits to this baseline I did yesterday. The changes haven't been reviewed by @kwwall yet, and I suspect I might have some cleanup to do, but the code is available.
sdey1977 commentedon Jun 24, 2016
@xeno6696 , if you could kindly have a quick look at this and share your views -
The below link suggests that if my interpreter is Browser(HTML content.. just printing the URL on the page) then escaping/encoding non US-ASCII is a safe way to render the text.
https://www.owasp.org/index.php/XSS_(Cross_Site_Scripting)_Prevention_Cheat_Sheet#RULE_.232_-_Attribute_Escape_Before_Inserting_Untrusted_Data_into_HTML_Common_Attributes
_What if I escape the URL parts as below -
{PROTOCOL=http,
HOST=core-jenkins.scansafe.cisco.com,
PORT=-1,
AUTHORITY=core-jenkins.scansafe.cisco.com,
QUERY=,
PATH=/佐贺诺伦-^ńörén.jpg}
boolean isAscii = CharMatcher.ASCII.matchesAllOf(urlPart);if(!isAscii){if(seg == URLSegment.AUTHORITY && null != parseMap.get(seg))value = encoder.canonicalize(IDN.toASCII(urlPart)); // for URLs like http://правительство.рф/..if(seg == URLSegment.PATH && null != parseMap.get(seg))urlPart = encoder.encodeForHTML(urlPart); //PATH=/佐贺诺伦-^ńörén.jpg}else{//regular US-ASCIIurlPart = encoder.canonicalize(parseMap.get(seg), allowMultiple, allowMixed);}Is this a safe way to render non US-ASCII chars ?
Please note that my intention here is not to validate a URL but to render a URL containing non US-ASCII safely on a browser (view only).
xeno6696 commentedon Jun 24, 2016
If your goal is to safely render a URL for a browser, you should be escaping the output, it should (sort of ) look like this:
then escape it like this:
if the target output is link text:
That's all you actually need to do. Ideally, perform the escaping as close to the page as possible, meaning if you're in a JSP use the esapi taglibs or the taglibs provided by the Encoder project.
xeno6696 commentedon Jun 24, 2016
@kwwall I'd like to close this out, but a change like this really does require a review. :-)
sdey1977 commentedon Jun 24, 2016
@xeno6696
Thanks for your response. However, it's not even a HREF .. let's say I just want to put it on the page within a < p > < /p >
To be precise, what I wanted to understand is, if "urlPart" in the below snippet can be considered safe for rendering like plain text ..
urlPart = encoder.encodeForHTML("/佐贺诺伦-^ńörén.jpg");
xeno6696 commentedon Jun 24, 2016
If you're just displaying a naked text URL (not providing an anchor tag) then yes, you would use escapeForHTML
kwwall commentedon Jun 24, 2016
For the record, it were an 'href' attribute for , a 'src' attribute
, etc. ... any attribute that behaves as a URL, then you want to
for
use URL encoding, not just simply HTML attribute output encoding. Otherwise
XSS possibilities still exist. Bit since you plan on just displaying it in
normal HTML context (e.g., between HTML tags, not in Javascript or CSS,
etc.), then you can just use
ESAPI.encoder().encodeForHtml(yourUriHere)
to do the output encoding.
@xeno6696 will try to review your code this wkend.
-kevin
Sent from my Droid; please excuse typos.
On Jun 24, 2016 11:36 AM, "sdey1977" notifications@github.com wrote:
xeno6696 commentedon Jun 28, 2016
Received your comments, will implement ASAP.
Merge pull request #391 from xeno6696/develop
tjain04 commentedon Apr 4, 2019
Do we have any release planned soon with this fix. ESAPI 2.1.0.1 does not have this fix.
kwwall commentedon Apr 5, 2019
Release 2.2.0.0 is planned RSN. Just having some maven struggles and doing some final code cleanup. Unfortunately, Dependency Check now seems to be broken. AFAICT, it's nothing that I did; I plan up updating ESAPI to use the latest version of Dep Ck. We're using 4.0.1. I'm going to try 5.0.0-M2 and if that doesn't fix it, I will contact the Dep Ck team. Join the new OWASP ESAPI Project Users Google group that was recently advertised as we will probably announce it first there.
-kevin
P.S.- In the meantime, if you are desperate, download it from GitHub, build it, and use the latest from the 'develop' branch where it is fixed.