1,827 questions
0
votes
0
answers
97
views
How to get the value and stop the JS-execution?
I want to get a value that is generated by JavaScript-code and is available in the HTTP-header of a request. Then I want to continue with that webClient as it has cookies and resources loaded.
I don't ...
- 13k
0
votes
0
answers
42
views
HtmlUnit get nested form for authentication
I've been trying to use html unit to authenticate to a page using okta. The html looks like this:
I can use getElementById to get the okta-login-container div, but I can't get anything inside of it. ...
- 864
0
votes
1
answer
35
views
How do I tell HtmlUnit to ignore a non found JavaScript file?
If you go to https://www.to.pima.gov/propertyInquiry/?stateCode=101016140 using Chrome on Windows 11, you will see a fair amount of information. I am trying to get to some of that information using ...
0
votes
1
answer
82
views
Java HtmlUnit not finding an HtmlTable with specific role
If you go to https://parcelinquirytreasurer.cochise.az.gov/ , type 3020702601 in the text field and click on Submit, you will see $514.20 at the (4,6) position in a table. If you click on that $514.20,...
0
votes
1
answer
68
views
htmlunit Content Security Policy eval is blocked
I wrote a scraper on a javascript heavy website (written in angular) and it works perfectly. But now after a few months I get error
SEVERE: ERROR Error: Uncaught (in promise)
My suspect is when I ...
- 61
0
votes
0
answers
22
views
Cannot use JavaScriptExecutor in HtmlUnit-Android Package
I cannot seem to find anything which lets me progress from this stumbling block, as there doesn't appear to be an 'execute' in JavaScriptExecutor. All suggestions as how I might overcome this ...
- 107
0
votes
1
answer
106
views
click() method from element object (HtmlUnit) is doing nothing
I am trying to crawl on this website: https://www.softpedia.com/get/Programming/Other-Programming-Files/Apidog.shtml using HtmlUnit.
I want to "click" on the "Download now" button, ...
-1
votes
1
answer
54
views
HtmlUnitAndroid click does not do anything
I am using HtmlUnit Android to scrap data from a website, I am able to create the web client and connect it to the website, but after clicking a button I cannot obtain further elements. In the code, ...
- 113
0
votes
0
answers
86
views
HTMLUnit button.click() doesn't return a new HtmlPage object
I am and HTMLUnit newbie and am trying to scrape this website for DATACENTER. I am able to get to the first page and scrape the necessary information. Within the page there is an array of buttons ...
- 25
-1
votes
1
answer
46
views
Html Unit with xpath not returning the expected result
I am trying to screen scrap the top news on a particular date of a particular news paper using google search using Html Unit. I am able to get the search results but when I am trying to access the top ...
- 832
1
vote
1
answer
87
views
Trying to select the correct button using a css selector with HTMLUnit
I have the following snippet of HTML code:
<div class="LocationsIndex__paginationBar__YE2Xo">
<div>
<nav>
<button class="...
- 25
-1
votes
1
answer
51
views
Java HtmlUnit No Longer Able To Find List<HtmlFieldSet> Using getByXPath(@class=)
If you go to https://parcelinquirytreasurer.cochise.az.gov/ , type 1010501508 in the text field and click on Submit, you will get a new web page (Page 2). There is a mailing address on Page 2. If you ...
1
vote
2
answers
311
views
clicking a button using htmlunit and java
i tried all the options from the stackoverflow site, I'm an intern in programming. Simply indicate how to click.
Site: https://calculator888.ru/random-generator/sluchaynoye-slovo
Button:
<div class=...
- 13
0
votes
2
answers
103
views
How to auto submit forms using Java headless browser HtmlUnit
I'm quite new to HtmlUnit but what I'm trying to do here is as follows
we have a crystal server where we need to call to fetch reports
we are using Restful APIs that are exposed from crystal server to ...
- 91
1
vote
1
answer
105
views
HTMLUnit with Java: Calling click() ignored because the target element is not displayed
I'm using htmlunit with Java, and I'm trying to click a span on a website that should increase a counter and unlock another button, but unfortunately, it is not working.
The warning states, that the ...
- 23
0
votes
0
answers
115
views
input value being empty even after waiting for the value to be updated dynamically with HtmlUint3 Android
I use the HtmlUint3 Android library and I want to get the value of a tag element and update it or send a value to it using JavaScript, or the value is set dynamically through JavaScript after the ...
0
votes
1
answer
364
views
HtmlUnit: Handling Hidden Google reCAPTCHA Token in Login Automation
I'm using HtmlUnit to automate the login process on a website. The website employs a Google reCAPTCHA to protect its login form, and the reCAPTCHA token is hidden in the HTML. I need to obtain and use ...
- 365
1
vote
1
answer
427
views
Using HTMLUnit in Android Studio to scrap website in android app, but WebClint is not recognizing as import in ActivityClass
I want to scrap a website, but I can't use jsoup because jsoup don't have JavaScript execution. I am trying to run HTMLUnit in my Android app with version: 3.3.0, but in activity class, its not ...
- 523
0
votes
1
answer
116
views
Using HtmlUnit to load dynamic web apps
Pardon the long code, but I have been trying to make a web page loader class that can load both static and dynamic pages. The code below works fine with mostly static web pages but it won't load SPA ...
- 35.7k
0
votes
2
answers
128
views
How to load a "local storage" value to HtmlUnit web client?
I'm trying to load a local storage value into the HtmlUnit web client before it loads a specific website:
HtmlPage initialPage = webClient.getPage(url);
if (httpHeadersSpec != null && ...
- 35.7k
0
votes
2
answers
99
views
HTMLUnit for Android extremly slow
Hello i want to build an app with htmlunit for android (Android Studio).
If I try to cget the page of youtube this tooks a lot of seconds (over 10). So that is too slow for user experience... other ...
- 1
0
votes
1
answer
253
views
htmlunit parse Html version 2.70
I have been trying to upgrade htmlunit (https://www.htmlunit.org/) to version 2.70 from version 2.27. I noticed that the static method HtmlParser.parseHtml() no longer exists.
I understand I have to ...
- 864
1
vote
2
answers
201
views
How to prevent HtmlUnit from throwing an error that is in the javascript?
I'm trying to login to one website, and after entering successfully username and password, and after clicking on submit button, I'm getting the following:
2023-05-27 16:46:41.864 ERROR 296 --- [nio-...
0
votes
1
answer
296
views
When will HtmlUnit Version 3 will be released?
Can anyone please let me know when will HtmlUnit Version 3 will be released in the maven repository?
- 381
0
votes
1
answer
241
views
How to use htmlunit to record all requests fired during rendering a page?
I'm using HTMLUnit that trying to record all requests fired when loading a local html file.
This is the testing file below:
<script type="text/javascript">
!(function () {
var ...
- 21
2
votes
1
answer
244
views
How to extract the HTML elements inside <div data-module-*> from a website source code using HTMLUnit?
I am trying to scrap the content (Example Thumbnail picture of a course, price etc.) from an educative website - Udemy, by searching in a general URL string (given below). The source code of the site ...
- 21
0
votes
0
answers
79
views
How to capture boostrap CSS change in HtmlUnit?
"show" class is added into modal when modal is trigger to be displayed by button, but unfortunately it is unable to be captured by XPath, showing as null.
Java
try (final WebClient webClient ...
- 107
0
votes
1
answer
191
views
Java HTMLUnit WebClient perceives enabled JavaScript as disabled, when parsin' t' page
As stated.
Used 2 B getting some JS error messages, but able to parse t' page partly now.
Still, among some largely generic am seeing the next message:
<noscript>
<strong&...
- 23
0
votes
2
answers
113
views
Parse raw string HTML to HtmlDivision instance (HtmlUnit)
I'm trying to cast a simple raw HTML string (coming from an AJAX call) to HtmlDivision instance using HtmlUnit library
String div = "<div ... /div>"
HtmlDivision div = (HtmlDivision) ...
- 174
0
votes
1
answer
130
views
How to use javascript DOM implementation from HtmlUnit
The problem is that I need to extract javascript DOM implementation like window and document variables. I need to do it because DOM is huge and I need only some of the variables to evaluate my code. I ...
0
votes
1
answer
98
views
Catch all ajax requests (grid refresh) using HtmlUnit
I am scrapping a website with this workflow process using HtmlUnit library.
Goes to root site like www.mysite.com
Do login (www.mysite.com/login)
Access to a page where table/grid is located (www....
- 174
0
votes
1
answer
103
views
How does Apache HttpClient is related to htmlunit?
I'm having below entry in pom.xml
<dependency>
<groupId>net.sourceforge.htmlunit</groupId>
<artifactId>htmlunit</artifactId>
<version>2.70.0</version&...
- 3,797
0
votes
1
answer
740
views
java.lang.IllegalArgumentException: Cannot locate declared field class org.apache.http.impl.client.HttpClientBuilder.dnsResolver
Reference:
https://devpal.co/blog/web-scraping-with-java-and-htmlunit/
I am doing exactly same steps as mentioned in the blog but getting error
java.lang.IllegalArgumentException: Cannot locate ...
- 3,797
0
votes
0
answers
270
views
Cannot resolve symbol 'gargoylesoftware' even when htmlunit dependancy is present under pom.xml
pom.xml
<dependency>
<groupId>net.sourceforge.htmlunit</groupId>
<artifactId>htmlunit</artifactId>
<version>2.70.0</version>
&...
- 3,797
0
votes
0
answers
150
views
I'm receiving an error when entering google Htmlunit Can you please explain it?
Im trying to make a program that goes into google and searches it's contents for programing blogs and the things alike. however when I enter the google domain I get a long error. Im using HtmlUnit and ...
1
vote
2
answers
140
views
How do I get Java/HtmlUnit to accept an old or bad website certificate?
If you go to https://eagletw.mohavecounty.us/treasurer/treasurerweb/search.jsp using Firefox on Linux, you will see that you can browse the website just fine.
But when I compile and run the following ...
1
vote
1
answer
69
views
How do I effectively click on an HtmlCell using Java/HtmlUnit?
If you go to
https://taxtest.navajocountyaz.gov/Pages/WebForm1.aspx?p=1&apn=205-27-014
view page source and search for grdCPhist in the page source, you won't find it.
But, if you click on Taxes, ...
0
votes
1
answer
76
views
How do I preserve line breaks when extracting text out of an HtmlElement
I am trying to extract three separate strings from:
https://taxtest.navajocountyaz.gov/Pages/WebForm1.aspx?p=1&apn=103-03-122
The owners names: Johnson Tommy A & Nell H Cprs
The owners street ...
2
votes
1
answer
439
views
Why isn't javac finding the com.gargoylesoftware.htmlunit package I am trying to import
I have downloaded the latest release of HtmlUnit (released on January 5, 2023), along with all of its dependencies, and put it in /opt/htmlunit_2.69.0
I have been, so far unsuccessfully, trying to ...
0
votes
1
answer
74
views
HTMLUNIT Java only redirects to final page when debugging - on run it doesn't
I have a problem when trying to reach a final page (after log in) when using HTMLUNIT (version 2.67.0)
When I debug my code (line by line), I can put my e-mail and password and go the final page after ...
- 569
0
votes
1
answer
89
views
Iterative download using HtmlUnit
My Issue:
I am attempting to download all files located here (https://mft.rrc.texas.gov/link/1bf41875-3edd-4660-8ec5-b4cd15880563) using Java/HtmlUnit. (Each 'download' is a returned as a .zip ...
- 121
0
votes
1
answer
72
views
How to set text in HtmlUnit Span outside of an element?
I want to simulate clicking on a button in a Learning Management System, Schoology, in order to change a setting on an assignment I'm creating using Java and HtmlUnit. When I get the element and click ...
- 1
0
votes
1
answer
116
views
HtmlUnit - EcmaError: ScriptException: ReferenceError:"Promise" is not defined
I am using HtmlUnit on my android project:
implementation 'net.sourceforge.htmlunit:htmlunit:2.15'
I am simply try to get a webpage:
private final static String LOGIN_URL = "https://sso....
- 960
1
vote
1
answer
349
views
Java Htmlunit Cloudflare protection stuck on redirecting to the final page
So, I am trying to access a website (using Java Htmlunit - version 2.20 - can´t update it, company policies) - it is a government website - https://www2.aneel.gov.br:443/aplicacoes_liferay/tarifa/ - ...
- 569
0
votes
1
answer
127
views
HtmlUnit - ScriptException: Exception invoking drawImage
I am using HtmlUnit on my android project:
implementation group: 'net.sourceforge.htmlunit', name: 'htmlunit-android', version: '2.67.0'
I am simply try to get a webpage:
private final static String ...
- 960
1
vote
0
answers
405
views
HtmlUnit: WebClient enabled JavaScript, but still got JavaScript load error
Tried to create a test using HTMLUnit for Confluence and had to login. But getting error that 'Javascript load error'. I am quite new to HTMLUnit and have only basic knowledge.
Tried for Chrome, ...
- 11
0
votes
1
answer
157
views
HtmlUnit returning empty list of DomElements
I am having trouble retrieving the list of Dom Elements when using the method getElementsByName from HtmlPage.
Here is the HTML Page. (Trying to get the CategoriaAgente from the select tag).
HTML (The ...
- 569
1
vote
0
answers
180
views
HtmlUnit WebClient enabled JavaScript, but still got JavaScript disabled message
I tried to use HtmlUnit to scrape the webpage of https://www.maersk.com/tracking/
I already set webClient.getOptions().setJavaScriptEnabled(true);
String url = "https://www.maersk.com/...
- 161
1
vote
0
answers
734
views
Why do I get a 421 Misdirected Request error with HtmlUnit and not with curl, OKHttp and standard browsers?
I have a HtmlUnit "script" that has been running for months, but started to fail about a week ago. Some changes were most probably made on the backend of the web site targeted by my script, ...
- 475
0
votes
0
answers
52
views
Getting the url after clicking a link with HtmlUnit
So this is my first question here after reading like hundreds of threads in this forum :D.
I want to click an HtmlAnchor with HtmlUnit and retrieve the underlying url.
Maybe I am using the wrong ...
