-1

I'm trying to scrape the data off this site.

The website shows a charging station, in this case you can click each to unravel the accordion and see the data per charger. I am trying to use this specific site as it has access to a specific set of charger data which is not available anywhere else (not the one linked but similar enough).

I got as far as trying to use Beautiful Soup, but then switched to Selenium, as it seems the content is generated dynamically.

I've tried to Click various DIV tags

elements = driver.find_element(By.CLASS_NAME,"accordion-item-content")
    elements.click()

this is not producing any affect. actually, it's determined to be non clickable.

Where should I be looking in terms of being able to scrape this website for charger availability based on the roll out information from the accordion?

The data will be extracted and saved into a database to track availability and usage of each of the available sockets at the charging location

I tried clicking on the divs, didn't work. There are also no div ID's so am using "By.CLASS_NAME". Unsuccessful in trying to interact with the page, however it does load using chrome.

1
  • Update the question with the relavent HTML of the accordians in text format. Commented Sep 12 at 0:37

2 Answers 2

1

Try selecting the item with .outlet-list .accordion-item .item-content and CSS_SELECTOR:

driver.find_element(By.CSS_SELECTOR,".outlet-list .accordion-item .item-content")

Then just click the element and the accordion should extend.

Sign up to request clarification or add additional context in comments.

1 Comment

Great, thank you! Really helpful. I appreciate it!
0

First of all, there are 10 such div elements with the class name accordion-item-content and not all of them are what you're looking for, and also not clickable. So, that's why you get that error.

So, the first thing you should do is to narrow down your search and target the container that is holding the desired DOM.

enter image description here

As per your descriptions, the green box drawn in the image above is the DOM that contains all informations related to the charging stattion. So you should first wait for this element to be visibilally located in order to interact with the elements within.

Here's how you can do that:

container = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.block.station-info")))

now, you should perfom all search, clicks, etc. on this container.

All the data under the Charging points drop-down arrow are already loaded in DOM and clicking on it just shows in on UI. So, you can just ignore clicking it.

And now, to click on the Location info, or Facilities nearby or Add photo, you can again narrow down you search to find these elements:

location_facilities_photo = container.find_elements(By.CSS_SELECTOR, 'div.list.accordion-list')

this gets you a list of the three elements, you may iterate over them to perform the click.

For example, to click on the Facilities nearby drop-down (the 2nd element in list), use the selector div.item-content and click:

location_facilities_photo[1].find_element(By.CSS_SELECTOR, 'div.item-content').click()

you can follow the same to click on the other two drop-downs.

Here's the sample code you can try and see it's working:

import time
from selenium.webdriver import Chrome, ChromeOptions
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait

options = ChromeOptions()
options.add_argument("--screen-info={0,0 1920x1080}")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option("useAutomationExtension", False)

driver = Chrome(options=options)
wait = WebDriverWait(driver, 10)

driver.get("https://chargefinder.com/en/stromtankstelle-losser-t-borghuis-23450760/pr6j7n")

container = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.block.station-info")))

location_facilities_photo = container.find_elements(By.CSS_SELECTOR, 'div.list.accordion-list')
location_facilities_photo[1].find_element(By.CSS_SELECTOR, 'div.item-content').click()

# just to hold the screen to see the click working
time.sleep(2)

2 Comments

this was a very helpful answer, thank you! Really appreciate it! Thanks to this and the post from Brentspine, I was able to achieve what I needed!
things to do when someone answers your question and you find it helpful: stackoverflow.com/help/…

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.