Workaround from a lazy loading webpage

[shahin]

Hi there! Hope you all are doing well. While making a parser I noticed that Yellowpage Canada uses lazy loading method in their webpage. However, the issue I'm facing here is that the page contains 84 Names but my parser is scraping only 40 Names out of them. Is there any workaround from this? Any corrective measure from your end in my code would be a great help. Thanks in advance. Here is the code:

http.Open "GET", "http://www.yellowpages.ca/search/si/1/Outdoor%20wedding/Edmonton", False
http.send
html.body.innerHTML = http.responseText

Set topics = html.getElementsByClassName("listing__name--link jsListingName")
For Each topic In topics
    Cells(x, 1) = topic.innerText
    x = x + 1
Next topic

 

[shahin]

Sorry to have found out late that the question I have erected here has been answered by Sir CHIHIRO few days back in a PM. The answer was:


If you inspect the page. You'll notice that it's broken into 3 separate responses.
You need to read three of them to get complete listings.

"/search/si/1/coffee/All+States" & "/search/si/2/coffee/All+States" & "/search/si/3/coffee/All+States".

 

[shahin]

One more thing to know about this which is; while inspecting a webpage It's not that easy always to find out where those responses lie. Are there any hard and fast rules to get them other than looking into the form element?

 

[shahin]

In case of this page "http://www.yellowpages.ca/search/si/1/pizza/Toronto, ON", the parser is fetching only 140 records out of around 2200 names.

 

[shahin]

Cross posted in: http://www.ozgrid.com/forum/showthread.php?t=203270&highlight=Scraping+names+webpage

 

[shahin]

Found it. The answer is exactly what Sir Chihiro said earlier on this. Basically, the site releases 40 records in each request. Like this:
First 40 records: http://www.yellowpages.ca/search/si/1/pizza/Toronto+ON
Next 40 records: http://www.yellowpages.ca/search/si/2/pizza/Toronto+ON
Next 40 records: http://www.yellowpages.ca/search/si/3/pizza/Toronto+ON

And so on until it is over 50 or something to get the whole records.