Can't scrape names from a web page using GET request

[shahin]

Running my vba script I can see that it can't reach the target page where i would like to extract the different names from. Clearly I have messed somewhere up in my code but can't find out. Hope somebody will help me find out where i'm making mistakes. Thanks. Here is the code:

Sub Getmethod()
Dim http As New MSXML2.ServerXMLHTTP, html As New HTMLDocument
Dim docs, doc, ele As Object
Dim StrData As String

StrData = "what=Plumbers/where=All+States"
    With http
        .Open "GET", "http://www.yellowpages.co.za/search/" & StrData, False
        .setRequestHeader "Content-Type", "text/xml"
        .send
        html.body.innerHTML = .responseText
    End With
  
    Set docs = html.getElementsByClassName("resultName")
    For Each doc In docs
        Set ele = doc.getElementsByTagName("a")(0)
        x = x + 1
        Cells(x, 1) = ele.innerText
    Next doc
Set html = Nothing: Set docs = Nothing: Set doc = Nothing: Set ele = Nothing
End Sub

 

[shahin]

And again if i try like this, it works that means json should not put any barrier to get data from.

Sub ypData()
Dim html As New HTMLDocument
Dim docs, doc, ele As Object

With CreateObject("MSXML2.serverXMLHTTP")
    .Open "GET", "http://www.yellowpages.co.za/search/plumbers/all+states/2", False
    .send
    html.body.innerHTML = .responseText
End With
Set docs = html.getElementsByClassName("resultName")
    For Each doc In docs
        Set ele = doc.getElementsByTagName("a")(0)
        x = x + 1
        Cells(x, 1) = ele.innerText
    Next doc
Set html = Nothing: Set docs = Nothing: Set doc = Nothing: Set ele = Nothing
End Sub

 

[shahin]

Here are the html elements for form:

<form action="/search/"class="searchForm" id="searchForm" autocomplete="off"><div id="whatDiv"><label class="searchLabel whatLabel" data-icon=""><input type="search" id="whatField" name="what"class="searchField whatField" placeholder="Search for companies and services" autocomplete="off" tabindex="1"></label></div><input type="submit" value="GO"class="searchBtn" tabindex="3" id="searchBtn"><label class="searchLabel whereLabel" data-icon=""><input type="search" id="whereField" name="where"class="searchField whereField" placeholder="Location" autocomplete="off" tabindex="2"></label></form>

 

[Chihiro]

Your URL string construction is off in your first code.

StrData = "what=Plumbers/where=All+States"

Should be...

StrData = "plumbers/All+states/1"

 

[shahin]

Thanks sir, Chihiro. It works like magic. But i could not understand why the parameters (what and where) should be off to get the results cause you know as of now i have been using the way i started.

 

[Chihiro]

Depends on site and how parameters are handled by web service.

In this instance, parameters are used to build url string and not passed directly.

 

[shahin]

Any rules to bear in mind for future references in case of manipulating get or post request parameter?

 

[Chihiro]

Just pay attention to what's displayed in Developer Tool and how web service etc works.

 

[shahin]

Thanks a zillion for saving my day.