How can I activate each item and parse their info?

[shahin]

Hi there! Hope you all are doing fine. Today i came across a different type of problem while scraping a webpage. So far, when there is a button to click to get to the target page, I scraped the href of that button and parsed the data of that page by reaching there making another request. However, in this case I found some img's instead of href's connected to each button. When I scraped those img's, I got genuine image links in lieu of any fruitful links that can lead me to the target area. The page I'm dealing with contains several images. When an image is clicked, new information concerning its' flavor comes up under the image. My goal is to parse all the flavors connected to each image. My script can parse the flavors of currently active image. How can I select other images and parse their flavor as well? Thanks in advance.

Sub Web_Data()
    Dim http As New XMLHTTP60, html As New HTMLDocument
    Dim post As HTMLHtmlElement, elem As Object

    With http
        .Open "GET", "https://www.optigura.com/uk/product/gold-standard-100-whey/", False
        .send
        html.body.innerHTML = .responseText
    End With

    For Each post In html.getElementsByClassName("colright")
        With post.getElementsByClassName("opt2")(0).getElementsByTagName("label")
            For N = 0 To .Length - 1
                x = x + 1: Cells(x, 1) = .item(N).innerText
            Next N
        End With
    Next post

'Here i parsed the button links connected to each image but found the image itself instead of any href

'    For Each elem In html.getElementsByClassName("img")
'        With elem.getElementsByTagName("img")
'        If .Length Then x = x + 1: Cells(x, 1) = .item(0).src
'        End With
'    Next elem
End Sub

The picture underneath can give you the clarity if anything I failed to describe.

 

[shahin]

I tried with Post request but got nothing. Perhaps I'm doing something wrong in making proper requests:

Sub httpPost()
    Dim http As New XMLHTTP60, html As New HTMLDocument
    Dim argumentString As String
   
    argumentString = "opt=flavor&opt1=207&opt2=47&ip=105"
    With http
        .Open "POST", "https://www.optigura.com/product/ajax/details.php", False
        .setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36"
        .setRequestHeader "Content-type", "application/x-www-form-urlencoded"
        .setRequestHeader "Accept", "application/json, text/javascript, */*; q=0.01"
        .send argumentString
        html.body.innerHTML = .responseText
    End With
    MsgBox http.responseText
End Sub

 

[shahin]

Finally, I've made it. To receive the required response it is necessary to send a GET request first then again send a POST request using the response from that get request. Here is the working one:

Sub httpPost()

    Dim http As New XMLHTTP60, html As New HTMLDocument
    Dim ArgumentStr As String
 
    ArgumentStr = "opt=flavor&opt1=207&opt2=47&ip=105"
 
    With http
        .Open "GET", "https://www.optigura.com/uk/product/gold-standard-100-whey/", False
        .send
    End With

    With http
        .Open "POST", "https://www.optigura.com/product/ajax/details.php", False
        .setRequestHeader "X-Requested-With", "XMLHttpRequest"
        .setRequestHeader "Content-type", "application/x-www-form-urlencoded"
        .setRequestHeader "Accept", "application/json, text/javascript, */*; q=0.01"
        .send ArgumentStr
        html.body.innerHTML = .responseText
    End With
 
    MsgBox http.responseText
 
End Sub