How to get logged-in in yahoo.com?

shahin · Oct 9, 2017

I'm trying to get logged-in using vba in yahoo.com but not being able to. I can't really understand what I'm missing here to get a valid response from that site. The one important thing I could notice that while arranging "POST" request it requires such an url: the part of which I'm also sending via "argstring" argument. However, I might be wrong but it seemed to me so. If anybody helps me on this as to how I can get a valid response from that site, i would be very grateful to him.

Site URL= "https://www.yahoo.com/"

Code:

Sub yahoo_login()

    Const link As String = "https://login.yahoo.com/account/challenge/password?.src=fpctx&.intl=us&.lang=en-US&authMechanism=primary&display=login&yid=shahin.iqbal&dname=shahin.iqbal&done=https%3A%2F%2Fwww.yahoo.com%2F&s=Qg--&sessionIndex=Qg--&c=OaD_g77.2bIVjU5p_lYq3ZtAhKImes9cNmxVpaCsq9.5fF1sYuBwAlaTxMXAHZac7j2CDD8r8EwDtYg19oIuTNbxvysl61f4Dqtbz7.VKc26f6nJCcbfDcfzZMXjUnkF9QEJsVUxiyq8lXCCcqDaA4qy8tL_vw0WMMQ7f42YxZfbdO8qlapWZxQR.f79UqyyN0KdYFujLkWWD85GyHzmx_YKbfr4y9prQFCV5FAioEzfAGJ705njwu0x9uS6RfDQcvkBwDJp.hIuGWXCqnMP6pwPb9zMPXDkdCsAZzwnZrKi.tuqvtr_RsA-~A&crumb=WFKvLRbcxEC&acrumb=NZ00R7FD"
    Dim http As New ServerXMLHTTP, html As New HTMLDocument
    Dim argstring As String

    argstring = ".src=fpctx&.intl=us&.lang=en-US&authMechanism=primary&display=login&yid=shahin.iqbal&dname=shahin.iqbal&done=https%3A%2F%2Fwww.yahoo.com%2F&s=Qg--&sessionIndex=Qg--&c=OaD_g77.2bIVjU5p_lYq3ZtAhKImes9cNmxVpaCsq9.5fF1sYuBwAlaTxMXAHZac7j2CDD8r8EwDtYg19oIuTNbxvysl61f4Dqtbz7.VKc26f6nJCcbfDcfzZMXjUnkF9QEJsVUxiyq8lXCCcqDaA4qy8tL_vw0WMMQ7f42YxZfbdO8qlapWZxQR.f79UqyyN0KdYFujLkWWD85GyHzmx_YKbfr4y9prQFCV5FAioEzfAGJ705njwu0x9uS6RfDQcvkBwDJp.hIuGWXCqnMP6pwPb9zMPXDkdCsAZzwnZrKi.tuqvtr_RsA-~A&crumb=WFKvLRbcxEC&acrumb=NZ00R7FD"

    With http
        .Open "POST", link, False
        .setRequestHeader "Content-type", "application/x-www-form-urlencoded"
        .setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36"
        .send argstring
        html.body.innerHTML = .responseText
    End With

    Debug.Print http.responseText

End Sub

There is a text file in the attachment which contains what I've got from the chrome dev tools. I'm afraid I was tracking the right URL to get logged-in, though.

Kenneth Hobson · Oct 9, 2017

If your post has login info, you should obfuscate it.

I used this years ago. They changed their login so it will take two Submit routines. This code is similar to the attachment which has a userform. This concept should be easy to adapt. If not, post back and I will look at it again or your approach. I usually use a winhttp approach to get source code from a website

Code:

Sub Test_LoginYahoo()
    LoginYahoo ThisWorkbook.Worksheets("Main").Range("A1").Value2, _
      ThisWorkbook.Worksheets("Main").Range("A2").Value2
End Sub

' Add references in Tools > References for:
' Microsoft HTML Object Library
' Microsoft Forms 2.0 Object Library
' Microsoft Internet Controls
Sub LoginYahoo(username As String, password As String)
    Const strURL_c As String = "http://mail.yahoo.com"
    Dim objIE As SHDocVw.InternetExplorer
    Dim ieDoc As MSHTML.HTMLDocument
    Dim tbxPwdFld As MSHTML.HTMLInputElement
    Dim tbxUsrFld As MSHTML.HTMLInputElement
    Dim btnSubmit As MSHTML.HTMLInputElement
  
    Excel.Application.Cursor = xlWait
    If InStr(username, "@") = 0 Then username = username & "@yahoo.com"
  
    On Error GoTo Err_Hnd
  
    'Create Internet Explorer Object
    Set objIE = New SHDocVw.InternetExplorer
    'Navigate the URL
    objIE.Navigate strURL_c
    objIE.Visible = False
    'Wait for page to load
    Do Until objIE.ReadyState = READYSTATE_COMPLETE: Loop
    'Do While objIE.Busy Or objIE.ReadyState <> READYSTATE_COMPLETE
    '    DoEvents
    'Loop
    'Set document object
    Set ieDoc = objIE.document
    ieDoc.getElementsByName("passwd").Item(0).Value = password
    ieDoc.getElementsByName("username").Item(0).Value = username
    ieDoc.forms("Login_form").Submit
  
Err_Hnd: '(Fail gracefully)
    objIE.Visible = True
    On Error GoTo 0
    Excel.Application.Cursor = xlDefault
End Sub

shahin · Oct 9, 2017

Thanks a lot Kenneth Hobson, for such an incredible demo. The way you write code is undoubtedly enviable. The thing is I've also written one using IE and it does lead me to get logged-in but I wanted to do the same using xmlhttp request. There should be some way. However, I'm attaching a demo which I've created few hours back using IE. As I've never worked with IE, there might be several flaws, especially the way I used "wait". Here is the attachment (it's a working one). In your spare time, please give me some advice how could I use wait in my scraper to make it more efficient.

shahin · Oct 9, 2017

Oh my god!!! You have already shown how to use that. I got frightened to see userform coz whenever i see it my head spins. I've never learnt vba basics. Thanks again.

Kenneth Hobson · Oct 9, 2017

I'll look at your #1 post method later today. For kicks, I added a few other ideas for the IE method. There are 2 concepts here: (1) loop until a field value is what you sent, and (2) wait until the current url is not what the last one was.

e.g.
This is a bit messy since I included comments FYI and for some debug tests to see if they helped or not and FMOI (for my own information).

Code:

Sub Test_LoginYahoo2()
    LoginYahoo2 "User", "Password"
End Sub

' Add references in Tools > References for:
' Microsoft HTML Object Library
' Microsoft Forms 2.0 Object Library
' Microsoft Internet Controls
Sub LoginYahoo2(username As String, password As String)
    Const strURL_c As String = "http://mail.yahoo.com"
    Dim url As String
    Dim objIE As SHDocVw.InternetExplorer
    Dim ieDoc As MSHTML.HTMLDocument
    Dim tbxPwdFld As MSHTML.HTMLInputElement
    Dim tbxUsrFld As MSHTML.HTMLInputElement
    Dim btnSubmit As MSHTML.HTMLInputElement
   
    Excel.Application.Cursor = xlWait
    If InStr(username, "@") = 0 Then username = username & "@yahoo.com"
   
    On Error GoTo Err_Hnd
   
    'Create Internet Explorer Object
    Set objIE = New SHDocVw.InternetExplorer
    'Navigate the URL
    objIE.navigate strURL_c
    objIE.Visible = False
    'Wait for page to load
    Do Until objIE.readyState = READYSTATE_COMPLETE: Loop
    Set ieDoc = objIE.document
    url = ieDoc.url
   
    ieDoc.getElementsByName("username").Item(0).Value = username
    'Do Until ieDoc.getElementsByName("username").Item(0).Value = username: Loop
    ieDoc.forms("signin").submit
    'ieDoc.getElementById("login-signin").Click
   
    Do Until ieDoc.url <> url: Loop
    'Do Until objIE.readyState = READYSTATE_COMPLETE '4: Loop
    Set ieDoc = objIE.document
   
    'ieDoc.getElementsByName("password").Item(0).Value = password
    'Do Until ieDoc.getElementsByName("password").Item(0).Value = password: Loop
    ieDoc.getElementById("login-passwd").Value = password
    Do Until ieDoc.getElementById("login-passwd").Value = password: Loop
    ieDoc.forms("verifyPassword").submit
    'ieDoc.getElementById("login-signin").Click
   
Err_Hnd: '(Fail gracefully)
    objIE.Visible = True
    On Error GoTo 0
    Excel.Application.Cursor = xlDefault
End Sub

shahin · Oct 9, 2017

Thanks Hobson, for your response. I didn't test your code either but at a glance it seems to be a unique one cause i never come across such usage to deal with time/loading. Hats off to you man. Btw, if I encounter any issues executing your new ideas I'll let you know. Thanks again.

shahin · Oct 9, 2017

Thanks again Hobson, for your idea. You are incredible man!!!!! I tried your method and found it working like never before with IE. Just a little tuning is needed. Perhaps, I failed to follow your instruction properly in the line just after click. I used there "wait" temporarily to make it a go. The rest are just working awesome. This looping concept is just mindblowing. I hope you will find a way to use some other "loop" instead of "wait" just after click.

Here is what I tried with:

Code:

Sub Test_LoginYahoo2()
    yahoo_login "some_username", "some_password"
End Sub


Sub yahoo_login(username As String, password As String)
    Dim IE As New InternetExplorer
    Dim post As Object
 
    Application.ScreenUpdating = False
    With IE
        .Visible = True
        .navigate "https://www.yahoo.com/"
        Do Until .readyState = READYSTATE_COMPLETE: Loop
        .document.getElementById("uh-signin").Click
        'Do Until .document.url <> url: Loop
        Application.Wait (Now + TimeValue("0:00:05"))
        .document.getElementById("login-username").Value = username
        Do Until .document.getElementById("login-username").Value = username: Loop
        .document.getElementById("login-signin").Click
        'Do Until .document.url <> url: Loop
        Application.Wait (Now + TimeValue("0:00:05"))
        .document.getElementById("login-passwd").Value = password
        Do Until .document.getElementById("login-passwd").Value = password: Loop
        .document.getElementById("login-signin").Click
        'Do Until .document.url <> url: Loop
        Application.Wait (Now + TimeValue("0:00:05"))
    End With
    Set post = IE.document.getElementsByClassName("Hidden")(0)
    MsgBox post.innerText
 
    IE.Quit
    Application.ScreenUpdating = False
End Sub

Kenneth Hobson · Oct 9, 2017

Sure, see what I said in post #5, item (2). Then see how I did it in the code.

Look at the 3 parts in the code.

Code:

Dim url AsString
url = ieDoc.url
'after click or submit
Do Until ieDoc.url <> url: Loop

If you have 3rd site to navigate, set url value before and check in loop for <>.

shahin · Oct 10, 2017

Finally, you've made me to be a fan of IE. Thanks Hobson. This is working exceeding my anticipation. I never expected this much from IE. The looping concept you taught me is awesome. Btw, don't forget about xmlhttp when you are completely free. Thankssssssssssss a zillion for walking me through the IE arena with much adoration.

Kenneth Hobson · Oct 10, 2017

That loop method usually works. I just worked on one where Application.Wait was needed.

Where did you get the text for the DevTools? I did not see it in:
https://developers.google.com/web/tools/chrome-devtools/

shahin · Oct 10, 2017

@Kenneth Hobson, did you mean the content of "devtools.txt" (Chrome Developer Tools) I've attached in the first place?

Kenneth Hobson · Oct 10, 2017

shahin · Oct 10, 2017

Just give me some time, I'll come up with an elaborative explanation as to how I got this.

shahin · Oct 10, 2017

Hi there!! One last suggestion I need on IE automation. If I log in for the first time in my account, then second time and then again third time ------, I see at some point that the scraper fails not because it was made inefficiently but because the account is already logged-in in that site and whenever the scraper doesn't find the usual process to log in it fails. My asking is how I can make a "if statement" or something so that the scraper will first look for whether the account is already logged-in. If it is not then it will do the process. Thanks.

This is the optimized version according to your guideline (partial portion of course):

Code:

Sub yahoo_login(username As String, password As String)
    Const strURL As String = "https://www.yahoo.com/"
    Dim IE As New InternetExplorer
    Dim ieDoc As New HTMLDocument
    Dim post As Object, ws As Worksheet
    Dim url As String, link As String, n_url As String
   
    Application.ScreenUpdating = False
    With IE
        .Visible = True
        .navigate strURL
        Do Until .readyState = READYSTATE_COMPLETE: Loop

        Set ieDoc = IE.document
        url = ieDoc.url
       
        .document.getElementById("uh-signin").Click
        Do Until ieDoc.url <> url: Loop
       
        .document.getElementById("login-username").Value = username
        Do Until .document.getElementById("login-username").Value = username: Loop
        .document.getElementById("login-signin").Click
       
        link = ieDoc.url
        Do Until ieDoc.url <> link: Loop

        .document.getElementById("login-passwd").Value = password
        Do Until .document.getElementById("login-passwd").Value = password: Loop
        .document.getElementById("login-signin").Click
       
        n_url = ieDoc.url
        Do Until ieDoc.url <> n_url: Loop

    End With

Thanks once again, Hobson.

shahin · Oct 10, 2017

Hi there!! I've got the issue sorted already I mentioned in my earlier post about. But, the thing is whether the way I used "error handler" is the ideal way. Please take a look whether there is any optimization needed in that error handling part. Here is the full code:

Code:

Sub Test_LoginYahoo2()
    yahoo_login "some_email", "some_pass"
End Sub

Sub yahoo_login(username As String, password As String)
    Const strURL As String = "https://www.yahoo.com/"
    Dim IE As New InternetExplorer
    Dim ieDoc As Object
    Dim post As Object, ws As Worksheet
    Dim url As String, link As String, n_url As String
 
    Application.ScreenUpdating = False
    With IE
        .Visible = True
        .navigate strURL
        Do Until .readyState = READYSTATE_COMPLETE: Loop

        Set ieDoc = IE.document
        url = ieDoc.url
        .document.getElementById("uh-signin").Click
        Do Until ieDoc.url <> url: Loop
        .document.getElementsByClassName("orko-button-primary orko-button")(0).Click
        On Error GoTo handler           '''----------if logged in then go to the next part
        .document.getElementById("login-username").Value = username
        Do Until .document.getElementById("login-username").Value = username: Loop
        .document.getElementById("login-signin").Click
     
        link = ieDoc.url
        Do Until ieDoc.url <> link: Loop

        .document.getElementById("login-passwd").Value = password
        Do Until .document.getElementById("login-passwd").Value = password: Loop
        .document.getElementById("login-signin").Click
     
        n_url = ieDoc.url
        Do Until ieDoc.url <> n_url: Loop
handler:             '''--------------next part starts from here
    End With
    With ThisWorkbook
        Set ws = .Sheets.Add(After:=.Sheets(.Sheets.Count))
        ws.Name = "Yahoo_table"
    End With
    x = 1
    For Page = 1 To 3
        With IE
            .Visible = True
            .navigate "https://football.fantasysports.yahoo.com/f1/69725/players?&sort=AR&sdir=1&status=" & Page & "&pos=O&stat1=S_S_2017&jsenabled=1"
            Do Until .readyState = READYSTATE_COMPLETE: Loop
        End With
     
        Set obj = ieDoc.getElementsByClassName("Table")(0)
        For Each posts In obj.Rows
            For Each post In posts.Cells
                y = y + 1
                Cells(x, y) = post.innerText
            Next post
            y = 0
            x = x + 1
        Next posts
    Next Page
 
    IE.Quit
    Application.ScreenUpdating = True
End Sub

How to get logged-in in yahoo.com?

shahin

Active Member

Attachments

Kenneth Hobson

Active Member

Attachments

shahin

Active Member

Attachments

shahin

Active Member

Kenneth Hobson

Active Member

shahin

Active Member

shahin

Active Member

Kenneth Hobson

Active Member

shahin

Active Member

Kenneth Hobson

Active Member

shahin

Active Member

Kenneth Hobson

Active Member

shahin

Active Member

shahin

Active Member

shahin

Active Member