Expecting someone to help me create a facebook bot

[shahin]

Hi there! Good afternoon. I tried to make a facebook bot to parse the profile links. However, it signs in and parses the content of left-sided bar inconsistently. I can't go further. Could anyone point me in the right direction so that i can rectify my mistakes I've made already in my code and move on. Here is the code:

Sub FBbot()
Dim http As New MSXML2.XMLHTTP60, html As New HTMLDocument
Dim topics As Object, topic As Object
Dim x As Long,strdata As String

x = 2

strdata = "email=sth.com&pass=xxx"
http.Open "POST", "https://www.facebook.com/login.php?login_attempt=1&lwv=110", False
http.setRequestHeader "Content-type", "application/x-www-form-urlencoded"
http.send strdata

html.body.innerHTML = http.responseText

Set topics = html.getElementsByClassName("_li")(0).getElementsByTagName("a")

For Each topic In topics
    Cells(x, 1) = topic.innerText
    x = x + 1
Next topic
End Sub

 

[Hui]

I can't really help but as I rarely scrape web sites and have never scraped facebook

In Facebook right click on what you want and Inspect Element

Make sure that the Topic and Innertext are the correct fields

You may also want to see if there are different view panes/windows that you need to look through (I'm not sure of the correct term)

 

[shahin]

Thanks sir for your kind reply. I'm trying to find that out also where I'm making mistakes.

 

[shahin]

I've got it working with the below code. It is necessary to do one thing manually, though. After running this script, when facebook page opens up, there will be a notification popping up mentioning "Allow" or "Block". Wait a little while to get the buffering done and then Just click on "Allow". That's it. Btw, before running this script make sure "Selenium Type Library" has been added in the reference.

Sub Facebook_crawler()
Dim driver As New WebDriver
Dim posts As Object, post As Object

With driver
    .Start "chrome", "https://www.facebook.com"
    .get "/login.php?login_attempt=1&lwv=111"

    .FindElementByXPath("//*[@id='email']").SendKeys ("someEmail") --Fill in according to your email
    .FindElementByXPath("//*[@id='pass']").SendKeys ("password") --Fill in with your password
    .FindElementByXPath("//*[@id='loginbutton']").Click
    .Wait 800
    .FindElementByXPath("//a[@id='findFriendsNav']").Click
    .Wait 500
End With

For x = 0 To 50
    driver.ExecuteScript "window.scrollTo(0, document.body.scrollHeight);"
    driver.Wait 500
Next x

Set posts = driver.FindElementsByXPath("//div[contains(@class,'friendBrowserNameTitle')]/a")

For Each post In posts
    i = i + 1
    Cells(i, 1) = post.Text
Next post
End Sub

 

[shahin]

Forgot to mention that the above crawler is able to scrape the names of friend request by going to the target page and scrolling down to a certain position.

 

[Hui]

Have you tried this in Excel 2016+

 

[shahin]

Thanks sir Hui, for showing me a new way of doing stuffs. Is it power query? I heard that it can do impossible things. Anyways, I use excel 2013.

 

[Hui]

Have a look just in case it's there

 

[shahin]

Found the solution. Now this scraper is able to harvest all the names from the friend list without manual intervention, as in clicking the "Allow" or "Block" notification. It can automate the whole operation. Don't forget to add "Selenium Type Library" in the reference library before execution. Here is the code:

Sub Test_stth()
Dim driver As New WebDriver

With driver
    .SetPreference "profile.default_content_setting_values.notifications", 2  'This is the line which is able to block the notification
    .Start "chrome", "https://www.facebook.com"
    .get "/login.php?login_attempt=1&lwv=111"

    .FindElementByXPath("//*[@id='email']").SendKeys ("username")     'Input the username
    .FindElementByXPath("//*[@id='pass']").SendKeys ("password")   'Input the password
    .FindElementByXPath("//*[@id='loginbutton']").Click
    .Wait 800
    .FindElementByXPath("//a[@id='findFriendsNav']").Click
    .Wait 500
End With

For x = 0 To 50
    driver.ExecuteScript "window.scrollTo(0, document.body.scrollHeight);"
    driver.Wait 500
Next x

Set posts = driver.FindElementsByXPath("//div[contains(@class,'friendBrowserNameTitle')]/a")

For Each post In posts
    i = i + 1
    Cells(i, 1) = post.Text
Next post
End Sub

<Edited as requested by user>

 

[shahin]

As I don't have enough privilege to rectify a certain line in the pasted code above, I'm pasting below the modified line. ID should be [@id='email'] not [@id='username']. Now I suppose You don't find any trouble getting the harvested results.

.FindElementByXPath("//*[@id='email']").SendKeys ("username")    'Input the username

 

[shahin]

This is the finest approach to get the list of names (can be found pressing "Find Friends" button) from facebook no matter how many scrolls are necessary to reach the bottom of that page in order to get them all.

Sub GetFriendList()
    Dim driver As New ChromeDriver, user As Object, pass As Object
    Dim posts As Object, post As Object, prevlen&, curlen&, R&
   
    With driver
        .SetPreference "profile.default_content_setting_values.notifications", 2  'blocking notification
        .get "https://www.facebook.com/login.php?login_attempt=1&lwv=111"
       
        Set user = .FindElementByXPath("//*[@id='email']", timeout:=2000) 'explicitly wait for 2 seconds
        user.SendKeys ("username")
       
        Set pass = .FindElementByXPath("//*[@id='pass']", timeout:=2000)  'It will wait upto 2 seconds before  throwing timeout exception
        pass.SendKeys ("password")
       
        .FindElementByXPath("//*[@id='loginbutton']", timeout:=2000).Click
        .FindElementByXPath("//a[@id='findFriendsNav']", timeout:=2000).Click
    End With
   
    prevlen = driver.FindElementsByXPath("//div[contains(@class,'friendBrowserNameTitle')]/a").Count
   
    'scroll to the bottom until no scroll is left
   
    Do
        prevlen = curlen
        driver.ExecuteScript "window.scrollTo(0, document.body.scrollHeight);"
        driver.Wait 2000  ''wait for 2 seconds (hardcoded unfortunately)
        Set posts = driver.FindElementsByXPath("//div[contains(@class,'friendBrowserNameTitle')]/a")
        curlen = posts.Count
        If prevlen = curlen Then Exit Do
    Loop

    For Each post In posts
        R = R + 1: Cells(R, 1) = post.Text
    Next post
End Sub