• Hi All

    Please note that at the Chandoo.org Forums there is Zero Tolerance to Spam

    Post Spam and you Will Be Deleted as a User

    Hui...

  • When starting a new post, to receive a quicker and more targeted answer, Please include a sample file in the initial post.

Get data from web site

Hi, chronot!
Don't you need the Carlito's & Mingo's (*) era data since convertibility's law of April 1st, 1992? :p:DD... or should I say :(:mad::eek:?
Regards!
PS: BTW, which Excel version are you using? For 2007/2010 there's the old & standard traditional method of the WebQuery and for 2013 there's the new & easier WEBSERVICE function.

EDITED

(*) PS2: For the community, former Argentine President and Minister of Economy & Finance in the nineties.
 
Last edited:
Hi, chronot!

Give a look at the uploaded file. As for the way that Ámbito's site exposes the data I couldn't happen to use WEBSERVICE function, so going with the old way there's the code:
Code:
Option Explicit

' global constants
Global Const gksWSWork = "Work_2010"

Sub WebQueryAll()
    '
    ' constants
    '  ws & rng
    Const ksWSData = "Hoja_2010"
    Const ksRngData = "Data2010Table"
    Const ksRngWork = "Data2010WorkTable"
    '  dates
    Const ksDateFrom = "11/01/2002"
    Const ksDateTo = "16/01/2014"
    '
    ' declarations
    Dim rngD As Range, rngW As Range
    Dim lRow As Long, iPage As Integer, sPreviousDate As String
    Dim I As Long, bOk As Boolean
    '
    ' start
    Debug.Print "Start at... " & Now()
    '  application
    Application.ScreenUpdating = False
    '  range
    Set rngD = Worksheets(ksWSData).Range(ksRngData)
    With rngD
        If .Rows.Count > 1 Then
            .Range(.Rows(2), .Rows(.Rows.Count)).ClearContents
        End If
    End With
    '  initialize
    lRow = 1
    sPreviousDate = ""
    '
    ' process
    iPage = 1
    bOk = True
    Do
        ' clear
        Set rngW = Worksheets(gksWSWork).Range(ksRngWork)
        With rngW
            If .Rows.Count > 2 Then
                .Range(.Rows(3), .Rows(.Rows.Count)).ClearContents
            End If
        End With
        ' get page
        WebQuery ksDateFrom, ksDateTo, iPage
        ' process page?
        Set rngW = Worksheets(gksWSWork).Range(ksRngWork)
        With rngW
            If .Cells(3, 1).Value <> "" And .Cells(3, 1).Value <> sPreviousDate Then
                sPreviousDate = .Cells(3, 1).Value
                .Range(.Rows(3), .Rows(.Rows.Count)).Copy rngD.Cells(lRow + 1, 1)
                lRow = lRow + .Rows.Count - 2
            Else
                bOk = False
            End If
        End With
        Set rngW = Nothing
        ' cyble
        iPage = iPage + 1
    Loop Until Not bOk
    '
    ' end
    Set rngD = Nothing
    Application.ScreenUpdating = True
    Beep
    Debug.Print "Ended at... " & Now()
    '
End Sub

Sub WebQuery(psDateFrom As String, psDateTo As String, piPage As Integer)
    ' constants
    Const ksURL = "http://www.ambito.com/economia/mercados/monedas/dolar/info/?ric=ARSB=&desde=@#@&hasta=#@#&pag=##"
    Const ksWildcardDateFrom = "@#@"
    Const ksWildcardDateTo = "#@#"
    Const ksWildcardPage = "##"
    ' declarations
    Dim sURL As String
    ' start
    sURL = Replace( _
                Replace( _
                    Replace( _
                        ksURL, _
                        ksWildcardDateFrom, _
                        psDateFrom), _
                    ksWildcardDateTo, _
                    psDateTo), _
                ksWildcardPage, _
                piPage)
    ' process
    Worksheets(gksWSWork).Activate
    With ActiveSheet.QueryTables.Add(Connection:="URL;" & sURL, Destination:=Range("$A$1"))
        .Name = "zz-&pag=" & Format(piPage, "0000")
        .FieldNames = True
        .RowNumbers = False
        .FillAdjacentFormulas = False
        .PreserveFormatting = True
        .RefreshOnFileOpen = False
        .BackgroundQuery = True
        .RefreshStyle = xlOverwriteCells
        .SavePassword = False
        .SaveData = True
        .AdjustColumnWidth = True
        .RefreshPeriod = 0
        .WebSelectionType = xlAllTables
        .WebFormatting = xlWebFormattingNone
        .WebPreFormattedTextToColumns = True
        .WebConsecutiveDelimitersAsOne = True
        .WebSingleBlockTextImport = False
        .WebDisableDateRecognition = False
        .WebDisableRedirections = False
        .Refresh BackgroundQuery:=False
    End With
    ' end
    DoEvents
End Sub

Just advise if any issue.

Regards!

PS: BTW, local support available thru a six-pack of Carlsberg, shared of course. And if it's hard to find them due and thanks actual regulations, at least Imperial Scotch Ale. ;):cool:
 

Attachments

  • Get data from web site (for chronot at chandoo.org).xlsm
    140.7 KB · Views: 19
Hi, chronot!

Do this:
a) Download the file
b) Open it
c) Run the procedure WebQueryAll (Alt-F8, select it, Run/Execute)
d) In a while (1'15" in my PC) it'll end
e) Check 1st worksheet for the whole data

Just advise if any issue.

Regards!
 
Wonderful, amazing SirJB7, thanks a lot, it works!!! :) It took about 5 minutes to end in my computer (Windows 8, Excel 2013, Intel Core i5 3.20 Ghz, 8 Gb ram memory.

I have some questions:
1) Why does the file have 3 sheets?
2) What's the meaning of all the code?
3) Is it possible to adapt this code to get data from other web sites?

I apologize for bothering you with all my questions. This subject is totally new for me and I don't have any idea about WEBSERVICE function.
 
Hi, chronot!

The issue then is it these parts:
Windows 8: downgrade immediately to 7
Excel 2013: if you don't need WEBSERVICE functions and don't use Power Pivot a lot, downgrade more than immediately to 2010 (check the below link)
http://chandoo.org/forum/threads/ch...iving-in-borrowed-time-by-2013-version.14296/

Answers:

1) Sheets no.
1st sheet is for holding data
2nd sheet is used for the web query (it might use another area in 1st one, but I prefer to keep it apart)
3rd sheet was to show why WEBSERVICE didn't work

2) What do you mean with "what's the meaning of all the code?", I don't understand your point. The code is what does the job of looping thru each page and downloading the data.

3) The idea and structure might be the same. Building an URL upon certain parameters and then performing a loop to retrieve each page. But it'll all depend on many factors, being the principal if the site exposes the data with different URL (i.e., 1 per page, or per range) of if it handles everything from the same "page" doing the job via JavaScript or any other language. This method is suitable for the 1s case, for then 2nd there's no much that could be done, AFAIK.

Don't worry, glad you solved it. Thanks for your feedback and for your kind words too. And welcome back whenever needed or wanted.

Regards!
 
Hello, SirJB7!

I really appreciate your answer, infinite thanks!! :)

Unfortunately, I can't downgrade to Windows 7 because I already bought the Windows 8 license and the same with Office 2013 365 Home Premium

What's the meaning of WEBSERVICE function? Is it a function like SUM, COUNTIF, etc?

Regarding question Nº 2, your explanation was great, that was all I wanted to know.

Your code is very revolutionary and it will help me get long historical data in minutes, wonderful!!

Best regards, have a great weekend!
 
Hi, chronot!
Just as a tip, remember that if you enter in an empty cell "=XXX(" (unquoted) and press the "fx" button at the left of the edition bar text box, you'll be prompted with the function wizard which shows every argument, gives a brief description of the function and by the bottom left link lets you access the built-in or online help. It works for any function.
The same from the VBA editor, you can place the cursor (keyboard, so click, don't hover with mouse) on any VBA reserved word and press F1 to access the same type of help.
If my code is so much revolutionary remember to provide it to your boss and let him make it as of its own, while you wait in the 2nd line... in all revolutions all the 1st lines are killed, imprisoned or exiled, hence the followers jump up one level. :) (I don't know if Sun-Tzu thought about this, but if he had, then he'd surely have included it his The Art Of War). :rolleyes:
Regards!
 
QueryTable is an useful object when need to load a table from a web page,
easy coding, far away from piloting IE, the slowest way !

alerte2.gif
But in this case, when dozens of pages must be imported,
QueryTable is slower than using a library like MSXML2

Tests from a laptop computer
i5-3210M 2.50GHz 4Go RAM / Win 7 SP1 / Excel 2003 SP3 / Wi-Fi :

◘ QueryTable code from SirJB7 best time : 9 min 32s, all others attempts reach 10 min …

◘ My below code using MSXML2 needs less than a minute, almost 45 seconds !
Code:
Private Declare Function InternetCheckConnectionA Lib "Wininet" (ByVal SITE$, ByVal one&, _
                                                                ByVal zero&) As Boolean

Function WebOK(Optional ByVal URL$ = "http://www.msn.com") As Boolean
           P& = InStr(9, URL, "/"):  If P Then URL = Left$(URL, P)
        WebOK = InternetCheckConnectionA(URL, 1, 0)
End Function


Sub Demo()
    Dim AR()

    For Each D In [{"DESDE", "HASTA"}]
        If Range(D).Value = "" Then _
           Range(D).Select: MsgBox "No " & D & " !", vbExclamation, "      Web Import": End
    Next

    URL$ = "http://www.ambito.com/economia/mercados/monedas/dolar/info/?ric=ARSB=&desde=" & _
          [DESDE].Text & "&hasta=" & [HASTA].Text & "&pag="

    If WebOK(URL) = False Then MsgBox "No Web !", vbExclamation, "    Web Import": End
    Me.Shapes("GO!").Visible = False
                          M& = Rows.Count - 2
                          C& = Me.UsedRange.Rows.Count
    If C > 2 Then [FECHA].Offset(1).Resize(C - 2, 3).Clear

    Do
        P& = P& + 1:  Application.StatusBar = "Web Import page " & P

        With CreateObject("MSXML2.XMLHTTP")
            .Open "POST", URL & P, False
            .send
            If .Status = 200 Then T$ = .responseText Else T = ""
        End With

        If T > "" Then
            With CreateObject("HTMLFile")
                .Write T:  C = 2

                For Each D In .getElementsByTagName("div")
                    If D.className = "numeros" Then
                        C = (C + 1) Mod 3

                        If C Then
                            AR(C, R&) = Replace(D.innerText, ",", ".")

                        Else
                            R = R + 1:  If R > M Then Exit Do
                            ReDim Preserve AR(0 To 2, 1 To R)
                            AR(0, R) = CLng(DateValue(D.innerText))
                        End If
                    End If
                Next

                C = .GetElementById("nextPG") Is Nothing
            End With
        End If
    Loop Until T = "" Or C

    If R Then
        With [FECHA].Offset(1).Resize(R, 3)
            .Columns(1).NumberFormat = "dd/mm/yyyy "
        .Columns("B:C").NumberFormat = "#,##0.000 "
                              .Value = Application.Transpose(AR)
        End With

    Else
        MsgBox "Compruebe las fechas !", vbExclamation, Space(19) & "Web Import"
    End If

       Application.StatusBar = False
    Me.Shapes("GO!").Visible = True
    End
End Sub
If both of you can upload the attached workbook and test it …
Follow the web page # progression at the bottom in the Status bar.


The very revolutionary way, 3 times faster at least than using MSXML2,
is to not let VBA scans the web pages but proceeds by
treating multi-tasking Windows VBScript independent agents !
But a much hard way coding …

Regards !

__________________________________________________________________
We are all very ignorant, what happens is that not all ignore the same things … (Albert Einstein)
 

Attachments

  • chronot Web Data .xls
    36 KB · Views: 22
Last edited:

Correction line #49 (#50 in the attached file) :​

Code:
                            If R < M Then R = R + 1 Else Exit Do
 
Last edited:
Hi, Marc L!

As usual you don't leave me other option that agreeing with you, but (as b(ut)ob(ut)hc uses to say, there's always a but...t) would you be as kind as running the original macro in an 12Mhz AT-286 (you can use a Rampage Plus memory board if you want) instead of doing it on a 4.77Mhz XT?
d) In a while (1'15" in my PC) it'll end
It took about 5 minutes to end in my computer (Windows 8, Excel 2013, Intel Core i5 3.20 Ghz, 8 Gb ram memory.
The issue then is it these parts:
Windows 8: downgrade immediately to 7
Excel 2013: if you don't need WEBSERVICE functions and don't use Power Pivot a lot, downgrade more than immediately to 2010 (check the below link)
http://chandoo.org/forum/threads/ch...iving-in-borrowed-time-by-2013-version.14296/
◘ QueryTable code from SirJB7 best time : 9 min 32s, all others attempts reach 10 min …
◘ My below code using MSXML2 needs less than a minute, almost 45 seconds !​
Or are you using a Windows 9 alpha version? :p
Nor a Mac? :DD
Regards!
 
Last edited:

Maybe the Wi-Fi lags … I could test on an i3 with Excel 2007 but I don't think it could be faster …
Any time from test with my attached file ?

On another project - easier - I just start to try the Let It Be Paul McCartney famous song way
- yes I know, he was far ahead of its time ! - and the result is, how could I say ? Just … Wow !
The site is pretty long to respond : 32 pages requests using MSXML2 in 127 seconds …
Let It Be way needs only 21 seconds, 6 times faster !

So I wanna know from chronot if my attached file is faster or not on its computer ?

There is still a chance that they will see
There will be an answer, let it be

To be continued …​
 
Last edited:
Hi, Marc L!
I used the MSXML2 many times, here at Chandoo i only remember this one:
http://chandoo.org/forum/threads/specific-data-and-html-beyond-excel-39-s-capabilities.9640/
and it's actually much faster than the Object Browser since it doesn't require an Internet Explorer instance.
http://msdn.microsoft.com/en-us/library/office/aa163921(v=office.10).aspx
http://www.vba-and-excel.com/vba/in...on-from-the-internet-using-the-xmlhttp-object
ReCarlsGardsBerg!
PS: BTW, might it be that I read an Albert quote (that's in my profile) as your signature?
 
Hello MarcL and SirJB7! Thank you very much for all your help!

MarcL: how can I execute the code? I press the GO button but a message pops up: "No Web!".
 

Works on my side !

SirJB7, have you launched it ? Did you have the same warning ?

You could put in comment line #23 If WebOK(URL)
Procedure WebOK checks if computer is connected to web and site URL responds or not …
Useful for laptops Wi-Fi !

Excel version in 32 or 64 bits ?

Did you try it several or just once ?

Regards.​
 

To grab the 101 web pages data of chronot's example, revolutionary
alternative Let It Be way needs only 25 seconds on the same computer
(versus 45 seconds for standard MSXML2 way versus almost 10 minutes for QueryTable through Excel 2003).

I already used it with success in other projects.

But chronot, if my previous code doesn't work in your computer, it will be the same for this one …
Expecting news …​
 
Last edited:
Hi, Marc L!

Did my job and got this printed in the immediate window (5 consecutive executions):
Duration : 27,624s
Duration : 22,873s
Duration : 21,492s
Duration : 26,143s
Duration : 22,783s

An average of 24/25 sec, vs. 1'15" of WebQuery version.

Chapeau, mon ami!

ReCarlsGardsBerg!

PS: Have one on my name... and debit on b(ut)ob(ut)hc account ;P
 
Hello SirJB7!
I have a question about your code: I use it and it works perfectly, but it doesn't import the first data of each page (the 1st line). Do you know why?
I really appreciate all the help, thanks a lot!
 

My code loads exactly the same dates than SirJB7's code !
And I suppose in this case SirJB7 warns me !

So could you give more details ?
 
Hi, chronot!
Are you using the exact same file as I uploaded or you modified it or adapted the code to one of yours? If not the 1st case, please upload you file.
And as Marc L wrote we both retrieve 3013 records.
Regards!

@Marc L
Hi!
You'd bet on it... ;)
Regards!
 
Back
Top