Fetching Functions
- Ray Pearce (Unlicensed)
$t→fetch()
Given a URL and post data, the fetch function makes a HTTP request and sets $t->result to be the resulting source code.
By convention we work with a string, $t, applying mADL functions to it. In the fetch function we load the string $t with the page source or structured data (XML or such like) that is to be interpreted.
The fetch may be a request for a web page from which the required information will be “parsed” into variables for display or further use, or may be a call to a web API
which returns structured data (eg. XML) without any formatting information. Regardless of the format of the return data, the correct fetch must be constructed.
Parameters | An array of options including the following. |
|---|---|
| $url | A String containing the requested URL. |
| $post | An Array or String containing post data or array. |
| $headers | An Array or String containing page header type. |
| $referer | An Array or String containing url or page to re-direct to. |
| $header_in_output=false | A boolean value that defines if the header is included in the output. |
| $timeout_second=30 | a numeric value for CURLOPT_CONNECTTIMEOUT. |
| $curlOpts = array() | key-value pair for curl options |
Return Values | |
|---|---|
| A String. All results from |
$t->refresh( )
This function resets $t->result to be the value of the previous fetch request.
$t->fetch("http://blinkmobile.com.au/");
if($t->contains("Logged Successful")){
// do something
}
// read more data from scrap
$t->refresh( );
return $t->result;
Return Values | |
|---|---|
| A String. Returns the last requested source into $t->result |
$t->submitForm( )
Often known as a "PassThru"... On occasions when you are retrieving a page source into an Interaction you may require authorisation from a login form.
SubmitForm function can be used to 1. Fetch the form at the requested url. 2. Submit the required parameters 3. Return the results of the fetch for additional formatting.
Parameters | |
|---|---|
| $mystring | A String. The reference to the form name. 0 can be used instead to select the first form found, 1 then next and so on. |
| $myarray | As Array of values to be passed to the form. |
Return Values | |
|---|---|
| A String. Returns requested source into $t->result |
$t->fetch("[http://support.blinkmobile.com.au/");]
$t->submitForm( 0,
array(
'username'=>'YourUserName',
'passwd'=>'YourPassword'
)
);
return $t->result;
// Page source returned here for additional processing
$t→fetch_info[]
Provides access to the cURL info about the most recent $t->fetch() that was performed. See http://php.net/manual/en/function.curl-getinfo.php
$t->fetch('http://www.google.com');echo $t->fetch_info['http_code']; // returns '200'
Using DOMDocument To Display For Fetch Results
Often you may come up against a web page which is hard to scrape using the standard mADL builder method on $t->rows( )
In this case we are looking at retrieving all the hyperlinks from a web directory page but it is dynamically build and we cannot capture a regular expression pattern for the fetch.
You can use The DOMDocument class from php to resolve this.
You still use $t->Fetch() but you add the result to an array and then iterate the array to a screen output.
$t->fetch("http://partystarter.com.au/browse_categories.php?id=2013");
$wsDom = new DOMDocument();
$wsDom->loadHTML($t->result);
foreach($wsDom->getElementsByTagName('a') as $thisLink) {
if ($thisLink->getAttribute('href')!=""){
echo "<a href=\" ".$thisLink->getAttribute('href') ."
\">".$thisLink->getAttribute('title')."</a><br>";
};