curl_multi_close
With the current examples using curl-multi-exec is still not much better then just using file_get_contents. Your script has to needlessly wait at least the maximum of all connection times (instead of the sum). A better solution would be to run the connections as another thread in parallel to other parts of the script:
<?php
$cm = new CurlManager();
$handle1 = $cm->register($url1);
$handle2 = $cm->register($url2);
$cm->start(); # start connections
$result1 = $cm->result($handle1); # surely NULL because connection has not been finished
sleep($seconds); # do something else (including use of other CurlManagers!)
$result1 = $cm->result($handle1); # maybe NULL if connection takes longer then $seconds
$cm->finish(); # wait if some connections are still open
$result1 = $cm->result($handle1);
$result2 = $cm->result($handle2);
?>
But I am not sure whether such a CurlManger is possible, especially if you may want to use multiple of them (for instance because some URLs are not known at the start of the script). It's a pain that PHP does not support real threads.
curl_multi_exec
(PHP 5)
curl_multi_exec — 現在の cURL ハンドルから、サブ接続を実行する
説明
int curl_multi_exec
( resource $mh
, int &$still_running
)
スタック内の各ハンドルを処理します。 このメソッドは、ハンドルがデータの読み書きを要するかどうかにかかわらずコール可能です。
返り値
cURL 定義済み定数 で定義された cURL コードを返します。
注意: これは、マルチスタック全体に関するエラーのみを返します。この関数が CURLM_OK を返したとしても、各転送で個別にエラーが発生している可能性があります。
例
例1 curl_multi_exec() の例
この例は、ふたつの cURL ハンドルを作成し、それをマルチハンドルに追加して並列実行します。
<?php
// cURL リソースを作成します
$ch1 = curl_init();
$ch2 = curl_init();
// URL およびその他適切なオプションを設定します。
curl_setopt($ch1, CURLOPT_URL, "http://lxr.php.net/");
curl_setopt($ch1, CURLOPT_HEADER, 0);
curl_setopt($ch2, CURLOPT_URL, "http://www.php.net/");
curl_setopt($ch2, CURLOPT_HEADER, 0);
// マルチ cURL ハンドルを作成します
$mh = curl_multi_init();
// ふたつのハンドルを追加します
curl_multi_add_handle($mh,$ch1);
curl_multi_add_handle($mh,$ch2);
$active = null;
// ハンドルを実行します
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
while ($active && $mrc == CURLM_OK) {
if (curl_multi_select($mh) != -1) {
do {
$mrc = curl_multi_exec($mh, $active);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}
}
// ハンドルを閉じます
curl_multi_remove_handle($mh, $ch1);
curl_multi_remove_handle($mh, $ch2);
curl_multi_close($mh);
?>
add a note
User Contributed Notescurl_multi_exec
jakob dot voss at gbv dot de
14-Feb-2009 01:10
14-Feb-2009 01:10
manixrock-~-gmail-~-com
06-Feb-2009 11:11
06-Feb-2009 11:11
I noticed some may still have a bit of a trouble using mcurl to continuously download stuff (adding and removing stuff from a queue) despite my earlier example, so here's a more complete example:
<?php
// Urls to download
$urls = array();
for ($i=0; $i<20; $i++)
$urls[] = 'http://www.example.com/search?q='.base_convert(mt_rand(0,999999), 10, 36);
$threads = 5;
$timeout = 30;
$mcurl = curl_multi_init();
$threadsRunning = 0;
$urls_id = 0;
for(;;) {
// Fill up the slots
while ($threadsRunning < $threads && $urls_id < count($urls)) {
echo 'Adding download link: '.$urls[$urls_id].'<br>';
flush();
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_URL, $urls[$urls_id++]);
curl_multi_add_handle($mcurl, $ch);
$threadsRunning++;
}
// Check if done
if ($threadsRunning == 0 && $urls_id >= count($urls))
break;
// Let mcurl do it's thing
curl_multi_select($mcurl);
while(($mcRes = curl_multi_exec($mcurl, $mcActive)) == CURLM_CALL_MULTI_PERFORM) usleep(100000);
if($mcRes != CURLM_OK) break;
while($done = curl_multi_info_read($mcurl)) {
$ch = $done['handle'];
$done_url = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
$done_content = curl_multi_getcontent($ch);
if(curl_errno($ch) == 0) {
$count = (preg_match('#of about <b>([^>]+)</b>#i', $done_content, $g) ? $g[1] : 'many');
echo "Link <a href='$done_url'>$done_url</a> found $count links.<br>\n";
flush();
} else {
echo "Link <a href='$done_url'>$done_url</a> failed: ".curl_error($ch)."<br>\n";
flush();
}
curl_multi_remove_handle($mcurl, $ch);
curl_close($ch);
$threadsRunning--;
}
}
curl_multi_close($mcurl);
echo 'Done.';
?>
[EDITED BY danbrown AT php DOT net: Edited with RFC 2606 in mind.]
jorge dot hebrard at gmail dot com
25-Jan-2009 05:40
25-Jan-2009 05:40
Based on the below comments, I made a simple class.
See note below code.
<?php
/**
* Multiple Curl Handlers
* @author Jorge Hebrard ( jorge.hebrard@gmail.com )
**/
class curlNode{
static private $listenerList;
private $callback;
public function __construct($url){
$new =& self::$listenerList[];
$new['url'] = $url;
$this->callback =& $new;
}
/**
* Callbacks needs 3 parameters: $url, $html (data of the url), and $lag (execution time)
**/
public function addListener($callback){
$this->callback['callback'] = $callback;
}
/**
* curl_setopt() wrapper. Enjoy!
**/
public function setOpt($key,$value){
$this->callback['opt'][$key] = $value;
}
/**
* Request all the created curlNode objects, and invoke associated callbacks.
**/
static public function request(){
//create the multiple cURL handle
$mh = curl_multi_init();
$running=null;
# Setup all curl handles
# Loop through each created curlNode object.
foreach(self::$listenerList as &$listener){
$url = $listener['url'];
$current =& $ch[];
# Init curl and set default options.
# This can be improved by creating
$current = curl_init();
curl_setopt($current, CURLOPT_URL, $url);
# Since we don't want to display multiple pages in a single php file, do we?
curl_setopt($current, CURLOPT_HEADER, 0);
curl_setopt($current, CURLOPT_RETURNTRANSFER, 1);
# Set defined options, set through curlNode->setOpt();
if (isset($listener['opt'])){
foreach($listener['opt'] as $key => $value){
curl_setopt($current, $key, $value);
}
}
curl_multi_add_handle($mh,$current);
$listener['handle'] = $current;
$listener['start'] = microtime(1);
} unset($listener);
# Main loop execution
do {
# Exec until there's no more data in this iteration.
# This function has a bug, it
while(($execrun = curl_multi_exec($mh, $running)) == CURLM_CALL_MULTI_PERFORM);
if($execrun != CURLM_OK) break; # This should never happen. Optional line.
# Get information about the handle that just finished the work.
while($done = curl_multi_info_read($mh)) {
# Call the associated listener
foreach(self::$listenerList as $listener){
# Strict compare handles.
if ($listener['handle'] === $done['handle']) {
# Get content
$html = curl_multi_getcontent($done['handle']);
# Call the callback.
call_user_func($listener['callback'],
$listener['url'],
$html,(microtime(1)-$listener['start']));
# Remove unnecesary handle (optional, script works without it).
curl_multi_remove_handle($mh, $done['handle']);
}
}
}
# Required, or else we would end up with a endless loop.
# Without it, even when the connections are over, this script keeps running.
if (!$running) break;
# I don't know what these lines do, but they are required for the script to work.
while (($res = curl_multi_select($mh)) === 0);
if ($res === false) break; # Select error, should never happen.
} while (true);
# Finish out our script ;)
curl_multi_close($mh);
}
}
$curlGoogle = new curlNode('http://www.google.com/');
$curlGoogle->setOpt(CURLOPT_HEADER, 0);
$curlGoogle->addListener('callbackGoogle');
$curlMySpace = new curlNode('http://www.myspace.com/');
$curlMySpace->addListener('callbackMySpace');
curlNode::request();
?>
NOTE: I think curl has a bug managing local files, when multiple handles are set.
I tested with a local file, giving me 0.1~ sec. latency.
Then I added another handle, google. With this new addition, my original file had a latency of 1.0~ sec. latency!!
But with external sites (try Yahoo and Google), the latency is not a problem. Google alone gives me 1.7~ sec. latency, and with Yahoo, Google gives me 1.8~ sec. latency.
If anyone can explain, I will be very granted.
manixrock-~-gmail-~-com
07-Dec-2008 11:13
07-Dec-2008 11:13
For anyone trying to dynamically add links to mcurl and removing them as they complete (for example to keep a constant of 5 links downloading), I found that the following code works:
<?php
$mcurl = curl_multi_init();
for(;;) {
curl_multi_select($mcurl);
while(($mcRes = curl_multi_exec($mcurl, $mcActive)) == CURLM_CALL_MULTI_PERFORM);
if($mcRes != CURLM_OK) break;
while($done = curl_multi_info_read($mcurl)) {
// code that parses, adds or removes links to mcurl
}
// here you should check if all the links are finished and break, or add new links to the loop
}
curl_multi_close($mcurl);
?>
You should note the return value of curl_multi_select() is ignored. I found that if you do the curl_multi_exec() loop only if the return value of curl_multi_select() is != -1 then new links added to mcurl are ignored. This may be a bug. Hope this saves someone some time.
Jimmy Ruska
13-Sep-2008 06:56
13-Sep-2008 06:56
> replying to viczbk.ru
Just sharing my attempt at it.
> while (($res = curl_multi_select($mh)) === 0) {};
This worked on my windows computer (php 5.2.5) but when I ran the curl program in my new centOS server (php 5.1.6) the function never updates unless curl_multi_exec() is added to the loop, taking away the point of using it to save cycles. curl_multi_select() also allows you to set a timeout but it doesn't seem to help, then again, don't see why people wouldn't use it anyway.
Even when curl_multi_select($mh) does work there's no way to know which of the sockets updated in status or if they've received partial data, completely finished, or just timed out. It's not reliable if you want to remove / add data as soon as things finish. Try calling just example.com or some other website with very little data. curl_multi_select($mh) can send non 0 value a couple of thousand times before finishing. Lazily adding a usleep(25000) or some minimal amount can also help not waste cycles.
viczbk.ru
09-Feb-2008 04:04
09-Feb-2008 04:04
http://curl.haxx.se/libcurl/c/libcurl-multi.html
"When you've added the handles you have for the moment (you can still add new ones at any time), you start the transfers by call curl_multi_perform(3).
curl_multi_perform(3) is asynchronous. It will only execute as little as possible and then return back control to your program. It is designed to never block. If it returns CURLM_CALL_MULTI_PERFORM you better call it again soon, as that is a signal that it still has local data to send or remote data to receive."
So it seems the loop in sample script should look this way:
<?php
$running=null;
//execute the handles
do {
while (CURLM_CALL_MULTI_PERFORM === curl_multi_exec($mh, $running));
if (!$running) break;
while (($res = curl_multi_select($mh)) === 0) {};
if ($res === false) {
echo "<h1>select error</h1>";
break;
}
} while (true);
?>
This worked fine (PHP 5.2.5 @ FBSD 6.2) without running non-blocked loop and wasting CPU time.
However this seems to be the only use of curl_multi_select, coz there's no simple way to bind it with other PHP wrappers for select syscall.
robert dot reichel at boboton dot com
20-Oct-2007 12:54
20-Oct-2007 12:54
I was testing PHP code provided by dtorop933@gmail.com in curl_multi_exec section of PHP Manual.
The part of the code '$err = curl_error($conn[$i])' should return error message for each cURL session, but it does not.
The function curl_error() works well with the curl_exec(). Is there any other solution for getting session error message with curl_multi_exec() or there is a bug in cURL library.
The script was tested with Windows XP and PHP-5.2.4
shichuanr at gmail dot com
06-Sep-2007 12:12
06-Sep-2007 12:12
For people who have problem running the above example script(Example 425. curl_multi_exec()), you can alter it a little bit to make it work properly by replacing the existing chunk of code with the one below:
<?php
// create both cURL resources
$ch1 = curl_init();
$ch2 = curl_init();
// set URL and other appropriate options
curl_setopt($ch1, CURLOPT_URL, "http://www.google.com/");
curl_setopt($ch1, CURLOPT_HEADER, 0);
curl_setopt($ch2, CURLOPT_URL, "http://www.php.net/");
curl_setopt($ch2, CURLOPT_HEADER, 0);
//create the multiple cURL handle
$mh = curl_multi_init();
//add the two handles
curl_multi_add_handle($mh,$ch1);
curl_multi_add_handle($mh,$ch2);
$running=null;
//execute the handles
do {
curl_multi_exec($mh,$running);
} while ($running > 0);
//close the handles
curl_multi_remove_handle($mh,$ch1);
curl_multi_remove_handle($mh,$ch1);
curl_multi_close($mh);
?>
substr("iscampifriese",1,11) dot " at beer dot com"
04-Jul-2007 01:10
04-Jul-2007 01:10
If you are using mulit handles and you wish to re-execute any of them (if they timed out or something), then you need to remove the handles from the mulit-handle and then re-add them in order to get them to re-execute. Otherwise cURL will just give you back the same results again without actually retrying.