Imagine following situations as a developer:
- Your task is to import bulk data via REST interfaces
- Your task is to transfer existing images to AWS S3
- Your task is to fetch bulk data from remote servers
What can be done to speed up such processes?
- Chunk the workload into smaller parts
- Use HTTP childs to scale the workload over a cluster of servers
- Control everything from one process
Guzzle is well known as a powerful and stable HTTP client for PHP.
It features a lot of functionality (mostly) on top of cURL, which give us the opportunity to scale out any massive workload with HTTP.
You can send multiple requests concurrently using promises and asynchronous requests.
You can use the GuzzleHttp\Pool object when you have an indeterminate amount of requests you wish to send.
Guzzle clients use a handler and middleware system to send HTTP requests.
See guzzle documentation here and here
Lets pack everything together:
- Concurrent HTTP Calls as asynchronous childs
- Use closures to provide callable
- Use promise from guzzle to control childs
- Pool Concept, allows to set maximum number of allowed childs
- once a child has finished, the next one will start until queue is empty
- Retry Middleware, which allows to retry broken HTTP requests
- Set Timeouts correctly to avoid hanging processes
You will find the code in guzzle-examples, it will produce output like this:
See the different order of the index numbers, the different calls take different time.
Feel free to use the code if you like :)