Laravel Jobs and Queues – Configuring, Sending Mail, Dispatching Jobs

It has been a lot of time since I started using Laravel Queues. During earlier days back than learning complex technologies on own was not easy and I somehow managed guts to learn them and apply in real-time projects.

Today I’m happy that I took that decision. While learning Laravel Framework I got introduced to a lot of intermediate and high-level programming concepts and one concept or say feature on this framework caught my attention was Queues because of its ability to run jobs or tasks in the background.

From the day I learnt queues in started to use it in each of the projects I do in laravel.

Table of Contents

What are Laravel Queues?

Consider you have a heavy processing task to be performed by application this may be generating large data for reporting, blasting emails for a large number of users or such notifying people when their task gets completed.

These are some of the examples and as they are time-consuming you cannot execute these task is normal manner such are on button click or page submit. So to properly handling these task laravel provides queue functionality where tasks data is storing in a database and then the queue will automatically detect any jobs and execute them in background.

Create a Fresh New Laravel Project

Laravel provides two different options for creating a fresh project.

Before you execute below command you must navigate to your directory where you would like to create this project and open the terminal from that particular path.

Using Laravel Installer

This installer is downloaded through composer. Before running this command composer must be installed on your computer.

composer global require laravel/installer

Then you can create a new project by typing below command.

laravel new your_project_name

Directly creating a project through composer command

This is another method and I always go through this method because it doesn’t need the additional installer to be installed other than a composer and also I can specify project version at the end.

composer create-project --prefer-dist laravel/laravel your_project_name "6.*"

At the end 6.* specifies which version you would like to install. You can visit Laravel versions and support policy page for more information.

Note

Directly creating project second option will take a few minutes. So don’t cancel the process.

Database Connections

In the root directory of the freshly created project, theirs an .env file which contains all the environment setting for the project.
There is an option for adding database connection which username and password.

DB_CONNECTION=mysql
DB_HOST=127.0.0.1
DB_PORT=3306
DB_DATABASE=laravel_queues
DB_USERNAME=root
DB_PASSWORD=

laravel_queues is our database name and then goto PHPMyAdmin and create a database of laravel_queues name.

Note

For demonstration purpose we are using MySql are database. You more robust performance laravel also provides drivers for Redis, Amazon SQS, Beanstalkd etc for more information you can visit drivers and prerequisites of laravel queues.

 

Queue Configuration and Settings

Since we are telling laravel to use our database as queue driver there is just a simple step to do.

That is to go to .env file and replace below code.

#Replace this line.
QUEUE_CONNECTION=sync 

#With this line.
QUEUE_CONNECTION=database

Note

You can also take a look at queue.php inside config a directory which contains an array of all available drivers.

 

Database Migration

Database migration takes place by creating a migration class inside the folder database/migrations. It contains the schema description of the table and can be migrated to the database.
But before start migration, you must inform laravel to create a migration class for queue table for this use below command.

php artisan queue:table

A migration file is created inside database/migrations folder.

Schema::create('jobs', function (Blueprint $table) {
    $table->bigIncrements('id');
    $table->string('queue')->index();
    $table->longText('payload');
    $table->unsignedTinyInteger('attempts');
    $table->unsignedInteger('reserved_at')->nullable();
    $table->unsignedInteger('available_at');
    $table->unsignedInteger('created_at');
});

Now all these migrations must be migrated to the database for that run below command.

php artisan migrate

Laravel php artisan migrate command output

By switching to PHPMyAdmin you can see that tables are created in the database.

Laravel database after running migrations

For testing, purpose let us also create two new tables they are countries_census, states_census.

Note

countries_census the table holds the total population of the country.
states_census the table holds the total population of each state which is then mapped to the country table through country_id column.

php artisan make:migration create_countries_census_table --create=countries_census

php artisan make:migration create_states_census_table --create=states_census

database/migrations/2019_08_19_000000_create_countries_census_table.php.

<?php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

class create_countries_census_table extends Migration
{
    /**
        * Run the migrations.
        * 
        * @return void
        */
    public function up()
    {
        Schema::create('countries_census', function (Blueprint $table) {
            $table->bigIncrements('country_id');
            $table->text('country_name');
            $table->unsignedInteger('total_population');
            $table->timestamp('updated_at')->nullable();
        });
    }

    /**
        * Reverse the migrations.
        *
        * @return void
        */
    public function down()
    {
        Schema::dropIfExists('countries_census');
    }
}

database/migrations/2019_08_19_000000_create_states_census_table.php.

<?php

use Illuminate\Database\Migrations\Migration;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Support\Facades\Schema;

class create_states_census_table extends Migration
{
    /**
        * Run the migrations.
        * 
        * @return void
        */
    public function up()
    {
        Schema::create('states_census', function (Blueprint $table) {
            $table->bigIncrements('state_id');
            $table->text('state_name');
            $table->unsignedInteger('country_id');
            $table->unsignedInteger('state_population');
        });
    }

    /**
        * Reverse the migrations.
        *
        * @return void
        */
    public function down()
    {
        Schema::dropIfExists('states_census');
    }
}
php artisan migrate

Laravel new tables countries census and states_census created through migration

Creating models for countries_census and states_census tables.

Laravel models are the representation of the database table which contains tables schematic information as well as one table to another table relationship description.

To create model use below command.

php artisan make:model CountryPopulationModel

In app/CountryPopulationModel.php a new model has been created.

<?php

namespace App;

use Illuminate\Database\Eloquent\Model;
use Illuminate\Support\Facades\Log;

class CountryPopulationModel extends Model
{
    protected $table="countries_census";
    protected $primaryKey = 'country_id';
    protected $fillable = ['country_name', 'total_population'];

    public function states()
    {
        return $this->hasMany(CountryStatePopulationModel::class, 'country_id', 'country_id');
    }
}

$table is database table name of than model.
$primaryKey is database table primary.
$fillable is columns of database table except for primary key.

Similarly, let us create a model for states_census.
In app/CountryStatePopulationModel.php.

<?php

namespace App;

use Illuminate\Database\Eloquent\Model;
use Illuminate\Support\Facades\Log;

class CountryStatePopulationModel extends Model
{
    protected $table="states_census";
    protected $primaryKey = 'state_id';
    protected $fillable = ['country_id', 'state_name', 'state_population'];

    public function country()
    {
        return $this->belongsTo(CountryPopulationModel::class, 'country_id', 'country_id');
    }
}

Dummy data is also added into both the tables and before running jobs, these tables look like below.
Laravel countries census table data before running jobs

Creating a controller for handling logic

Command to create a new controller.

php artisan make:controller CountryController

app/Http/Controllers/CountryController.php

<?php

namespace App\Http\Controllers;

use App\CountryPopulationModel;
use App\Jobs\ProcessCountriesPopulation;
use Illuminate\Http\Request;
use Illuminate\Support\Facades\Log;

class CountryController extends Controller
{
    public function process(Request $request){
        
    }
}

Routing

routes/web.php

Route::get('countries-census/process', 'CountryController@process');

Creating New Job for Queue

Use php artisan the command for creating a new job class.

php artisan make:job ProcessCountriesPopulation

ProcessCountriesPopulation class is created inside the application root directory app/Jobs/ProcessCountriesPopulation.php path.
This job class will periodically check for population data for different countries, sum them and update the latest population of the country in countries_census table.

The ProcessCountriesPopulation job class looks like below.

<?php

namespace App\Jobs;

use App\CountryPopulationModel;
use App\CountryStatePopulationModel;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Illuminate\Support\Facades\Log;

class ProcessCountriesPopulation implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;


    /**
        * Create a new job instance.
        *
        * @return void
        */
    public function __construct()
    {
        
    }

    /**
        * Execute the job.
        *
        * @return void
        */
    public function handle()
    {
        
    }
}    

Pre-made methods __construct and handle are available.

Note

__construct() the method is used for passing dynamic data to the job.

handle() the method is executed whenever a new job is dispatched. This dispatch event is listened by command queue:work.

<?php

namespace App\Jobs;

use App\CountryPopulationModel;
use App\CountryStatePopulationModel;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Illuminate\Support\Facades\Log;

class ProcessCountriesPopulation implements ShouldQueue
{
    use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;

    protected $country_census;

    /**
     * Create a new job instance.
     *
     * @return void
     */
    public function __construct(CountryPopulationModel $country)
    {
        Log::info('Entered Job ProcessCountriesPopulation __constructor method');
        $this->country_census = $country;
    }

    /**
     * Execute the job.
     *
     * @return void
     */
    public function handle()
    {
        Log::info('Entered Job ProcessCountriesPopulation handle method');
        $country_states_population_sum =  CountryStatePopulationModel::where(["country_id" => $this->country_census["country_id"]])->sum('state_population');
        
        $country = CountryPopulationModel::find($this->country_census["country_id"]); 
        $country->total_population = $country_states_population_sum;
        $country->save();

        Log::info('Exited from Job ProcessCountriesPopulation handle method');
    }
}

In the above job, the class constructor method __construct(CountryPopulationModel $country) takes an object which is of type CountryPopulationModel as an argument. This object is then assigned to a class variable $this->country_census.
Log::info logs the message in the log file which is placed in the path laravel_queues/storage/logs/laravel.log.

Note

Background logic is always performed in handle() method of job class this place where the jobs are inserted to the job table.

Dispatching Jobs to Queue

app/Http/Controllers/CountryController.php

<?php

namespace App\Http\Controllers;

use App\CountryPopulationModel;
use App\Jobs\ProcessCountriesPopulation;
use Illuminate\Http\Request;
use Illuminate\Support\Facades\Log;

class CountryController extends Controller
{
    public function process(Request $request){
        Log::info('Entered Job CountryController process method');

        $country = CountryPopulationModel::find(101);
        
        ProcessCountriesPopulation::dispatch($country);
        
        Log::info('Exited Job CountryController process method');
    }
}

In ProcessCountriesPopulation::dispatch($country); the static dispatch() method passed $country object to job class constructor.

Running Queue Workers on Development Machine

php artisan queue:work

This command will detect any pending jobs is jobs table and executes them one by one.

After going to route countries-census/process in browser. You can see that a new row is already been inserted in jobs table. And below log message appears in laravel_queues/storage/logs/laravel.log a file.

[2020-04-09 11:48:54] local.INFO: Entered Job CountryController process method  
[2020-04-09 11:48:54] local.INFO: Entered Job ProcessCountriesPopulation __constructor method  
[2020-04-09 11:48:54] local.INFO: Exited Job CountryController process method  
[2020-04-09 11:48:55] local.INFO: Entered Job ProcessCountriesPopulation handle method  
[2020-04-09 11:48:55] local.INFO: Exited from Job ProcessCountriesPopulation handle method  

Jobs Table before running queue.

Laravel job table before running queue work command
When the job is processed queue automatically remove the completed job rows.

Population data inside the database table countries_census is also updated.
Laravel country census table after running queue work command

Processing of queued jobs.
laravel processing of queued jobs in command line

Using Laravel Queues for sending mail

Here you’ll see a simple example of sending an email using queues.

For creating new email class.

php artisan make:mail WelcomeEmail

In laravel_queues/app/Mail/WelcomeEmail.php

public function build()
{
    return $this->from('abc@gmail.com')->view('emails.welcome-email');
}

If you want to send email using SMTP then you must specify below credentials in .env file

MAIL_DRIVER=smtp

MAIL_HOST=smtp.googlemail.com

MAIL_PORT=587

MAIL_USERNAME=*****@gmail.com

MAIL_PASSWORD=

MAIL_ENCRYPTION=tls

For sending emails use Mail class and inside send() method pass the name of the email class you would like to send.

Mail::to("to-email@gmail.com")->send(new WelcomeEmail());

Running Queue Workers on Production Server

Use cron job option provided by servers and set the frequency to each minute. Which means that the cron job will execute the artisan console command schedule method each minute which is placed at the path laravel_queues/app/Console/Kernel.php.

I have created a personalized script while using executing job command inside console which I’ll be sharing with you.

$schedule->command('queue:restart')->everyFifteenMinutes()->withoutOverlapping();

$schedule->command('queue:work --sleep=3 --tries=3')->everyMinute()->sendOutputTo(storage_path() . '/logs/queue-jobs.log')->withoutOverlapping();

First command queue:restart will restart the processing job listeners, every fifteen minutes. The withoutOverlapping method will prevent the same jobs from overlapping with each other.

Laravel has also provided a detailed overview of the Overlapping of tasks.

Caution

Running jobs without overlapping will be draining servers physical memory can also leave server hanged up.

Next command queue:work will listen to new jobs every minute. You can also specify flags which have different purposes.
Flag --sleep will send queue to sleep mode for specified seconds once all jobs are processed.
Flag --tries will attempt to try executing queued for a specified number of times before sending them for failed_jobs.

How to handle failed jobs in Queues?

Queued jobs which were unable to execute even after many attempts are sent to failed_jobs the table. This table consists of information such as connection type, payload, exception information and date-time when a job has failed.

Larael failed queue jobs table

For re-attempting, these failed jobs use below command.

php artisan queue:retry all

Command queue:retry all will retrieve a list of all jobs that are failed and will try to execute them. You can also specify which job to executed by specifying its primary key number like php artisan queue:retry 2 this will execute the job with primary key 2.

List failed jobs

To see all the jobs which are failed use php artisan queue:failed command. This will display all the failed jobs in the terminal.

Laravel show table of failed jobs in terminal

Queueing Jobs by Priorities

Queue executes jobs on default occurrence. But you can specify the priority of a job during dispatching stage.

dispatch((new ProcessCountriesPopulation($country))->onQueue('high'));

Conclusion

Time has come to say goodbye. Coming to the end of this post you have all the basic and intermediate information and examples of working and queues in the development and production stage. This post was regarding Laravel Jobs and Queues – Configuring, Sending Mail, Dispacting Jobs and were are happy to have taken time to go read this post. Support us by sharing this post which will help us grow and comment if you have any doubts we will reach you soon.

Popular Posts