How to integrate Elasticsearch in your Laravel App - 2019 edition

Back-end, Laravel

How to integrate Elasticsearch in your Laravel App – 2019 edition

Tony Messias

Tony Messias

This is a 2019 edition of one of our most popular articles about Laravel and Elasticsearch.

Searching is an important part of applications, and it can be overlooked as a simple task. “Just add some LIKE boolean matches and we are good to go”. Well, while the LIKE clause can be handy, sometimes we have to accommodate a more robust searching system.

One of the most popular ways is using Elasticsearch. It is a very powerful tool and it comes with a variety of useful features and complementary tools. We are going to cover the basics here and leave you with some links for more resources if you want to dig further.

Topics

What is Elasticsearch?

From the official website:

Elasticsearch is the distributed search and analytics engine at the heart of the Elastic Stack. Logstash and Beats facilitate collecting, aggregating, and enriching your data and storing it in Elasticsearch. Kibana enables you to interactively explore, visualize, and share insights into your data and manage and monitor the stack. Elasticsearch is where the indexing, search, and analysis magic happen.

Elasticsearch Introduction

In other words: we can use Elasticsearch for logging (see the Elastic Stack) and for searching. We are only covering the searching usage in this article.

About madewithlove

We are a remote company from Belgium. We love building digital products and the teams to run it. But mainly, we are happy people. Interested in getting to know us better? Take a quick look on our culture pageFeel like applying? We accept open applications from all over the world here.

Basics about Elasticsearch

We used to translate the Relational concepts to Elasticsearch, but now this kind of comparison is out-dated. In order to fully understand the tool, we better start from scratch, no SQL comparison.

First of all, Elasticsearch is document-oriented and talks REST, so it can be used in any language. Now, let’s dive a bit deeper on its basic concepts.

Index and Types

As I said before, Elasticsearch is a document-oriented search-engine. That means we search, sort, filter, etc., documents. A document is represented in JSON format and it holds information that can be indexed. We usually store (aka index, from “to index”) documents with similar mapping structure (fields) together and we call it an index. There can be one index for users, another for articles and another for products, for example.

Inside the index, we can have one or more types. We usually have one type, but having multiple types in an index can be useful sometimes. Let’s say we have a Contact entity which is the parent (inheritance) of Lead and Vendor entities. Although we could store both entities in the same “contacts” type inside the “contacts” index, it might be interesting to store these contacts in separate types inside the “contacts” index, so having “leads” and “vendors” types inside the “contacts” index.

It’s worth saying that Elasticsearch is schema-free but not schema-less (see here), this means that we can index whatever we want and it will figure out the data types, but we can’t have the same field holding different data types. In order to have better query results and avoid unexpected behavior, we better define those data types per field. And stick to them.

Check the docs for more accurate and well-described documentation.

Update: As pointed out by Peter Steenbergen on Twitter, Elasticseach is moving away from types for the next major version (8), so you will only have single-typed indexes in the future. You can read more about it here.

Local environment

It’s likely that you don’t have Elasticsearch running in your local machine. We are going to be using Docker here, but don’t worry, you can run it without Docker by following the official docs, I just wanted to play with Docker a bit more.

Install Docker and docker-compose in your machine. Now that you have it installed, you can run:

docker run -d -e "discovery.type=single-node" \
    -e "bootstrap.memory_lock=true" \
    -p 9200:9200 \
    elasticsearch:6.8.1

If you don’t have an Elasticsearch Docker image in your machine, it will pull one from the Docker registry, so it might take a little while the first time.

  • the -d flag means we want to run it detached, in other words, don’t block the terminal;
  • the -e flag sets some environment variables in the container. We need two here, one for saying that this is a single-node cluster, and another one that enables SWAP when Elasticsearch runs out of memory (needed in my case, you can find more about it here);
  • the -p 9200:9200 param is saying to Docker that we want the port 9200 bound in our localhost:9200 port, so we can access it as if it was on localhost.

If you have any problems with the vm_max_map_count kernel settings, as I did, the docs got you covered here.

If everything went well, you can do a curl request to check if your Elasticsearch server is running:

curl localhost:9200
{
  "name" : "BtziAML",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "qW73OEpQSq-k7uCsc3gCnQ",
  "version" : {
    "number" : "6.8.1",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "1fad4e1",
    "build_date" : "2019-06-18T13:16:52.517138Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

A response like this means we are good to go. If you can’t resolve it, it might be that your Elasticsearch server is still booting, check out if your container is running using docker ps (it should be).

The demo application

The first thing to know is that we have to have DATA to use Elasticsearch, so in this example, we have a seed command that populates the database and, while it does that, it indexes all of the data on Elasticsearch (because of the observer, see below). I’ll show it in a while, first let’s see how we can integrate it with our Eloquent models.

You can create a Laravel app with composer or using the Laravel installer, like so:

$ laravel new es-laravel-example

Now you have a Laravel application ready to go. But since we are going to be using Elasticsearch in this app, we need to pull a Composer dependency before we change anything, so run composer require elasticsearch/elasticsearch inside your new es-laravel-example folder. If you don’t have composer, check it out here.

Then generate the auth scaffolding:

$ php artisan make:auth

If you run php artisan serve and go to localhost:8000 in your browser, you will see the Laravel welcome page:

Laravel welcome page

Let’s get started. We are going to use the concept of articles to demonstrate here. So we need to create an Article model and its migration. We can do so by running: php artisan make:model -mf Article, where the -m flag tells artisan to also create the migration for this model, and the -f creates the model factory.

We have to tweak the migration, there should be a new file inside the database/migrations/ folder called create_articles_table prefixed with a DateTime. Open it and set the fields we are going to use, like so:

<?php

use Illuminate\Support\Facades\Schema;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Database\Migrations\Migration;

class CreateArticlesTable extends Migration
{
    public function up()
    {
        Schema::create('articles', function (Blueprint $table) {
            $table->bigIncrements('id');
            $table->string('title');
            $table->text('body');
            $table->json('tags');
            $table->timestamps();
        });
    }
}

Since we are using the tags field as an array, we need to configure our model to cast it properly. To do so, open the Article.php model that we’ve just created, and configure it so:

<?php

namespace App;

use Illuminate\Database\Eloquent\Model;

class Article extends Model
{
    protected $casts = [
        'tags' => 'json',
    ];
}

We can create a seeder for our model so we have data available. Run php artisan make:seeder ArticlesTableSeeder and open it. Inside its run() method, all we have to do is add these two lines:

<?php

use Illuminate\Database\Seeder;

class ArticlesTableSeeder extends Seeder
{
    public function run()
    {
        DB::table('articles')->truncate();
        factory(App\Article::class)->times(50)->create();
    }
}

After creating the ArticlesTableSeeder, register it in your database/seeds/DatabaseSeeder.php, like this:

public function run()
{
    $this->call(ArticlesTableSeeder::class);
}

The seeder is using Laravel’s Model Factory functionality to create 50 fake articles for us. But we haven’t configured it yet. To do so, open the database/factories/ArticleFactory.php file and add a new factory entry, like so:

<?php

/* @var $factory \Illuminate\Database\Eloquent\Factory */

use App\Article;
use Faker\Generator as Faker;

$factory->define(Article::class, function (Faker $faker) {
    $tags = collect(['php', 'ruby', 'java', 'javascript', 'bash'])
        ->random(2)
        ->values()
        ->all();

    return [
        'title' => $faker->sentence(),
        'body' => $faker->text(),
        'tags' => $tags,
    ];
});

Now we can add a simple view and route so that we can list articles routes/web.php:

@extends('layouts.app')

@section('content')
    <div class="container">
        <div class="card">
            <div class="card-header">
                Articles <small>({{ $articles->count() }})</small>
            </div>
            <div class="card-body">
                @forelse ($articles as $article)
                    <article class="mb-3">
                        <h2>{{ $article->title }}</h2>

                        <p class="m-0">{{ $article->body }}</body>

                        <div>
                            @foreach ($article->tags as $tag)
                                <span class="badge badge-light">{{ $tag}}</span>
                            @endforeach
                        </div>
                    </article>
                @empty
                    <p>No articles found</p>
                @endforelse
            </div>
        </div>
    </div>
@stop

We are using blade here and what this does is loop over a given $articles variable and showing it in the page. You can run yarn to pull your dependencies, or npm install if you don’t have yarn installed. Then run npm run dev (or yarn dev) to build the assets.

After that, it’s time to run our Seeder. However, we haven’t configured our database yet, let’s use SQLite just because it’s simpler, run:

$ touch database/database.sqlite

and edit the .env file to change the DB credentials to:

DB_CONNECTION=sqlite

Remove every other DB_* entry. Now, you have to stop the php artisan serve (if it’s still running) and run it again. After that, the app reloaded the configs. It’s time to run the migrations and seed our database. Run php artisan migrate --seed and then open the app in your browser (at localhost:8000). You should see something like this:

Articles list

Now, let’s implement a search endpoint. At first, we will implement it using plain SQL. We will write a Repository here, it’s useful for fetching data. Our Repository interface would be something like this:

<?php

namespace App\Articles;

use Illuminate\Database\Eloquent\Collection;

interface ArticlesRepository
{
    public function search(string $query = ''): Collection;
}

And the Eloquent implementation would be like:

<?php

namespace App\Articles;

use App\Article;
use Illuminate\Database\Eloquent\Collection;

class EloquentRepository implements ArticlesRepository
{
    public function search(string $query = ''): Collection
    {
        return Article::query()
            ->where('body', 'like', "%{$query}%")
            ->orWhere('title', 'like', "%{$query}%")
            ->get();
    }
}

Now, we can bind the interface in the AppServiceProvider, like so:

<?php

namespace App\Providers;

use App\Articles;
use Illuminate\Support\ServiceProvider;

class AppServiceProvider extends ServiceProvider
{
    /**
     * Register any application services.
     *
     * @return void
     */
    public function register()
    {
        $this->app->bind(
            Articles\ArticlesRepository::class,
            Articles\EloquentRepository::class
        );
    }
}

Cool. Now, let’s create the search route in our routes/web.php file, like so:

@extends('layouts.app')

@section('content')
    <div class="container">
        <div class="card">
            <div class="card-header">
                Articles <small>({{ $articles->count() }})</small>
            </div>
            <div class="card-body">
                <form action="{{ url('search') }}" method="get">
                    <div class="form-group">
                        <input
                            type="text"
                            name="q"
                            class="form-control"
                            placeholder="Search..."
                            value="{{ request('q') }}"
                        />
                    </div>
                </form>
                @forelse ($articles as $article)
                    <article class="mb-3">
                        <h2>{{ $article->title }}</h2>

                        <p class="m-0">{{ $article->body }}</body>

                        <div>
                            @foreach ($article->tags as $tag)
                                <span class="badge badge-light">{{ $tag}}</span>
                            @endforeach
                        </div>
                    </article>
                @empty
                    <p>No articles found</p>
                @endforelse
            </div>
        </div>
    </div>
@stop

Now you can search for something like:

Searching 101.

It works. Nice job! We can finally implement the Elasticsearch version of it.

Integrating Elasticsearch

Since Elasticsearch talks REST, what we’re going to do here is basically hook into the Eloquent models we want to index on it and send some HTTP requests to the Elasticsearch API. The concepts described here I took from a Laracon Talk linked at the bottom. It is using Laravel, but the concepts can be applied to any language/framework.

We are going to be using Model Observers in this example, so we have a regular Eloquent Model, in our case Article. Then we can write a generic pair of trait and observer that will handle indexing for all of our models (the ones that use the trait, of course), so we would have something like:

<?php

namespace App\Search;

use App\Article;
use Elasticsearch\Client;

class ElasticsearchObserver
{
    /** @var \Elasticsearch\Client */
    private $elasticsearch;

    public function __construct(Client $elasticsearch)
    {
        $this->elasticsearch = $elasticsearch;
    }

    public function saved($model)
    {
        $this->elasticsearch->index([
            'index' => $model->getSearchIndex(),
            'type' => $model->getSearchType(),
            'id' => $model->getKey(),
            'body' => $model->toSearchArray(),
        ]);
    }

    public function deleted($model)
    {
        $this->elasticsearch->delete([
            'index' => $model->getSearchIndex(),
            'type' => $model->getSearchType(),
            'id' => $model->getKey(),
        ]);
    }
}

We need to bind this observer into all of our Models that we want to index in Elasticsearch. We can do that by introducing a new Searchable trait. This trait will also provide the methods the observer uses.

<?php

namespace App\Search;

trait Searchable
{
    public static function bootSearchable()
    {
        // This makes it easy to toggle the search feature flag
        // on and off. This is going to prove useful later on
        // when deploy the new search engine to a live app.
        if (config('services.search.enabled')) {
            static::observe(ElasticsearchObserver::class);
        }
    }

    public function getSearchIndex()
    {
        return $this->getTable();
    }

    public function getSearchType()
    {
        if (property_exists($this, 'useSearchType')) {
            return $this->useSearchType;
        }

        return $this->getTable();
    }

    public function toSearchArray()
    {
        // By having a custom method that transforms the model
        // to a searchable array allows us to customize the
        // data that's going to be searchable per model.
        return $this->toArray();
    }
}

We can register our Observer in our model like this:

<?php

namespace App;

use App\Search\Searchable;
use Illuminate\Database\Eloquent\Model;

class Article extends Model
{
    use Searchable;

    protected $casts = [
        'tags' => 'json',
    ];
}

Now whenever we create, update or delete an entity using our Eloquent Article model, it triggers the Elasticsearch Observer to update its data on Elasticsearch. Note that this happens synchronously during the HTTP request, a better way is to use queues and has an Elasticsearch handler that indexes data asynchronously to not slow down the user’s request.

The Elasticsearch Repository

We can now feed Elasticsearch with our data. We can still have that SQL implementation as a backup to fall back in case our Elasticsearch servers crash. In order to do so, we can create another implementation of our Repository interface, like so:

<?php

namespace App\Articles;

use App\Article;
use Elasticsearch\Client;
use Illuminate\Support\Arr;
use Illuminate\Database\Eloquent\Collection;

class ElasticsearchRepository implements ArticlesRepository
{
    /** @var \Elasticsearch\Client */
    private $elasticsearch;

    public function __construct(Client $elasticsearch)
    {
        $this->elasticsearch = $elasticsearch;
    }

    public function search(string $query = ''): Collection
    {
        $items = $this->searchOnElasticsearch($query);

        return $this->buildCollection($items);
    }

    private function searchOnElasticsearch(string $query = ''): array
    {
        $model = new Article;

        $items = $this->elasticsearch->search([
            'index' => $model->getSearchIndex(),
            'type' => $model->getSearchType(),
            'body' => [
                'query' => [
                    'multi_match' => [
                        'fields' => ['title^5', 'body', 'tags'],
                        'query' => $query,
                    ],
                ],
            ],
        ]);

        return $items;
    }

    private function buildCollection(array $items): Collection
    {
        $ids = Arr::pluck($items['hits']['hits'], '_id');

        return Article::findMany($ids)
            ->sortBy(function ($article) use ($ids) {
                return array_search($article->getKey(), $ids);
            });
    }
}

We opted for performing the search on Elasticsearch, and then perform a findMany SQL search with the items that returned from the search. In previous versions of this article, I’ve covered another where we hydrated the model instances from the indexed data. But I find this mixed Elasticsearch+SQL approach easier and less error-prone since we can opt to only index the searchable data instead of all the model’s attributes.

The trick to switching the repository is to replace the binding in the ServiceProvider, like so:

<?php

namespace App\Providers;

use App\Articles;
use Illuminate\Support\ServiceProvider;

class AppServiceProvider extends ServiceProvider
{
    /**
     * Register any application services.
     *
     * @return void
     */
    public function register()
    {
        $this->app->bind(Articles\ArticlesRepository::class, function () {
            // This is useful in case we want to turn-off our
            // search cluster or when deploying the search
            // to a live, running application at first.
            if (! config('services.search.enabled')) {
                return new Articles\EloquentRepository();
            }

            return new Articles\ElasticsearchRepository(
                $app->make(Client::class)
            );
        });
    }
}

Whenever we request an ArticlesRepository interfaced object from the IoC container, it will actually give an ElasticsearchRepository instance if it’s enabled otherwise it will fall back to the Eloquent version of it.

We need to do some customizations to configure the Elasticsearch client, we can bind it in the AppServiceProvider or create a new one, I’m going to use the existing AppServiceProvider, so something like:

<?php

namespace App\Providers;

use App\Articles;
use Elasticsearch\Client;
use Elasticsearch\ClientBuilder;
use Illuminate\Support\ServiceProvider;

class AppServiceProvider extends ServiceProvider
{
    /**
     * Register any application services.
     *
     * @return void
     */
    public function register()
    {
        $this->app->bind(Articles\ArticlesRepository::class, function () {
            // This is useful in case we want to turn-off our
            // search cluster or when deploying the search
            // to a live, running application at first.
            if (! config('services.search.enabled')) {
                return new Articles\EloquentRepository();
            }

            return new Articles\ElasticsearchRepository(
                $app->make(Client::class)
            );
        });

        $this->bindSearchClient();
    }

    private function bindSearchClient()
    {
        $this->app->bind(Client::class, function ($app) {
            return ClientBuilder::create()
                ->setHosts($app['config']->get('services.search.hosts'))
                ->build();
        });
    }
}

Now that we have the code almost ready, we need to finish the configuration. You might have noticed the usages of the config helper method in some places in the implementation. That loads the configuration files data. Here is the configuration I used in the config/services.php:

<?php

return [
    // ...
    'search' => [
        'enabled' => env('ELASTICSEARCH_ENABLED', false),
        'hosts' => explode(',', env('ELASTICSEARCH_HOSTS')),
    ],
];

We set the configuration here and tell Laravel to check the environment variables to find our configuration. We can set it locally in our .env file, like so:

ELASTICSEARCH_ENABLED=true
ELASTICSEARCH_HOSTS="localhost:9200"

We are exploding the hosts here to allow passing multiple hosts using a comma-separated list, but we are not using that at the moment. If you have your php server running, don’t forget to reload it so it fetches the new configs. After that, we need to populate Elasticsearch with our existing data. To do so, we are going to need a custom artisan command. Create one using php artisan make:command ReindexCommand (see code below). This command will also be really useful later on if we come to change the schemas of our Elasticsearch indexes, we could change it to drop the indexes and reindex every piece of data we have (or using aliases for a more zero-downtime approach).

Create the CLI command running:

$ php artisan make:command ReindexCommand --command="search:reindex"

Now, open it up and edit it, like so:

<?php

namespace App\Console\Commands;

use App\Article;
use Elasticsearch\Client;
use Illuminate\Console\Command;

class ReindexCommand extends Command
{
    /**
     * The name and signature of the console command.
     *
     * @var string
     */
    protected $signature = 'search:reindex';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Indexes all articles to Elasticsearch';

    /** @var \Elasticsearch\Client */
    private $elasticsearch;

    /**
     * Create a new command instance.
     *
     * @return void
     */
    public function __construct(Client $elasticsearch)
    {
        parent::__construct();

        $this->elasticsearch = $elasticsearch;
    }

    /**
     * Execute the console command.
     *
     * @return mixed
     */
    public function handle()
    {
        $this->info('Indexing all articles. This might take a while...');

        foreach (Article::cursor() as $article)
        {
            $this->elasticsearch->index([
                'index' => $article->getSearchIndex(),
                'type' => $article->getSearchType(),
                'id' => $article->getKey(),
                'body' => $article->toSearchArray(),
            ]);

            $this->output->write('.');
        }

        $this->info('\nDone!');
    }
}

Now, we can run this command to feed our Elasticsearch server with data:

$ php artisan search:reindex
Indexing all articles. This might take a while...
..................................................
Done!

Now, restart your artisan serve (so it reloads the config), and search. You should see something like this:

Searching with Elasticsearch

We could have achieved a similar result with plain SQL. Yes, we could. But Elasticsearch brings other toys to the table. Let’s say, for instance, that you care more about matches in the title field than any other field and you have some tags searching, like so:

Searching for tags

If you check it out, each of the results has either PHP or Javascript or both tags. Now, let’s define that relevance rule we got about the title field:

<?php

'query' => [
    'multi_match' => [
        'fields' => ['title^5', 'body', 'tags'],
        'query' => $query,
    ],
],

We are defining here that the matches in title field are 5 times more relevant than the other fields. If you reload, nothing happens. But watch now:

Search where title has more relevance.

The first match doesn’t have the right tags, but the title matches with the last term we used, so it boosts up. Cool, we only had to do a few configuration changes, right?

Defining relevance is a very sensitive topic. It might need some meetings and discussions, as well as prototypes before you and your team can decide on what to use.

Wrapping up

We covered the basics here and how to integrate your Laravel app with Elasticsearch. If you want to know more about how to improve your queries, we got you covered! Check out these posts Basic understanding of text search in Elasticsearch and also How to build faceted search with facet counters using Elasticsearch.

Also, did you know that Laravel has its own full-text search official package? It’s called Laravel Scout, it supports Algolia out-of-the-box and you can write custom drivers. It seems to be only for full-text search, though. In case you need to do some fancy searches with aggregations, for example, you can write your own integration. There are packages out there to help you, check out our elasticsearcher package, for example.

You can check this repository for an example of what I’m describing here.

See you next time!

Hire Tony as a speaker?

Contact us
Tony Messias

Tony Messias

Also known as ‘that cat man’ or ‘that guy working with Kurt Cobain looking over his shoulder all the time’, this Brazilian developer never stops learning. Because time is money, he never reads just one book at a time. Why take it easy, when difficult is an extra option? Tony is currently waiting for his wife to finish college. After that, they might hit the road for a couple of months and explore the world in search of paradise.