Skip to content
This repository was archived by the owner on Nov 19, 2024. It is now read-only.

Commit 1dbba4c

Browse files
doboothdobooth
authored
Merge pull request #8517 from magento-performance/MCP-88
Update indexing.md
2 parents bd9b0a9 + d357710 commit 1dbba4c

File tree

1 file changed

+239
-1
lines changed

1 file changed

+239
-1
lines changed

‎src/guides/v2.4/extension-dev-guide/indexing.md

Lines changed: 0 additions & 1 deletion
This file was deleted.

‎src/guides/v2.4/extension-dev-guide/indexing.md

Lines changed: 239 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,239 @@
1+
---
2+
group: php-developer-guide
3+
title: Indexing overview
4+
---
5+
6+
Indexing is how Magento transforms data such as products and categories, to improve the performance of your [storefront](https://glossary.magento.com/storefront). As data changes, the transformed data must be updated or reindexed. Magento has a very sophisticated architecture that stores lots of merchant data (including [catalog](https://glossary.magento.com/catalog) data, prices, users, and stores) in many database tables. To optimize storefront performance, Magento accumulates data into special tables using indexers.
7+
8+
For example, if you change the price of an item from $4.99 to $3.99. Magento must reindex the price change to display it on your storefront.
9+
10+
Without indexing, Magento would have to calculate the price of every product on the fly, taking into account [shopping cart](https://glossary.magento.com/shopping-cart) price rules, bundle pricing, discounts, tier pricing, etc. Loading the price for a product would take a long time, possibly resulting in cart abandonment.
11+
12+
## Indexing terminology
13+
14+
Dictionary
15+
: Original data entered to the system. Dictionaries are organized in <a href="http://en.wikipedia.org/wiki/Database_normalization" target="_blank">normal form</a> to facilitate maintenance (updating the data).
16+
17+
Index
18+
: Representation of the original data for optimized reading and searching. Indexes can contain results of aggregations and various calculations. Index data can be always re-created from a dictionary using a certain algorithm.
19+
20+
Indexer
21+
: Object that creates an index.
22+
23+
### Create custom indexers
24+
25+
Magento contains several indexers out of the box, but you might want to add your own if your customization requires data searches, which are not optimized by the Magento default indexers.
26+
27+
This topic provides a high level description of how indexing is implemented from a developer's point of view, and practical advice for how to add your own indexer.
28+
29+
## How Magento implements indexing
30+
31+
The following components are involved in the indexing process:
32+
33+
<table>
34+
<tbody>
35+
<tr>
36+
<th>Component</th>
37+
<th>Description</th>
38+
</tr>
39+
<tr>
40+
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/Indexer" target="_blank">Magento_Indexer</a></td>
41+
<td>Implements:
42+
<ul>
43+
<li>indexer declaration</li>
44+
<li>indexer running</li>
45+
<li>indexer running mode configuration</li>
46+
<li>indexer status</li>
47+
</ul>
48+
</td>
49+
</tr>
50+
<tr>
51+
<td><a href="{{ site.mage2bloburl }}/{{ page.guide_version }}/lib/internal/Magento/Framework/Mview" target="_blank">Magento\Framework\Mview</a></td>
52+
<td>
53+
<ul>
54+
<li>Allows tracking database changes for a certain <a href="https://glossary.magento.com/entity" target="_blank">entity</a> (product, <a href="https://glossary.magento.com/category" target="_blank">category</a>, etc.) and running change handler.</li>
55+
<li>Emulates the <a href="http://en.wikipedia.org/wiki/Materialized_view" target="_blank">materialized view</a> technology for MySQL using triggers and separate materialization process (provides executing <a href="https://glossary.magento.com/php" target="_blank">PHP</a> code instead of SQL queries, which allows materializing multiple queries).</li>
56+
</ul>
57+
</td>
58+
</tr>
59+
</tbody>
60+
</table>
61+
62+
{:.bs-callout-warning}
63+
`Magento_Indexer` replaces the Magento 1.x `Magento_Index` module. Use `Magento_Indexer` for all new development.
64+
65+
### Indexing types
66+
67+
Each index can perform the following types of reindex operations:
68+
69+
* Full reindex, which means rebuilding all the indexing-related database tables
70+
71+
Full reindexing can be caused by a variety of things, including creating a new web store or new customer group.
72+
73+
You can optionally fully reindex at any time using the [command line]({{ page.baseurl }}/config-guide/cli/config-cli-subcommands-index.html).
74+
75+
* Partial reindex, which means rebuilding the database tables only for the things that changed (like changing a single product attribute or price)
76+
77+
The type of reindex performed in each particular case depends on the type of changes made in the dictionary or in the system. This dependency is specific for [each indexer](#m2devgde-indexing-outofbox).
78+
79+
The following figure shows the logic for partial reindexing.
80+
81+
![Partial indexing workflow]({{ site.baseurl }}/common/images/index_indexers_flow.png){:width="300px"}
82+
83+
### Indexer status {#m2devgde-indexing-status}
84+
85+
Depending on whether index data is up to date, an indexer status value is one of the following:
86+
87+
Database Status|Admin Status|Description
88+
`valid`|Ready|Data is synchronized, no reindex required
89+
`invalid`|Reindex Required|The original data was changed, the index should be updated
90+
`working`|Processing|Indexing is in progress
91+
92+
The database status can be seen when viewing the SQL table `indexer_state`.
93+
The admin status can be seen when viewing the indexer grid in Magento Admin or when running the index status from the CLI.
94+
95+
The Magento indexing mechanism uses the status value in reindex triggering process. You can check the status of an indexer in the [Admin](https://glossary.magento.com/admin) panel in **System >** Tools **> Index Management** or manually using the [command line]({{ page.baseurl }}/config-guide/cli/config-cli-subcommands-index.html#view-indexer-status).
96+
97+
### Using application lock mode for reindex processes
98+
99+
Starting with 2.4.3, you can enable `use_application_lock` mode for reindexing through the use of environment variables, or in `app/etc/env.php`:
100+
101+
```php
102+
<?php
103+
return [
104+
'indexer' => [
105+
'use_application_lock' => true
106+
]
107+
];
108+
```
109+
110+
In case of a failure, this mode will return the correct status of the indexer.
111+
112+
The current status can be obtained from the indexer grid in Magento Admin or through the index status in the CLI.
113+
114+
When this option is used, the values in the SQL table `indexer_state` may not be up to date.
115+
116+
### Indexing modes {#m2devgde-indexing-modes}
117+
118+
Reindexing can be performed in two modes:
119+
120+
* Update on Save - index tables are updated immediately after the dictionary data is changed.
121+
122+
{:.bs-callout-info}
123+
**Update on Save** indexers must use custom code (plugins, events or any other working approach) in order to trigger reindexing when entities are saved/deleted etc.
124+
125+
* Update by Schedule - index tables are updated by cron job according to the configured schedule.
126+
127+
{:.bs-callout-info}
128+
**Update by Schedule** does not support the `customer_grid` indexer. You must either use **Update on Save** or reindex the customer grid manually (`bin/magento indexer:reindex customer_grid`). See the [Help Center article](https://support.magento.com/hc/en-us/articles/360025481892-New-customer-records-are-not-displayed-in-the-Customers-grid-after-importing-them-from-CSV).
129+
130+
To set these options:
131+
132+
1. Log in to the [Magento Admin](https://glossary.magento.com/magento-admin).
133+
1. Click **System >** Tools **> Index Management**.
134+
1. Select the checkbox next to each type of indexer to change.
135+
1. From the **Actions** list, click the indexing mode.
136+
1. Click **Submit**.
137+
138+
You can also reindex from the [command line]({{ page.baseurl }}/config-guide/cli/config-cli-subcommands-index.html#configure-indexers)
139+
140+
The following figure shows an example of setting indexers to Update by Schedule:
141+
142+
![Changing indexer modes]({{ site.baseurl }}/common/images/index_index-modes.png){:width="600px"}
143+
144+
### Mview {#m2devgde-mview}
145+
146+
The `mview.xml` file is used to track database changes for a certain entity.
147+
148+
For example part of `Magento/Catalog/etc/mview.xml` is tracking category to product relation described in the following record:
149+
150+
```xml
151+
<!-- ... -->
152+
<view id="catalog_category_product" class="Magento\Catalog\Model\Indexer\Category\Product" group="indexer">
153+
<subscriptions>
154+
<table name="catalog_category_entity" entity_column="entity_id" />
155+
<table name="catalog_category_entity_int" entity_column="entity_id" />
156+
</subscriptions>
157+
</view>
158+
<!-- ... -->
159+
```
160+
161+
Explanation of nodes:
162+
163+
* The `view` node defines an indexer. The `id` attribute is a name of the indexer table, the `class` attribute is indexer executor, the `group` attribute defines the indexer group.
164+
* The `subscriptions` node is a list of tables for tracking changes.
165+
* The `table` node defines the certain table to observe and track changes. The attribute `name` is a name of an observable table, the attribute `entity_column` is an identifier column of entity to be re-indexed. So, in case of `catalog_category_product`, whenever one or more categories is saved, updated or deleted in `catalog_category_entity` the `execute` method of `Magento\Catalog\Model\Indexer\Category\Product` will be called with argument `ids` containing ids of entities from column defined under `entity_column` attribute. If indexer type is set to "Update on Save" the method is called right away after the operation. If it set to "Update by Schedule" the mechanism creates a record in the change log table using MYSQL triggers.
166+
167+
A change log table is created according to the naming rule - INDEXER_TABLE_NAME + '_cl', in case of `catalog_category_product` it will be `catalog_category_product_cl`.
168+
The table contains the `version_id` auto-increment column and `entity_id` column that contains identifiers of entities to be re-indexed.
169+
For each `table` node the framework automatically creates MYSQL AFTER triggers for each possible event (INSERT, UPDATE, DELETE).
170+
171+
For the table `catalog_category_entity` triggers will be created with the following statements.
172+
INSERT operation:
173+
174+
```mysql
175+
BEGIN
176+
INSERT IGNORE INTO `catalog_category_product_cl` (`entity_id`) VALUES (NEW.`entity_id`);
177+
END
178+
```
179+
180+
UPDATE operation:
181+
182+
```mysql
183+
BEGIN
184+
IF (NEW.`entity_id` <=> OLD.`entity_id`
185+
OR NEW.`attribute_set_id` <=> OLD.`attribute_set_id`
186+
OR NEW.`parent_id` <=> OLD.`parent_id`
187+
OR NEW.`created_at` <=> OLD.`created_at`
188+
OR NEW.`path` <=> OLD.`path`
189+
OR NEW.`position` <=> OLD.`position`
190+
OR NEW.`level` <=> OLD.`level`
191+
OR NEW.`children_count` <=> OLD.`children_count`)
192+
THEN INSERT IGNORE INTO `catalog_category_product_cl` (`entity_id`) VALUES (NEW.`entity_id`);
193+
END IF;
194+
END
195+
```
196+
197+
DELETE operation:
198+
199+
```mysql
200+
BEGIN
201+
INSERT IGNORE INTO `catalog_category_product_cl` (`entity_id`) VALUES (OLD.`entity_id`);
202+
END
203+
```
204+
205+
The method `Magento\Framework\Mview\ViewInterface::update` is responsible for handling records in the changelog. The method is being called by CRON and
206+
it defines IDs to be re-indexed from the change log by last applied `version_id` and calls the `execute` method for each particular indexer with IDs as an argument.
207+
208+
### How to reindex
209+
210+
You can reindex by:
211+
212+
* Using a [cron job]({{ page.baseurl }}/config-guide/cli/config-cli-subcommands-cron.html), which is preferred because indexing runs every minute.
213+
* Using the [`magento indexer:reindex [indexer]`]({{ page.baseurl }}/config-guide/cli/config-cli-subcommands-index.html#config-cli-subcommands-index-reindex) command, which reindexes selected indexers, or all indexers, one time only.
214+
215+
## Magento indexers {#m2devgde-indexing-outofbox}
216+
217+
The Magento `Open Source` application implements the following indexers (use [bin/magento indexer:info]({{ page.baseurl }}/reference/cli/magento.html#indexerinfo) to list the indexers):
218+
219+
| Indexer name | Indexer method name | Indexer class | Description |
220+
| --- | --- | --- | --- |
221+
| Design Config Grid | `design_config_grid` | [Magento\Theme\Model\Indexer\Design\Config]({{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/Theme/Model/Indexer/Design/Config.php) | |
222+
| Customer Grid | `customer_grid` | [Magento\Framework\Indexer\Action\Entity]({{ site.mage2bloburl }}/{{ page.guide_version }}/lib/internal/Magento/Framework/Indexer/Action/Entity.php) | Rebuilds the customer grid index. Not supported by the `Update by Schedule` indexing mode. See the [Help Center article](https://support.magento.com/hc/en-us/articles/360025481892-New-customer-records-are-not-displayed-in-the-Customers-grid-after-importing-them-from-CSV). |
223+
| Category products | `catalog_category_product` | [Magento\Catalog\Model\Indexer\Category\Product]({{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/Catalog/Model/Indexer/Category/Product.php) | Creates category/products association |
224+
| Product categories | `catalog_product_category` | [Magento\Catalog\Model\Indexer\Product\Category]({{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/Catalog/Model/Indexer/Product/Category.php) | Creates category/products association |
225+
| Product price | `catalog_product_price` | [Magento\Catalog\Model\Indexer\Product\Price]({{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/Catalog/Model/Indexer/Product/Price.php) | Pre-calculates product prices |
226+
| Product entity attribute value | `catalog_product_attribute` | [Magento\Catalog\Model\Indexer\Product\Eav]({{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/Catalog/Model/Indexer/Product/Eav.php) | Reorganizes the EAV product structure to flat structure |
227+
| Stock | `cataloginventory_stock` | [Magento\CatalogInventory\Model\Indexer\Stock]({{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/CatalogInventory/Model/Indexer/Stock.php) | |
228+
| Catalog rule product | `catalogrule_rule` | [Magento\CatalogRule\Model\Indexer\Rule\RuleProductIndexer]({{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/CatalogRule/Model/Indexer/Rule/RuleProductIndexer.php) | |
229+
| Catalog product rule | `catalogrule_product` | [Magento\CatalogRule\Model\Indexer\Product\ProductRuleIndexer]({{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/CatalogRule/Model/Indexer/Product/ProductRuleIndexer.php) | |
230+
| Catalog search | `catalogsearch_fulltext` | [Magento\CatalogSearch\Model\Indexer\Fulltext]({{ site.mage2bloburl }}/{{ page.guide_version }}/app/code/Magento/CatalogSearch/Model/Indexer/Fulltext.php) | |
231+
232+
Magento Commerce Edition contains all indexers of Magento Open Source Edition and the following ones:
233+
234+
| Indexer name | Indexer method name | Indexer class | Description |
235+
| --- | --- | --- | --- |
236+
| Inventory | `inventory` | `Magento\InventoryIndexer\Indexer\InventoryIndexer` | Inventory index (MSI) |
237+
| Product/Target Rule | `targetrule_product_rule` | `Magento\TargetRule\Model\Indexer\TargetRule\Product\Rule` | Indexes product/rule association |
238+
| Target Rule/Product | `targetrule_rule_product` | `Magento\TargetRule\Model\Indexer\TargetRule\Rule\Product` | Indexes rule/product association |
239+
| Sales Rule | `salesrule_rule` | `Magento\AdvancedSalesRule\Model\Indexer\SalesRule` | Indexes sales rule |

0 commit comments

Comments
 (0)