Monitoring

For arlanix/notification 1.0.0

Monitoring

arlanix/module-notification-monitoring gives the store's rules eleven events about the store itself: failed and new admin sign-ins, failed cron jobs, a stopped cron, invalid indexers, errors in the log, imports and exports that finished or failed, and a store view whose orders stopped. Which go where is up to the store's rules: see Monitoring in the manual.

Who looks

Event Code Noticed by
Admin login failed admin_login_failed Magento's backend_auth_user_login_failed event.
Admin logged in admin_login backend_auth_user_login_success; admin.new_ip and admin.new_device from the addresses and browsers the admin signed in from in the year before.
Cron job failed cron_job_failed The check: jobs of cron_schedule in error since its last look.
Cron stopped cron_stopped The check, an admin page and the health URL: the newest executed_at of cron_schedule against cron_stale_minutes.
Indexer needs reindex indexer_invalid The check: each indexer's state.
Orders stopped orders_stalled The check: each store view's orders and carts against the same minutes of the weeks before.
Application error app_error A processor of Magento's main logger, at or above app_error_level.
Import finished, Import failed import_finished, import_failed A plugin on Magento\ImportExport\Model\Import::importSource().
Export finished, Export failed export_finished, export_failed Plugins on the export queue consumer and ExportManagementInterface::export().

The check is the cron job arlanix_notification_monitoring_check, every five minutes in the arlanix_notification group, and the command:

bin/magento arlanix:notification:monitoring:check

A check tells a change of state, not each look: a problem, then its recovery as Info. The states are kept in arlanix_notification_monitoring_state, updated conditionally, so two checks at once tell a change once.

Orders that stop

The check looks at each active store view whose Watch the orders is on, and counts its orders (sales_order.created_at) and the carts its shoppers changed (quote.updated_at) in the last Minutes looked at, and in the same minutes of the week in each of the weeks before. What is expected is the median of those weeks, so one sale or one outage does not move it. Then:

Found Told
Fewer orders expected than Orders expected at least Nothing: too few orders come at that time of the week to tell.
Orders at or above Orders below (%) of those expected Nothing.
Fewer, and carts at or above Carts as usual from (%) of those expected Orders stopped, reason checkout_broken: shoppers are there, their orders fail.
Fewer, and carts below Shoppers gone below (%) Orders stopped, reason traffic_gone: the storefront cannot be reached.
Fewer, and carts between the two Nothing: too few carts to say shoppers are there, too many to say they are gone.
Fewer, and too few weeks of carts kept to compare Nothing. Magento deletes carts after Quote Lifetime (days); a week before the oldest cart left is not counted.

A change is told once Looks in a row before it is sent looks (2) have seen it, both ways, so one quiet window does not send it; pending_count in arlanix_notification_monitoring_state keeps the count. A week is 604,800 seconds, so after the clocks change, the weeks from before the change are compared an hour off until they fall out of the comparison.

Every setting but the looks in a row is read for each store view: set one apart under its scope, or switch the check off where orders are too few. The counts go through the sales and checkout connections, which a split database keeps apart, over Magento's indexes on the store view and the time.

A stopped cron

Cron runs the check, so a stopped cron is noticed from outside it: an admin page asks at most once in five minutes, and the health URL each time it is asked. Cron stopped is sent at once, past the queue, whose consumer cron starts.

The health check

For an uptime monitor such as UptimeRobot, Pingdom or a load balancer: set Health check key in Stores > Configuration > Arlanix > Notifications > Monitoring (stored encrypted), and have the monitor ask

GET https://store.example.com/arlanix_monitoring/health/cron?key=…

or send the key in the X-Health-Key header instead.

Answer Means
200 {"cron":"running","minutes_since_last_run":1,…} A job ran within cron_stale_minutes.
503 {"cron":"stopped",…} None has. Asking also sends Cron stopped, once.
503 {"cron":"unknown","minutes_since_last_run":null,…} No job has ever run, by cron_schedule: a new installation whose cron is not set up yet, or the table emptied. Nothing is sent.
403 A wrong key.
404 No key is set: the URL is off.

It is a storefront route, answered with Cache-Control: no-store; keep it out of Varnish and the full-page cache if your configuration caches by path. Before pointing a monitor at it:

  • Store codes in URLs (Add Store Code to Urls): ask with a store code, such as https://store.example.com/default/arlanix_monitoring/health/cron?key=….
  • Maintenance mode answers every storefront request with Magento's own 503, so the monitor reports a stop during a deploy: pause it then, or let its address through with bin/magento maintenance:allow-ips.
  • Basic auth or an IP allowlist, as a staging site often has: give the monitor the credentials, or allow its addresses. The key can go in X-Health-Key beside an Authorization header.
  • A new installation answers 503 until cron has run once.

Application errors

A Monolog processor on Magento\Framework\Logger\Monolog, working with Monolog 2 and 3, turns a record at or above app_error_level (default 500, Critical) into the event, with its message, level, exception class and the file and line it came from.

  • The same message is sent once in ten minutes.
  • A record written inside a database transaction is sent at the end of the request, as the transaction may be rolled back.
  • A record about notifications themselves is not sent, so a failing delivery cannot feed itself.

Imports and exports

Every import through Magento's import model goes through Import::importSource(): the Import page, the CSV API, Adobe Commerce's scheduled imports, and integrations built on the model. The plugin tells how it ended:

  • importSource() returned true: Import finished, with the created, updated and deleted counts and the errors of the error aggregator. Rows with errors do not make it fail.
  • It returned false or threw: Import failed, with the first error or the exception's message. The exception is thrown on.

The file and the admin come from the Import History row Magento loads while the import runs. An import from the Import page has one; an import that keeps no history leaves import.file and import.user empty.

An export is a message on the queue export, made by the consumer exportProcessor, which catches its own failure and only logs it. So the export counts as finished when the consumer has written var/export/<file>. A failure's reason is kept by a plugin on ExportManagementInterface::export() before the consumer catches it; a failure after that, such as an unwritable directory, is told with a pointer to the log. With no consumer running, no export is made and nothing is told.

What tells an import or an export never changes its outcome: an error in gathering the event's data is logged, and the import or the export goes on.

What it keeps

Table Holds
arlanix_notification_monitoring_state Each check's last state: cron, each indexer, each store view's orders, and the looks that have seen a change not yet told.
arlanix_notification_monitoring_admin_seen The addresses and browsers each admin signed in from in the last year, as HMAC-SHA-256 with the store's encryption key, removed with the admin user.

The admin's IP address and browser are personal fields: a rule sends them only when its own text names them.

arlanix_notification_monitoring_admin_seen starts empty, so each admin's first sign-in after the installation is from a new address and a new device.

Behind a proxy

The address is Magento's RemoteAddress, which reads REMOTE_ADDR. Behind a load balancer, a CDN or Varnish that does not hand the client's address on, that is the proxy's: every sign-in comes from it, From a new address is never Yes after the first, and a failed sign-in names the proxy. Have the web server set the client's address (nginx real_ip, Apache mod_remoteip), or tell RemoteAddress the header, with your proxies as trusted, in a module's di.xml:

<type name="Magento\Framework\HTTP\PhpEnvironment\RemoteAddress">
    <arguments>
        <argument name="alternativeHeaders" xsi:type="array">
            <item name="x-forwarded-for" xsi:type="string">HTTP_X_FORWARDED_FOR</item>
        </argument>
        <argument name="trustedProxies" xsi:type="array">
            <item name="load-balancer" xsi:type="string">10.0.0.10</item>
        </argument>
    </arguments>
</type>

With trustedProxies, the last address of the header that is not a proxy's is taken; without it, the first, which the client can write itself.

A browser is told by the cookie arlanix_admin_device on the admin's path: a random token, HttpOnly, set again at each sign-in to last a year. A private window, another browser or cleared cookies sign in as a new device. A new encryption key makes every address and browser new once.

Settings

Path Admin field Default
arlanix_notification/monitoring/cron_stale_minutes Cron stopped after (minutes) 15
arlanix_notification/monitoring/health_key Health check key Encrypted. Empty turns the URL off.
arlanix_notification/monitoring/app_error_level Application errors from 500 400 Error, 500 Critical, 550 Alert, 600 Emergency.
arlanix_notification/order_flow/enabled Watch the orders 1 Each store view.
arlanix_notification/order_flow/window_minutes Minutes looked at 10 Each store view.
arlanix_notification/order_flow/order_weeks Weeks of orders compared 4 Each store view.
arlanix_notification/order_flow/cart_weeks Weeks of carts compared 3 Each store view.
arlanix_notification/order_flow/min_expected_orders Orders expected at least 3 Each store view.
arlanix_notification/order_flow/orders_below_percent Orders below (%) 25 Each store view.
arlanix_notification/order_flow/carts_normal_percent Carts as usual from (%) 60 Each store view.
arlanix_notification/order_flow/carts_collapsed_percent Shoppers gone below (%) 20 Each store view.
arlanix_notification/order_flow/confirmations Looks in a row before it is sent 2 Default scope.