Kemp.rest / Examples / Code / Local rank grids
KEMP.REST / EXAMPLES / CODE
Local rank grids on a clock
Written by Kevin Kemp in PHP and Laravel for AnswerPins. Queued jobs, JSON, and the DataForSEO Google Maps REST API. API credentials stay in the environment, not in this code.
View the codeA local rank report is a Google Maps search from many spots, not one search from the office. AnswerPins lays a square grid around the business, runs a live Maps search at each point through DataForSEO, and records the rank. The whole thing runs on a schedule, so a new account gets a grid within the hour and later scans stay on a weekly or monthly clock.
Type of code
Laravel console schedule, an Artisan command, a queued job, and a service. No page request starts a scan.
Technology
PHP, Laravel scheduler and queues, Eloquent, JSON, and the DataForSEO Google Maps live REST endpoint (/v3/serp/google/maps/live/advanced). HTTP calls go out in pools of 10. Google Places supplies the center pin when the listing has coordinates.
Timing
- The host cron hits the scheduler every minute. There is no always-on worker.
answerpins:queue-gridsruns at minute 15 of every hour. A signup is scanned inside that hour.- The queue worker starts every minute, stops when empty, and is capped at 55 seconds so it cannot overlap the next minute. Jobs get 3 tries. The grid job itself allows 2 tries, waits 300 seconds between them, and times out at 300 seconds.
- A service that already has a queued scan, or a finished scan inside the plan window, is skipped. The window is 7 days on the managed plan and 30 days on the self-managed plan, with an extra hour so a scan is not queued twice as the window rolls over.
- Profile checks run at 05:00. Competitor refresh runs at 05:30. The alert digest runs at 08:00 America/Chicago.
What it does
- Finds the grid center from the Google listing pin, then the address, then a tracked town.
- Sets the spacing from the furthest tracked town, held between a half mile and 5 miles, so the grid covers the area the business actually serves.
- Builds the square. Row 0 is the north edge, so the points read like a map. Longitude step shrinks with latitude.
- Posts a Maps search at each point, 10 at a time, with
search_this_areaoff so Google does not lock the result to that map view. - A point with no businesses (DataForSEO status 40102) is a real empty result. A failed call is an error. If more than half the points error, the job fails and can retry.
- Saves rank, top 3 names, how many points found the business, how many were in the top 3, and the average. Missing ranks count as 21.
Problem it solved
Rank was no longer a single search from one computer. Each tracked service and city got a repeatable grid, on a plan cadence, without someone pressing a button.
Code
Back to summary · routes/console.php, QueueGrids.php, RunGridScan.php, GridScanner.php
// —— routes/console.php ——
<?php
use Illuminate\Support\Facades\Schedule;
// The host has no always-on worker. One cron entry runs the scheduler every minute.
// That queues due checks hourly, drains the queue for up to 55 seconds, then exits.
Schedule::command('answerpins:queue-checks')->hourly();
Schedule::command('queue:work --stop-when-empty --max-time=55 --tries=3')->everyMinute()->withoutOverlapping();
Schedule::command('answerpins:check-profiles')->dailyAt('05:00');
Schedule::command('answerpins:refresh-competitors')->dailyAt('05:30');
Schedule::command('answerpins:send-alerts')->dailyAt('08:00')->timezone('America/Chicago');
// Heartbeat: proof the cron is alive.
Schedule::call(fn () => cache()->forever('answerpins:heartbeat', now()->toIso8601String()))->everyMinute()->name('heartbeat');
Schedule::command('answerpins:queue-grids')->hourlyAt(15); // a new signup gets a first grid within the hour
// —— app/Console/Commands/QueueGrids.php ——
<?php
namespace App\Console\Commands;
use App\Jobs\RunGridScan;
use App\Models\Business;
use App\Models\GridScan;
use App\Services\GridScanner;
use Illuminate\Console\Command;
/** Hourly: queues a local rank grid for each tracked service that is due, and for the plan's included own keywords (weekly on Pro Managed, monthly on Self Managed). */
class QueueGrids extends Command
{
protected $signature = 'answerpins:queue-grids';
protected $description = 'Queue local rank grid scans that are due';
public function handle(GridScanner $scanner): int
{
if (! config('answerpins.engines.google.key')) {
$this->warn('DataForSEO is not set up.');
return self::SUCCESS;
}
$queued = 0;
Business::query()->whereHas('account', fn ($q) => $q->where('status', 'active'))->with('account', 'services')
->each(function (Business $b) use ($scanner, &$queued) {
$plan = config('answerpins.grids.'.($b->account->plan === 'owner' ? 'owner' : 'managed'));
$center = null;
$spacing = null;
foreach ($b->services->where('tracked', true) as $service) {
$keyword = mb_strtolower($service->name);
$recent = $b->gridScans()->where('keyword', $keyword)
->where(fn ($q) => $q->where('status', 'queued')->orWhere('created_at', '>', now()->subDays($plan['every_days'])->addHour()))->exists();
if ($recent) {
continue;
}
$center ??= $scanner->center($b);
if (! $center) {
return;
}
$spacing ??= $scanner->spacingFor($b, $center, $plan['size']);
$scan = GridScan::create([
'business_id' => $b->id, 'service_id' => $service->id, 'keyword' => $keyword,
'center_lat' => $center[0], 'center_lng' => $center[1], 'size' => $plan['size'], 'spacing_miles' => $spacing,
'points' => $scanner->points($center[0], $center[1], $plan['size'], $spacing),
]);
RunGridScan::dispatch($scan->id);
$queued++;
}
if ($scanner->includedKeywords($b)->isNotEmpty()) {
$center ??= $scanner->center($b);
if ($center) {
$spacing ??= $scanner->spacingFor($b, $center, $plan['size']);
$queued += $scanner->queueKeywordGrids($b, $center, $spacing, $plan['size'], $plan['every_days']);
}
}
});
// Cities, oldest first, up to the account's cities: after a downgrade the extra cities stay but stop being scanned.
$cities = \App\Models\City::whereHas('business.account', fn ($q) => $q->where('status', 'active'))->with('business.account')->orderBy('id')->get();
foreach ($cities->groupBy('business_id') as $list) {
$account = $list->first()->business->account;
$max = (int) (($account->limits ?? [])['cities'] ?? config('answerpins.plans.'.$account->plan.'.limits.cities', 0));
foreach ($list->take($max) as $city) {
$queued += $scanner->queueCity($city, config('answerpins.grids.cities.every_days'));
}
}
$this->info("Queued {$queued} grid scans.");
return self::SUCCESS;
}
}
// —— app/Jobs/RunGridScan.php ——
<?php
namespace App\Jobs;
use App\Models\GridScan;
use App\Services\GridScanner;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Throwable;
class RunGridScan implements ShouldQueue
{
use Queueable;
public int $tries = 2;
public array $backoff = [300];
public int $timeout = 300;
public function __construct(public int $scanId) {}
public function handle(GridScanner $scanner): void
{
$scan = GridScan::find($this->scanId);
if ($scan && $scan->status !== 'done') {
$scanner->run($scan);
}
}
public function failed(Throwable $e): void
{
GridScan::whereKey($this->scanId)->update(['status' => 'failed', 'error' => mb_substr($e->getMessage(), 0, 1000)]);
}
}
// —— app/Services/GridScanner.php ——
<?php
namespace App\Services;
use App\Jobs\RunGridScan;
use App\Models\Business;
use App\Models\City;
use App\Models\GridScan;
use Illuminate\Http\Client\Pool;
use Illuminate\Support\Facades\Http;
use RuntimeException;
/**
* Local rank grid: a Google Maps search (through DataForSEO) at every point of a square grid around the business,
* recording where the business ranks at each point. Points outside the top 20 are recorded as not found.
*/
class GridScanner
{
private const MAPS_URL = 'https://api.dataforseo.com/v3/serp/google/maps/live/advanced';
public function __construct(private GooglePlaces $places, private AnswerMatcher $matcher) {}
/** Centre of the grid: the Google listing's pin, else the business address or first town looked up. */
public function center(Business $business): ?array
{
$snap = $business->profileSnapshots()->latest('captured_at')->latest('id')->first();
if ($snap && isset($snap->data['location']['latitude'])) {
return [(float) $snap->data['location']['latitude'], (float) $snap->data['location']['longitude']];
}
if ($business->google_place_id && config('answerpins.google.places_key')) {
try {
$listing = $this->places->details($business->google_place_id)['listing'];
if ($listing['lat'] !== null) {
return [(float) $listing['lat'], (float) $listing['lng']];
}
} catch (\Throwable) {
}
}
$town = $business->towns()->where('tracked', true)->first();
if ($town?->lat !== null && $town?->lat !== '') {
return [(float) $town->lat, (float) $town->lng];
}
$query = $business->address ?: trim(($business->name ?? '').' '.($town?->name ?? ''));
return $query && config('answerpins.google.places_key') ? $this->places->locate($query) : null;
}
/**
* Spacing that stretches the grid to reach the furthest tracked town, so every town is covered.
* Kept between ½ mile and 5 miles; 1 mile when no town can be placed.
*/
public function spacingFor(Business $business, array $center, int $size): float
{
$state = $business->state();
$furthest = 0.0;
foreach ($business->towns()->where('tracked', true)->get() as $town) {
if ($town->lat === null && config('answerpins.google.places_key')) {
try {
if ($p = $this->places->locate($state ? "{$town->name}, {$state}" : $town->name)) {
$town->update(['lat' => $p[0], 'lng' => $p[1]]);
}
} catch (\Throwable) {
}
}
if ($town->lat !== null) {
$furthest = max($furthest, $this->miles($center[0], $center[1], (float) $town->lat, (float) $town->lng));
}
}
if ($furthest <= 0) {
return (float) config('answerpins.grids.spacing_miles');
}
return round(min(5.0, max(0.5, $furthest / max(1, intdiv($size, 2)))), 2);
}
private function miles(float $lat1, float $lng1, float $lat2, float $lng2): float
{
$r = 3958.8;
$dLat = deg2rad($lat2 - $lat1);
$dLng = deg2rad($lng2 - $lng1);
$a = sin($dLat / 2) ** 2 + cos(deg2rad($lat1)) * cos(deg2rad($lat2)) * sin($dLng / 2) ** 2;
return 2 * $r * asin(min(1, sqrt($a)));
}
/** @return list<array{row: int, col: int, lat: float, lng: float}> */
public function points(float $lat, float $lng, int $size, float $miles): array
{
$dLat = $miles / 69.0;
$dLng = $miles / (69.0 * max(0.2, cos(deg2rad($lat))));
$mid = intdiv($size, 2);
$points = [];
for ($row = 0; $row < $size; $row++) {
for ($col = 0; $col < $size; $col++) {
// Row 0 is the north edge, so the grid reads like a map.
$points[] = ['row' => $row, 'col' => $col, 'lat' => round($lat + ($mid - $row) * $dLat, 7), 'lng' => round($lng + ($col - $mid) * $dLng, 7)];
}
}
return $points;
}
/** The searches checked from a city: each tracked service with each tracked town, like "sunset sailing charters sag harbor". */
public function cityKeywords(Business $business): array
{
$towns = $business->towns()->where('tracked', true)->pluck('name');
$keywords = [];
foreach ($business->services()->where('tracked', true)->get() as $service) {
foreach ($towns as $town) {
$keywords[] = ['service_id' => $service->id, 'keyword' => mb_strtolower($service->name.' '.$town)];
}
}
return $keywords;
}
/** Own keywords that get a Pin Grid on this plan, oldest first. Later ones are checked on Google without a map. */
public function includedKeywords(Business $business): \Illuminate\Support\Collection
{
$plan = $business->account?->plan === 'owner' ? 'owner' : 'managed';
$limit = (int) config('answerpins.grids.keyword_grids.'.$plan, 0);
return $business->questions()->where('intent', 'keyword')->where('status', 'approved')->orderBy('id')->limit($limit)->get();
}
/**
* Queue a home-area Pin Grid for each included own keyword.
* When $everyDays is set, a scan newer than that is left alone. A rescan passes null and only skips one already queued.
*/
public function queueKeywordGrids(Business $business, array $center, float $spacing, int $size, ?int $everyDays = null): int
{
$queued = 0;
foreach ($this->includedKeywords($business) as $question) {
$keyword = mb_strtolower(trim($question->text));
$recent = $business->gridScans()->where('keyword', $keyword)->whereNull('city_id')
->where(fn ($q) => $q->where('status', 'queued')->when($everyDays, fn ($q) => $q->orWhere('created_at', '>', now()->subDays($everyDays)->addHour())))
->exists();
if ($recent) {
continue;
}
$scan = GridScan::create([
'business_id' => $business->id, 'keyword' => $keyword,
'center_lat' => $center[0], 'center_lng' => $center[1], 'size' => $size, 'spacing_miles' => $spacing,
'points' => $this->points($center[0], $center[1], $size, $spacing),
]);
RunGridScan::dispatch($scan->id);
$queued++;
}
return $queued;
}
/**
* Queue a scan of the city for each of its searches. Skips a search already queued, or, when $everyDays is given,
* one scanned more recently than that (the weekly run); the first scan and an admin rescan pass null.
*/
public function queueCity(City $city, ?int $everyDays = null): int
{
$cfg = config('answerpins.grids.cities');
$queued = 0;
foreach ($this->cityKeywords($city->business) as $k) {
$recent = $city->gridScans()->where('keyword', $k['keyword'])
->where(fn ($q) => $q->where('status', 'queued')->when($everyDays, fn ($q) => $q->orWhere('created_at', '>', now()->subDays($everyDays)->addHour())))->exists();
if ($recent) {
continue;
}
$scan = GridScan::create([
'business_id' => $city->business_id, 'service_id' => $k['service_id'], 'city_id' => $city->id, 'keyword' => $k['keyword'],
'center_lat' => $city->lat, 'center_lng' => $city->lng, 'size' => $cfg['size'], 'spacing_miles' => $cfg['spacing_miles'],
'points' => $this->points($city->lat, $city->lng, $cfg['size'], $cfg['spacing_miles']),
]);
RunGridScan::dispatch($scan->id);
$queued++;
}
return $queued;
}
public function run(GridScan $scan): void
{
$scan->loadMissing('business');
$points = $scan->points ?: $this->points($scan->center_lat, $scan->center_lng, $scan->size, $scan->spacing_miles);
$zoom = config('answerpins.grids.zoom');
foreach (array_chunk(array_keys($points), 10) as $chunk) {
$responses = Http::pool(fn (Pool $pool) => array_map(fn ($i) => $pool->as((string) $i)
->withBasicAuth(config('answerpins.engines.google.key'), (string) config('answerpins.engines.google.password'))
->timeout(90)
->post(self::MAPS_URL, [[
// search_this_area is on by default: it locks the search to the map view around the point, like pressing
// "Search this area", which real searchers don't do (a New York spot then only lists New York businesses).
'keyword' => $scan->keyword, 'language_code' => 'en', 'depth' => 20, 'search_this_area' => false,
'location_coordinate' => $points[$i]['lat'].','.$points[$i]['lng'].','.$zoom,
]]), $chunk));
foreach ($chunk as $i) {
[$points[$i]['rank'], $points[$i]['top']] = $this->rankIn($scan->business, $responses[(string) $i] ?? null);
}
}
$failed = count(array_filter($points, fn ($p) => ($p['rank'] ?? null) === 'error'));
if ($failed > count($points) / 2) {
throw new RuntimeException("Google Maps searches failed at {$failed} of ".count($points).' points.');
}
$ranks = array_map(fn ($p) => is_int($p['rank']) ? $p['rank'] : null, $points);
$scan->update([
'status' => 'done',
'points' => array_map(fn ($p) => array_merge($p, ['rank' => is_int($p['rank']) ? $p['rank'] : null]), $points),
'found' => count(array_filter($ranks)),
'top3' => count(array_filter($ranks, fn ($r) => $r && $r <= 3)),
'avg_rank' => round(array_sum(array_map(fn ($r) => $r ?: 21, $ranks)) / count($ranks), 2),
'scanned_at' => now(),
'error' => null,
]);
}
/**
* The business's position in one Maps search (int, null when not in the top 20, or 'error'),
* plus the top 3 names Google showed there, so owners can see who beats them in each neighbourhood.
*/
private function rankIn(Business $business, $res): array
{
// 40102 "No Search Results": Google shows no businesses from that spot (out on the water, open country),
// so nobody is found there. That is a real answer, not a failed search.
if ($res instanceof \Illuminate\Http\Client\Response && $res->json('tasks.0.status_code') === 40102) {
return [null, []];
}
if (! $res instanceof \Illuminate\Http\Client\Response || $res->json('tasks.0.status_code') !== 20000) {
return ['error', []];
}
$pack = collect($res->json('tasks.0.result.0.items') ?? [])->where('type', 'maps_search')->values()->map(fn ($it) => [
'position' => $it['rank_group'] ?? null, 'title' => $it['title'] ?? '', 'domain' => $it['domain'] ?? null, 'place_id' => $it['place_id'] ?? null,
'rating' => $it['rating']['value'] ?? null, 'reviews' => $it['rating']['votes_count'] ?? null,
])->all();
$rank = null;
foreach ($pack as $p) {
if ($business->google_place_id && $p['place_id'] === $business->google_place_id) {
$rank = (int) $p['position'];
break;
}
}
$rank ??= $this->matcher->mapRank($business, $pack);
$top = array_map(fn ($p) => ['name' => $p['title'], 'rating' => $p['rating'], 'reviews' => $p['reviews']], array_slice($pack, 0, 3));
return [$rank, $top];
}
}