Live Google results through DataForSEO

Written by Kevin Kemp in PHP and Laravel for AnswerPins. JSON over the DataForSEO REST API, including the AI Overview. API credentials stay in the environment, not in this code.

View the code

Google does not offer an API for the AI Overview or the local pack on a normal search. AnswerPins asks DataForSEO for a live Google results page, placed at the town’s coordinates, then pulls three things out of that one response: the AI Overview, the local pack, and the organic results. This is the AI Overview side of the same work as the service-area page builder.

Type of code

A Laravel service that implements an AnswerEngine interface. A queued check calls ask(). The class does not schedule itself. The hourly check queue does.

Technology

PHP, the Laravel HTTP client, JSON, and DataForSEO’s live advanced organic REST endpoint (/v3/serp/google/organic/live/advanced). Login and password are HTTP Basic auth from config. The call asks DataForSEO to load the async AI Overview before it returns.

Timing

The HTTP timeout is 120 seconds. A live Google page is slow.
DataForSEO often returns task status 40101, an internal search error that succeeds on the next try. The same is true of 50000 and 50401. The call retries up to 3 times and waits 3 seconds between tries. Unit tests skip the wait.
Anything else, including HTTP failure or a status other than 20000, is thrown so the queue can retry the job on its own backoff.

What it does

Sends the question as the keyword. If the town has a coordinate, the search is placed there. Otherwise it uses the United States location code and leaves the town in the question.
Reads ai_overview items into text. Google listing cards become numbered lines with the name in bold, so a position in the answer can be scored. Images and empty bold markers are stripped.
Keeps citations, and drops Google’s own /searchviewer links.
Maps the local pack to position, title, domain, phone, rating, review count, the lines under the name, and whether the slot is paid.
Maps organic results to rank, domain, URL, and title, and keeps DataForSEO’s check_url so the exact search can be opened later.

Problem it solved

Each tracked question got a current Google page, not a cached rank from a commercial suite. The AI answer, the map pack, and the blue links came from the same live search, at the town the business cares about.

Code

Back to summary · app/Services/Engines/GoogleAiOverviewEngine.php

<?php

namespace App\Services\Engines;

use Illuminate\Support\Facades\Http;
use Illuminate\Support\Str;
use RuntimeException;

/**
 * Google's AI Overview, read from a live Google search through DataForSEO (no official Google API exists).
 * The search is placed at the town's coordinates. The map pack from the same search is kept for competitor tracking.
 */
class GoogleAiOverviewEngine implements AnswerEngine
{
    public function ask(string $question, array $location): array
    {
        $task = [
            'keyword' => $question,
            'language_code' => 'en',
            'device' => config('answerpins.engines.google.device', 'mobile'),
            'load_async_ai_overview' => true,
        ];
        if (! empty($location['coordinate'])) {
            $task['location_coordinate'] = $location['coordinate'];
        } else {
            $task['location_code'] = 2840; // United States; the town is in the question
        }

        // DataForSEO often answers 40101 ("Internal SE Server Error") and succeeds on the next try, so retry quickly first.
        for ($try = 1; $try <= 3; $try++) {
            $res = Http::withBasicAuth(config('answerpins.engines.google.key'), (string) config('answerpins.engines.google.password'))
                ->timeout(120)
                ->post('https://api.dataforseo.com/v3/serp/google/organic/live/advanced', [$task]);
            $status = $res->json('tasks.0.status_code');
            if ($status === 20000 || ! in_array($status, [40101, 50000, 50401], true)) {
                break;
            }
            sleep(app()->runningUnitTests() ? 0 : 3);
        }
        if ($res->failed() || $status !== 20000) {
            throw new RuntimeException('DataForSEO HTTP '.$res->status().' task '.$status.': '.($res->json('tasks.0.status_message') ?? mb_substr($res->body(), 0, 300)));
        }

        $items = collect($res->json('tasks.0.result.0.items') ?? []);
        $overview = $items->firstWhere('type', 'ai_overview');

        ['text' => $text, 'citations' => $citations] = $overview ? $this->readOverview($overview) : ['text' => '', 'citations' => []];

        $mapPack = $items->where('type', 'local_pack')->values()->map(fn ($p, $i) => [
            'position' => $p['rank_group'] ?? $i + 1,
            'title' => $p['title'] ?? null,
            'domain' => $p['domain'] ?? null,
            'phone' => $p['phone'] ?? null,
            'rating' => $p['rating']['value'] ?? null,
            'reviews' => $p['rating']['votes_count'] ?? null,
            // Google's own lines under the name: town or category, open hours, a review snippet.
            'lines' => array_values(array_filter(array_map('trim', preg_split('/\R/', (string) ($p['description'] ?? ''))))),
            'url' => $p['url'] ?? null,
            'cid' => $p['cid'] ?? null,
            'paid' => (bool) ($p['is_paid'] ?? false),
        ])->all();

        $organic = $items->where('type', 'organic')->values()->map(fn ($o) => [
            'rank' => $o['rank_group'] ?? null, 'domain' => $o['domain'] ?? null, 'url' => $o['url'] ?? null, 'title' => $o['title'] ?? null,
        ])->all();

        return ['text' => $text, 'citations' => $citations, 'map_pack' => $mapPack, 'organic' => $organic, 'shown' => (bool) $overview,
            'serp_url' => $res->json('tasks.0.result.0.check_url')];
    }

    /**
     * Turns the AI Overview into readable text. Business cards (Google listing links) become numbered lines
     * ("1. **Clean Cut Plumbing** 5.0 (271) Plumber"), so the position in the answer can be read; images are dropped.
     */
    public function readOverview(array $overview): array
    {
        $lines = [];
        $n = 0;
        $refs = collect($overview['references'] ?? []);
        foreach ($overview['items'] ?? [] as $item) {
            $itemRefs = collect($item['references'] ?? []);
            $refs = $refs->merge($itemRefs)->merge($item['links'] ?? []);
            $isCard = $itemRefs->contains(fn ($r) => str_contains($r['url'] ?? '', '/searchviewer'));
            // A listing card reads "Name 4.7 (15) Plumber Open 11809 I-70BL 1.2 mi View All ..."; the name is what comes before the rating.
            $cardText = trim(Str::before((string) ($item['text'] ?? ''), ' View All'));
            if ($isCard && preg_match('/^(.+?)\s+(\d\.\d\s*\(\d[\d,]*\).*)$/u', $cardText, $m)) {
                $lines[] = ++$n.'. **'.$m[1].'** '.$m[2];
            } elseif ($isCard && $cardText !== '') {
                $lines[] = ++$n.'. **'.$cardText.'**';
            } else {
                $lines[] = $this->clean((string) ($item['markdown'] ?? $item['text'] ?? ''));
            }
        }
        $text = implode("\n\n", array_filter($lines));
        if ($text === '') {
            $text = $this->clean((string) ($overview['markdown'] ?? ''));
        }

        $citations = $refs->filter(fn ($r) => ! empty($r['url']) && ! str_contains($r['url'], '/searchviewer'))
            ->map(fn ($r) => ['url' => $r['url'], 'title' => $r['title'] ?? $r['domain'] ?? null])
            ->unique('url')->values()->all();

        return ['text' => $text, 'citations' => $citations];
    }

    private function clean(string $markdown): string
    {
        $markdown = preg_replace('/!\[[^\]]*\]\([^)]*\)/', '', $markdown); // images and map tiles
        $markdown = preg_replace('/(\*\*\s*){2,}/', '', $markdown); // empty bold markers left behind

        return trim(preg_replace("/[ \t]+\n/", "\n", $markdown));
    }
}