Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ For information on how to run each project, see the README in each directory.
| Project | Description |
| ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| [Anchor Browser web scraper](/anchor-browser-web-scraper) | Automated web monitoring using Trigger.dev's job scheduling and Anchor Browser's AI-powered browser automation tools. |
| [Apify lead research](/apify-lead-research) | Lead research for sales with [Apify](https://apify.com) Actors: a scheduled Google Maps scrape that suspends until Apify's webhook arrives, website enrichment, and a [chat agent](https://trigger.dev/docs/ai-chat/overview) that finds and runs Actors after you approve them |
| [Article summary workflow](/article-summary-workflow) | Create audio summaries of newspaper articles using a human-in-the-loop workflow with [ReactFlow](https://reactflow.dev/), [Trigger.dev Realtime](https://trigger.dev/docs/realtime/overview) and [waitpoints](https://trigger.dev/blog/v4-beta-launch#waitpoints) |
| [Batch LLM evaluator](/batch-llm-evaluator) | Batch processing tool for evaluating LLM responses from a single prompt using Vercel's [AI SDK](https://sdk.vercel.ai/docs/introduction) and [Trigger.dev Realtime](https://trigger.dev/docs/realtime/overview) |
| [Building effective agents](/building-effective-agents) | 5 different patterns for building effective AI agents with Trigger.dev; [Prompt chaining](/building-effective-agents/src/trigger/trigger/translate-copy.ts), [Routing](/building-effective-agents/src/trigger/trigger/routing-questions.ts), [Parallelization](/building-effective-agents/src/trigger/trigger/parallel-llm-calls.ts), [Orchestrator-workers](/building-effective-agents/src/trigger/trigger/orchestrator-workers.ts) |
Expand Down
18 changes: 18 additions & 0 deletions apify-lead-research/.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Your Trigger.dev project ref (from the project settings page in the dashboard)
TRIGGER_PROJECT_REF=proj_xxxxxxxxxxxxxxxxxxxxxxxx

# Your Trigger.dev secret key (API keys page in the dashboard). Used by the
# Next.js server actions to create chat sessions and mint session tokens.
TRIGGER_SECRET_KEY=tr_dev_xxxxxxxxxxxxxxxxxxxxxxxx

# Used by the tasks. `trigger dev` reads them from this file. For deployed runs,
# set them on the Environment Variables page in the Trigger.dev dashboard.
#
# Apify API token: https://console.apify.com/settings/integrations?utm_source=trigger.dev&utm_medium=integrations
APIFY_TOKEN=apify_api_xxxxxxxxxxxxxxxxxxxxxxxx
# Anthropic API key: https://console.anthropic.com/settings/keys
ANTHROPIC_API_KEY=sk-ant-xxxxxxxxxxxxxxxxxxxxxxxx

# Only needed if you're not using cloud.trigger.dev (e.g. self-hosting):
# TRIGGER_API_URL=https://your-trigger-instance.example.com
# NEXT_PUBLIC_TRIGGER_API_URL=https://your-trigger-instance.example.com
6 changes: 6 additions & 0 deletions apify-lead-research/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
node_modules
.env
.trigger
.next
next-env.d.ts
*.tsbuildinfo
127 changes: 127 additions & 0 deletions apify-lead-research/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
# Lead research with Apify and Trigger.dev

A lead research desk for a sales team, built with [Trigger.dev](https://trigger.dev) and [Apify](https://apify.com) Actors:

- A **scheduled task** finds businesses on Google Maps for a search term and location.
- An **enrichment task** reads each business's website and extracts contact details and a summary.
- A **chat agent** answers questions the fixed pipeline doesn't cover, such as recent reviews, by finding and running an Apify Actor after you approve it.

Each part runs Apify Actors in a different way, because each has a different job:

| Pattern | Task | Use it when |
| --- | --- | --- |
| Start the Actor and suspend until its webhook arrives | `run-actor`, used by `discover-leads` | The Actor takes minutes to hours |
| Wait for the Actor inline | `enrich-lead` | The Actor finishes in seconds |
| Agent tools with approval | `research-agent` | The question decides which Actor runs |

## How it works

### Bulk discovery: suspend on a webhook

`discover-leads` (`src/trigger/discover-leads.ts`) is a [scheduled task](https://trigger.dev/docs/tasks/scheduled). It runs [Google Maps Scraper](https://apify.com/compass/crawler-google-places) through `run-actor` (`src/trigger/run-actor.ts`), which:

1. Creates a [wait token](https://trigger.dev/docs/wait-for-token).
2. Starts the Actor with an [ad-hoc webhook](https://docs.apify.com/platform/integrations/webhooks/ad-hoc-webhooks) that posts to the token's URL when the run ends, for all four terminal events: succeeded, failed, aborted and timed out.
3. Suspends on `wait.forToken()` until Apify calls back, then reads the dataset.

A suspended run uses no compute, so a two-hour scrape costs the same on the Trigger.dev side as a two-minute one. Nothing polls Apify while it works.

Nobody watches a scheduled job, so the Actor's spend is capped. Google Maps Scraper charges per place (pay-per-event), so the cap that applies is `maxTotalChargeUsd`. `discover-leads` also passes `maxItems`, which caps pay-per-result Actors. When a run reaches `maxTotalChargeUsd`, Apify aborts it, so `run-actor` returns the partial dataset of aborted and timed-out runs with their `status`, and only throws when the Actor fails. `run-actor` doesn't retry, because a retry would start a new paid Actor run.

### Enrichment: wait inline

`enrich-lead` (`src/trigger/enrich-lead.ts`) runs [Website Content Crawler](https://apify.com/apify/website-content-crawler) on one page with `call()`, which waits for the Actor to finish. Crawling one page takes seconds, so suspending would save almost nothing, and waiting inline keeps the code short. For the bulk scrape, the same approach would keep the run, and its compute bill, alive for the whole scrape.

The task then asks the model to extract emails, phone numbers, social profiles and a summary from the page.

### Agent: Apify Store tools with approval

`research-agent` (`src/trigger/research-agent.ts`) is a [`chat.agent`](https://trigger.dev/docs/ai-chat/overview) with three tools (`src/trigger/apify-tools.ts`):

- `searchApifyActors` searches [Apify Store](https://apify.com/store) for an Actor.
- `getApifyActorInputSchema` returns the Actor's input fields, so the model doesn't guess them.
- `runApifyActor` runs the Actor through `run-actor`. It has `needsApproval: true`, so every paid run waits for a person to approve it. The output is trimmed to about 50,000 characters, without empty fields, with the `datasetId` for the full dataset.

## Setup

1. Create a project in the [Trigger.dev dashboard](https://cloud.trigger.dev) and copy its project ref and a dev secret key (API keys page).
2. Get an [Apify API token](https://console.apify.com/settings/integrations?utm_source=trigger.dev&utm_medium=integrations) and an [Anthropic API key](https://console.anthropic.com/settings/keys).
3. Configure the environment:

```sh
cp .env.example .env
# fill in TRIGGER_PROJECT_REF, TRIGGER_SECRET_KEY, APIFY_TOKEN and ANTHROPIC_API_KEY
```

The tasks use Claude Sonnet 5.5, set in `src/trigger/model.ts`. Smaller models follow Actor input schemas less reliably, so expect more failed or denied runs if you switch to one.

4. Install and run both processes (two terminals):

```sh
pnpm install
pnpm dev:trigger # the tasks and the agent
pnpm dev # the Next.js app
```

## Try it

### `discover-leads`

Scheduled tasks don't take a payload, so each schedule carries its search in the external ID, as `<search term> | <location>`. In the dashboard, open **Schedules**, create a schedule for `discover-leads`, and set the external ID, for example:

```
coffee roastery | Prague, Czechia
```

To run it once without waiting for the cron, open `discover-leads` on the **Tasks** page, press **Test schedule** and fill in the same **External ID**. The run returns the leads and triggers `enrich-lead` for the first five with a website. In the run trace, `run-actor` shows the time the Actor was working as a `wait.forToken()` span.

### `enrich-lead`

Test it from the dashboard with:

```json
{
"name": "25 Coffee Roasters",
"website": "https://25coffeeroasters.com/"
}
```

### `run-actor`

`discover-leads` and the agent call it, but you can also test it directly:

```json
{
"actorId": "apify/website-content-crawler",
"input": {
"startUrls": [{ "url": "https://docs.apify.com/academy" }],
"maxCrawlPages": 1,
"proxyConfiguration": { "useApifyProxy": true }
},
"maxItems": 10
}
```

### `research-agent`

Open [http://localhost:3000](http://localhost:3000). The dashboard playground can't approve tool calls, so approval only works in this app. Ask, for example:

> Write a personalized opening line for a cold email to The Miners Coffee Roastery in Prague, based on their recent Google reviews.

The agent searches Apify Store, reads the Actor's input schema and asks before running it. Deny a run to see the agent try something else.

## Deploy

```sh
pnpm deploy:trigger
```

Set `APIFY_TOKEN` and `ANTHROPIC_API_KEY` on the Environment Variables page for the Prod environment. Deploy the Next.js app anywhere with a prod `TRIGGER_SECRET_KEY`.

The app has no authentication: `mintChatAccessToken` issues a token for any chat ID, and every approved run spends your Apify and Anthropic credit. Before you make it public, put it behind your own sign-in and limit how many agent runs can run at once, for example with a [queue](https://trigger.dev/docs/concurrency) on `research-agent`.

## Things to know

- **Cancelling a task doesn't stop its Actor run.** Abort it in [Apify Console](https://console.apify.com?utm_source=trigger.dev&utm_medium=integrations) or with `apify.run(runId).abort()`.
- **`enrich-lead` retries up to three times**, and each retry crawls the page again. A one-page crawl is cheap, so this keeps the code simple. `run-actor` doesn't retry at all, because its runs can be expensive.
- **`apify-client` is marked as external** in `trigger.config.ts`. Bundled, it fails with `TypeError: ProxyAgent is not a constructor`.
5 changes: 5 additions & 0 deletions apify-lead-research/next.config.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
import type { NextConfig } from "next";

const nextConfig: NextConfig = {};

export default nextConfig;
36 changes: 36 additions & 0 deletions apify-lead-research/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
{
"name": "apify-lead-research",
"version": "0.1.0",
"private": true,
"scripts": {
"dev": "next dev",
"build": "next build",
"start": "next start",
"typecheck": "tsc --noEmit",
"dev:trigger": "trigger dev",
"deploy:trigger": "trigger deploy"
},
"dependencies": {
"@ai-sdk/anthropic": "4.0.78",
"@ai-sdk/react": "4.0.140",
"@trigger.dev/sdk": "4.7.3",
"ai": "7.0.137",
"apify-client": "2.25.0",
"next": "16.4.0",
"react": "19.3.0",
"react-dom": "19.3.0",
"react-markdown": "10.1.0",
"remark-gfm": "4.0.1",
"zod": "4.6.6"
},
"devDependencies": {
"@tailwindcss/postcss": "4.3.3",
"@trigger.dev/build": "4.7.3",
"@types/node": "22.20.5",
"@types/react": "19.3.0",
"@types/react-dom": "19.3.0",
"tailwindcss": "4.3.3",
"trigger.dev": "4.7.3",
"typescript": "5.9.3"
}
}
Loading