Web Intent — Global

Overview — The big picture

Read this first. This tab explains what Web Intent is and how the pieces fit together, for any audience. Then choose the User tab (how to use it) or the Developer tab (how it's built).

What it does

When a visitor lands on a client's website, a small injected widget reports the visit to Sopro. There are two kinds of visits:

1. Known visitor (email click) — the visitor arrived from a tracked Sopro email link, so the exact person is already known. The visit is stored against that prospect.
2. Anonymous visitor — no identity is known. Sopro identifies the company behind the visitor's IP address. Matching companies are then, overnight, turned into new contactable prospects.

That second path is what the business calls "IP Match and Engage" / "Web Intent".

On top of both paths sits a real‑time alerting layer called goals: when a visit hits a key page, returns after a set number of days, or reaches a set number of visits, Sopro can email the client instantly so sales can act while interest is hot.

The journey at a glance

Systems involved

The pipeline spans four repositories / solutions in this workspace:

#	Repository / Solution	Role in the pipeline	Framework
1	sopro-personalisation	Serves the client‑side widget (hq.js) and the IpLookup endpoint that calls AnonymousIP	.NET Core 3.1
2	soproprospecting (SoPro.sln)	WebsiteWidgetController (email‑click visits) + WidgetPageVisit entity + Hangfire nightly scheduler	.NET Framework
3	anonymous-ip (AnonymousIP.sln)	Identifies a company from an IP address (Cache → Internal → Clearbit)	.NET 6
4	sopro-sodastream-core	Azure Queue consumers: persist anonymous visits, run the nightly automation, find prospect emails	.NET

External services: Clearbit Reveal API, SoproData / SoProMasterDB API, IP geolocation, Azure Queue Storage, Hangfire.

How visits are filtered — bot detection & short‑visit cleanup

Not every HTTP request to the widget becomes a recorded visit. Two safeguards keep the data clean:

🤖 Bot detection

Automated crawlers, scrapers, and bots are filtered at two layers:

• Client‑side — the widget JavaScript checks an IsBot flag before sending any tracking data. If the visitor's User‑Agent or request pattern matches known bot signals, the visit is silently dropped.
• Server‑side (anonymous-ip) — before performing an expensive company look‑up, the IP check provider inspects whether the IP belongs to a datacenter (e.g. Microsoft Azure, Google Cloud) and the organisation's own email provider (e.g. Office 365, Gmail). This catches headless browsers and API clients running on cloud infrastructure.

If either layer flags the visit as a bot, it is discarded — no visit is recorded, no company look‑up is performed, and no credits are consumed.

⏱️ The 5‑second rule

The widget sends periodic heartbeats to measure how long a visitor stays on the page. Anonymous visits start at 5‑second intervals and back off to 10s → 15s → 30s → 60s for longer sessions. After 60 seconds, a cleanup task checks whether the visitor stayed long enough:

• ≤ 5 seconds → the visit is considered non‑engagement (likely an accidental click, a bounce, or an un‑rendered bot) and is deleted.
• > 5 seconds → at least one heartbeat fired, so the visit is kept.

This eliminates fleeting noise while preserving genuine visits.

🚫 Intent Tracker Exclusions

When a client marks a domain as excluded in the Sopro portal (general exclusion or competitor exclusion), a dedicated queue task — ProcessIntentTrackerExclusions — propagates that exclusion into the Web Intent data. It updates the WebIntentCompanyCampaignTargeting join table to mark all matching companies as IsExclusion = 1, which prevents the nightly automation from ever turning them into prospects. The exclusion can be scoped to a single campaign, all campaigns for a client, or globally.

How the pieces fit together

The diagram below is the whole journey as a guide — every application involved and both visitor paths (📧 known email‑click and 🕵️ anonymous company). The two paths end differently:

• 📧 Known / email‑click ("Prospect Visit") ends at 🎯 goals → instant alert emails — it never enters email finding.
• 🕵️ Anonymous / Web Intent can also raise company goal alerts, and additionally flows overnight to the ⭐ IP Email Finder task that produces a brand‑new contactable prospect.

Each box (swimlane) is one application.

💡 Reading it: the 📧 known path saves a WidgetPageVisit, evaluates goals, and — if a goal is met — ends by sending an instant alert email to the prospect owner. It does not feed email finding. The 🕵️ anonymous path identifies the company, can raise its own company goal alerts, and every night the automation qualifies companies and hands them to ⭐ TaskIPEmailFinder — the last task before a new prospect appears (Web Intent path only).

🔔 Instant alerts — goals (both paths)

Both visit types are checked in real time against three goals, and a matching goal can email your team immediately:

Goal	Fires when…
🎯 Visited a key page	The visitor opens a configured high‑intent page (e.g. /pricing)
🔁 Returning interest	The visitor comes back after a set number of days
📈 Repeat visitor	The visitor makes a set number of separate visits

• 📧 Known visitor → TrackGoals → alert to the prospect owner (nodes GOAL → GA1 above).
• 🕵️ Company (anonymous) → ProcessWebGoals → alert to the sales owners (nodes CG → GA2 above).

Full detail is in the User and Developer tabs.

Where to go next

• [User tab] — what you set up (targeting), what you see (companies, visit counts, lead/stage status), timing, privacy, and FAQs.
• [Developer tab] — the full technical walkthrough: endpoints, services, providers, queues, database tables, and source‑file references.

💰 Clearbit Credits — How They Work

Web Intent identifies companies using a three‑layer provider system:

1. Cache — companies we've already seen (free, instant)
2. Internal Database — Sopro's own company database (free)
3. Clearbit — external API that costs credits per lookup

How credits are consumed

• Each client gets a Clearbit account with a monthly credit allowance
• A credit is consumed only when the first two providers fail and Clearbit returns a successful company match
• Credits are not consumed for: cached companies, internal DB matches, bot visits, excluded IPs, or visits that bounce within 5 seconds

When credits run out

When a client exhausts their Clearbit credits, several things happen automatically:

1. The AreClearbitCreditsSpent flag is set to true on all the client's PropertySettings rows — this stops the widget from sending further IP lookups to the AnonymousIP service
2. The IsAfterLimitTrackingActive flag takes over — visits are still recorded (IP, page URL, referrer, device info) but without doing any company identification, saving the visit records in the AfterLimitAnonymousRequest table
3. The client's CS (Customer Success) and Ops owners are emailed automatically with a notification that credits are spent
4. The client's IsNotifiedForClearbitCredits flag prevents duplicate notifications

How credits are restored

Credits can be topped up via the AnonymousIP API (/api/updateAccountCredits). Once credits are restored, the AreClearbitCreditsSpent flag must be manually set back to false in the PropertySettings table, and the IsNotifiedForClearbitCredits flag reset — this re-enables full company identification on the widget.

IsAfterLimitTrackingActive (After-Limit Tracking)

When IsAfterLimitTrackingActive is true (enabled automatically when credits run out, or manually by Ops), the widget switches to a lightweight tracking mode:

• The browser still sends visit data (IpLookup POST)
• The IsAfterLimitTracking flag in the request signals the server to skip the AnonymousIP / Clearbit lookup entirely
• Instead, the visit is saved directly to the intenttrackervisit queue with IsAfterCreditLimit = true
• These visits are stored in the AfterLimitAnonymousRequest table (not PageVisitAnonymous), containing: ClientId, IP address, page visited, referrer, device type, OS, and browser
• A weekly cleanup job (RemoveOldUnknownAndAfterLimitVisits, runs Sundays at 9pm) deletes AfterLimitAnonymousRequest records older than 90 days

This ensures clients never lose visit data entirely — even when credits are exhausted, you can still see which companies are showing interest (by IP), just without full company identification.