IP Match & Engage — Technical Documentation

Alternate names: Anonymous IP, Company Match & Engage, IP Match, Web Widget, Web Intent

Last updated: February 2026

---

Table of Contents

1. Overview
2. System Architecture
3. Repository Breakdown
4. End-to-End Flow
5. Provider Chain (IP-to-Company Resolution)
6. Widget Lifecycle
7. Queue Processing Pipeline
8. Source & Engage Lifecycle
9. Credit Management (Clearbit)
10. Cross-System Integration Points
11. Key Entities
12. Configuration Reference

---

1. Overview

IP Match & Engage is Sopro's system for identifying anonymous website visitors by their IP address, resolving them to companies, and then automatically engaging relevant prospects within those companies through outbound campaigns.

What It Does

1. Identifies — A JavaScript widget embedded on a client's website captures the visitor's IP address
2. Resolves — The IP is run through a multi-provider pipeline (Cache → Internal DB → Clearbit Reveal API) to identify the visiting company
3. Enriches — The company is enriched with Sopro's own company data (SoProMasterData) including industry, size, and location
4. Tracks — The visit is recorded and queued for downstream processing as a "web intent" signal
5. Qualifies — A daily automation job qualifies identified companies against campaign targeting criteria (industry, size, exclusions)
6. Engages — Matching prospects are sourced from SoProMasterData, verified, and funnelled through an email finder to join outbound campaigns

Repositories Involved

Repository	Role	Tech Stack
anonymous-ip	IP-to-company resolution API + Azure Functions for async batch processing	.NET 6, ASP.NET Core, Azure Functions v4, EF Core, Clearbit API
sopro-personalisation	JavaScript widget served to client websites, visitor tracking, personalisation engine	ASP.NET Core MVC, Razor-generated JS, Dapper
sopro-sodastream-core	Queue consumers (intent tracker, daily automation, email finder), Portal UI for clients, widget setup	.NET 8, ASP.NET Core MVC, Azure Queues, Dapper
sopro-sodastream	Legacy CRM — no IP Match integration (feature was built entirely in sopro-sodastream-core)	ASP.NET MVC 5

---

2. System Architecture

See the SKILL.md file at .github/skills/ip-match-and-engage/SKILL.md for the full architecture diagram. The system flows:

Client Website (hq.js) → sopro-personalisation (plugin.sopro.io)
    → anonymous-ip (IP resolution) → Clearbit / SoProMasterData (enrichment)
    → Azure Queue (intenttrackervisit, Queue 30)
    → SoProQueueConsumer (TaskIntentTrackerVisit)
    → Daily automation (Queue 42) → prospect sourcing → email finding (Queue 43)
    → Prospect in campaign

---

3. Repository Breakdown

3.1 anonymous-ip — IP Resolution API

Purpose: Resolves IP addresses to companies via a tiered provider chain, manages Clearbit credits, and supports both synchronous and batch (async) processing.

Component	Key Files	Responsibility
WebAPI	AnonymousIPController.cs	REST endpoints: getCountryAndCompanyByIP, createJob, getProcessedItems, createAccount, etc.
Services	MainService.cs, FindingService.cs	Orchestration: validate API key → check ignored IPs → run provider chain → update cache
Providers	CacheProvider.cs, InternalDataProvider.cs, ClearbitProvider.cs	Tiered IP-to-company lookup
IP Checking	IPCheckingService.cs, IPApiProvider.cs	Bot/datacenter/proxy/VPN detection via ipapi.is
External APIs	ClearbitRevealApiService.cs, SoproDataService.cs	Clearbit Reveal for IP lookup, SoProMasterData for company enrichment
Account Mgmt	AccountService.cs, ClearbitPartnershipApiService.cs	Per-client Clearbit child accounts, monthly credit tracking
Azure Functions	ProcessJobFromQueue, ProcessSearchFromQueue, RetryFailedSearch	Async batch processing via Azure Queue Storage
Data	AnonymousIPDataContext (EF Core)	34+ entities: Search, Job, Company, Cache, ClientApp, ClientAccount, etc.

Authentication: API key passed as apiKey query parameter, validated against ClientApp.ApiKey in the database.

3.2 sopro-personalisation — Web Widget

Purpose: A dynamic JavaScript widget (hosted at plugin.sopro.io) embedded on client websites that tracks visits, identifies companies via IP, personalises page content, and integrates with WebChat.

Component	Key Files	Responsibility
Script Engine	ScriptController.cs → Hq()	Dynamically generates JavaScript from Razor views, performs security checks
IP Lookup	ScriptController.cs → IpLookup()	Calls AnonymousIP API, builds Prospect response, queues to intent tracker
Widget Setup	ApiController.cs → GetWidgetCode() / GetWidgetToken()	Creates PropertySettings record with integration token
Personalisation	PersonalisationRuleService.cs	URL-matched rules for text/image/background-image DOM replacement
Security	HqRateLimitMiddleware, IPBlacklistMiddleware, UserAgentMiddleware	Rate limiting (50/min/IP), threat IP blacklist, bot detection

Key entity — PropertySettings: Per-client widget configuration with feature flags: IsAnonymousIpActive, PersonalisationActive, ModalActive, CheckIsBot, CheckExclusionsInclusions, IsExcludedSoproIps, AreClearbitCreditsSpent, IsFullWebChatPersonalizationActive, IsAfterLimitTrackingActive, InsertUnknownVisits

3.3 sopro-sodastream-core — Queue Processing & Portal

Component	Key Files	Responsibility
WebIntentAPI	WebIntent.cs controller	Receives heartbeat signals from widget, forwards to queue
TaskIntentTrackerVisit	Queue 30 consumer	Processes IP-detected company visits → creates DapperPageVisitAnonymous records
TaskIntentTrackerHeartbeat	Queue 34 consumer	Updates session duration for visits
TaskWebIntentAutomationDaily	Queue 42 consumer	The core "Engage" pipeline — qualifies companies, fetches prospects, runs verification, queues for email finding
TaskIPEmailFinder	Queue 43 consumer	Discovers emails for qualified prospects, creates prospect in campaign
Portal - IPDetectorController	IPDetectorController.cs	Client UI: view identified companies, engage prospects, configure targeting
Portal - WebsiteWidgetService	WebsiteWidgetService.cs	Widget setup: creates Clearbit accounts, activates IsAnonymousIpActive

---

4. End-to-End Flow

Phase 1 — Widget Bootstrap (Client-Side)

1. The client's website contains an inline <script> tag that creates a global window.outbase object
2. The script checks for _obid (prospect email GUID) in the querystring or cookie
3. It dynamically loads https://plugin.sopro.io/hq.js?key={integrationToken}&_obid={guid}&__obr={currentPageUrl}

Phase 2 — Server-Side Script Generation (ScriptController.Hq())

1. Validates the integration token → retrieves PropertySettings from the database
2. Runs security checks: bot detection (100+ patterns), browser validation, rate limiting (50/min/IP), IP blacklist
3. Checks URL/IP exclusion rules from PropertySettingsExclusionInclusion
4. If _obid is present, looks up the known prospect in Sopro DB
5. Renders Razor views into JavaScript: Core → WebChat → Personalise → Admin → Final
6. Returns concatenated JavaScript as application/javascript

Phase 3 — Client-Side IP Lookup

1. Core script sets _obid cookie (30-day expiry) and POSTs page visit to tracking URL
2. Starts heartbeat every 3 seconds while tab is active
3. If IsAnonymousIpActive enabled AND no known prospect → calls POST /iplookup
4. /iplookup calls AnonymousIP API → receives IPDetectionResponse → queues to intent tracker (Queue 30) → returns Prospect JSON

Phase 4 — Personalisation (Client-Side)

1. MutationObserver watches the DOM
2. Applies PersonalisationRule records matched by URL (AllPages, ExactMatch, Contains, StartsWith, EndsWith, Regex)
3. Three replacement types: text, img (image src), bgimg (CSS background-image)

Phase 5 — Intent Tracker Processing (Background)

1. TaskIntentTrackerVisit (Queue 30) validates widget, normalizes company size
2. If company identified → creates DapperPageVisitAnonymous record
3. If unknown → creates unknown visit record (if InsertUnknownVisits enabled)

Phase 6 — Daily Automation (Queue 42)

TaskWebIntentAutomationDaily is the heart of the engage pipeline:

1. Load targeting config from CampaignWebIntentTargeting
2. Get identified companies from web intent visit data
3. Qualify against targeting criteria (industry, size) and exclusions
4. Fetch prospects from SoProMasterData API based on job title filters
5. Create SourceAndEngageQueuedProspect records (Status = Pending)
6. Data verification — batches of 20, calls IPMatchAndEngageDataVerification()
7. Valid prospects queued to IPEmailFinder (Queue 43)

Phase 7 — Email Finding (Queue 43)

1. Retrieves SourceAndEngageQueuedProspect
2. Enforces per-domain limit (max 20 per company per campaign)
3. Email found → creates prospect in campaign (Status → AwaitingEngagement)
4. Email not found → Status → EmailNotFound

---

5. Provider Chain (IP-to-Company Resolution)

Providers are executed in order defined by client's FindersOrder config. Execution stops at first usable result.

Provider Details

1. Cache Provider (Free)

• Source: Cache table in AnonymousIP DB
• Sampling: UseCacheForXOutOfYRequests config (e.g., "5/7") — uses search.Id % outOfY for deterministic selection
• Expiry: configurable via CachedEntryExpiresAfterDays (typically 6 months)
• Refresh: if past RefreshAfterDays, calls SoProMasterData API to refresh
• Enrichment: if cached company lacks SoProDataCompanyId, enriches via SoProMasterData API

2. Internal Data Provider (Free)

• Source: IpAddressCompany table — curated internal IP-to-company mappings
• Always attempts enrichment via SoProMasterData API
• Returns IsUsable = true only if SoProMasterData company found (when MustIncludeSDCompanyInFinalResponse is true)

3. Clearbit Provider (1 credit)

• Source: Clearbit Reveal API (reveal.clearbit.com/v1/companies/find?ip=X)
• Credit check: validates ClientAccountHistory.CreditsLeft > 0, syncs with Clearbit every 1000 lookups
• On success: enriches with SoProMasterData, uses Clearbit's ConfidenceScore
• No company: saves as IpAddressISP record
• Credit exhaustion: disables widget, sends notification, sets AreClearbitCreditsSpent = true

SoProMasterData Enrichment Chain

All providers enrich through SoProMasterData API: LinkedIn URL → LinkedIn ID → Email domain

---

6. Widget Lifecycle

Widget Setup Flow

1. Portal admin verifies widget installation
2. Portal checks WidgetValidation.HasClearbit flag
3. If not provisioned: creates Clearbit child account via POST /api/createAccount
4. Activates by setting PropertySettings.IsAnonymousIpActive = 1
5. Widget embedded via: <script src="https://plugin.sopro.io/hq.js?key={integrationToken}"></script>

Widget Script Variants

Variant	Route	Description
hq.js (standard)	GET /hq.js	Security checks server-side; IP lookup client-side
hq2.js (SRI-compatible)	GET /hq2.js	Both security and IP lookup server-side, for Subresource Integrity compliance

Security Layers

Layer	Mechanism	Detail
Bot Detection	User-Agent pattern matching	100+ known crawler/scraper/automation patterns
Browser Validation	Header checking	Rejects missing User-Agent or Referer
Rate Limiting	In-memory counter	50 per IP per minute (HqRateLimitMiddleware)
IP Blacklist	Database + IPsum list	Blocks known-bad IPs (IPBlacklistMiddleware)
URL Exclusions	Per-client rules	Equal, StartsWith, Contains, RegExp match types
IP Exclusions	Per-client + global	Client-specific + Sopro office IPs

---

7. Queue Processing Pipeline

Queue	ID	Trigger	Consumer	Input	Output
intenttrackervisit	30	Widget /iplookup	TaskIntentTrackerVisit	IPDetectionResponse	DapperPageVisitAnonymous record
intenttrackerheartbeat	34	Widget heartbeat (3s)	TaskIntentTrackerHeartbeat	Session ID + timestamp	Updated session duration
webintentautomation	42	Daily schedule	TaskWebIntentAutomationDaily	Campaign targeting config	SourceAndEngageQueuedProspect records
ipemailfinder	43	Output of Queue 42	TaskIPEmailFinder	Prospect ID	Prospect in campaign

AnonymousIP Batch Processing (Azure Functions)

Function	Queue	Purpose
ProcessJobFromQueue	anonymous-ip-job	Reads batch job → enqueues individual searches
ProcessSearchFromQueue	anonymous-ip-search	Processes single IP through provider chain
RetryFailedSearch	anonymous-ip-search-poison	Re-queues failed searches

---

8. Source & Engage Lifecycle

Status Reference

Status	Value	Description
Pending	0	Prospect created, awaiting email finding
AwaitingEngagement	1	Email found, ready for outbound
Engaged	2	Outbound message sent
EmailNotFound	3	Email finder could not discover email
Exclusion	4	Prospect on exclusion/suppression list
Duplicate	5	Already exists in the campaign
Irrelevant	6	Doesn't match targeting criteria
Skipped	7	Per-domain prospect limit reached (max 20)
Responder	8	Prospect responded to outreach
Undelivered	9	Email bounced
ReengagedAgain	10	Re-engaged from a subsequent company visit
EmailNotVerified	11	Email found but failed verification
VerificationFailed	12	SoProMasterData data verification rejected

Campaign Configuration

• Campaign.IPMatchAndEngage must be true
• Campaign.IPMatchAndEngageDate records activation date
• Campaign.SourceAndEngageJobTitleFilter — filters prospects by job title
• Client.SourceAndEngageCredits — credit balance
• EmailProfileTypeEnum.IPMatchAndEngage = 2

---

9. Credit Management (Clearbit)

Credit Lifecycle

1. Provisioning: Portal → POST /api/createAccount → Clearbit child account → ClientAccount with CreditsPerMonth
2. Monthly reset: ClientAccountHistory starts fresh each month
3. Per-lookup deduction: 1 credit per Clearbit call
4. Alignment: Every 1000 lookups, syncs with Clearbit's actual count
5. Exhaustion: disables widget, sends notification, sets AreClearbitCreditsSpent = true
6. Updates: Portal → POST /api/updateAccountCredits

---

10. Cross-System Integration Points

From	To	Mechanism	Purpose
sopro-personalisation	anonymous-ip	REST API	IP-to-company resolution
sopro-personalisation	Azure Queues	Queue (intenttrackervisit)	Queue visit data
anonymous-ip	Clearbit Reveal API	REST API	IP-to-company lookup (paid)
anonymous-ip	Clearbit Partnership API	REST API	Child account & credit management
anonymous-ip	SoProMasterData API	REST API	Company enrichment
anonymous-ip	IPApi	REST API	Bot/datacenter/proxy/VPN detection
anonymous-ip	Sopro Platform	REST API	Disable widget when credits exhausted
sopro-sodastream-core Portal	anonymous-ip	REST API	Clearbit account provisioning
sopro-sodastream-core Portal	sopro-personalisation DB	Direct SQL	Activate/deactivate widget
sopro-sodastream-core QueueConsumer	anonymous-ip DB	Direct SQL (read-only)	Read search/company data
sopro-sodastream-core QueueConsumer	SoProMasterData API	REST API	Fetch prospects, run verification
sopro-sodastream-core WebIntentAPI	Azure Queues	Queue (heartbeat)	Forward heartbeat signals

---

11. Key Entities

AnonymousIP DB

Entity	Key Fields	Purpose
Search	IPAddress, ClientAppId, CompanyId, FinderId, ConfidenceLevel, TotalCost	IP search request and result
Job	ClientAppId, Total, Processed, Found, Status	Batch search job
Company	CompanyName, Website, EmailDomain, LinkedinCompanyId, SoproDataCompanyId	Discovered company
Cache	IPAddress, CompanyId, LastFoundDate	IP-to-company cache
IpAddressCompany	IPAddress, CompanyName, EmailDomain, LinkedInId	Internal curated mappings
ClientApp	ApiKey, FindersOrderId	API consumer
ClientAccount	ClientId, AccountId, CreditsPerMonth	Links client to Clearbit
ClientAccountHistory	Credits, CreditsSpent, CreditsLeft, Year, Month	Monthly credit tracking
Finder	Name, DefaultPrice	Provider definition
FindersOrder	Order (comma-separated IDs)	Provider execution order

sopro-personalisation DB

Entity	Key Fields	Purpose
PropertySettings	IntegrationToken, Domain, ClientId, feature flags	Per-client widget config
PersonalisationRule	IntegrationToken, PageUrl, PageUrlMatchType, RuleContent	DOM replacement rules
PropertySettingsExclusionInclusion	URL/IP rules with comparison types	Per-client URL/IP excludes
IpAddressBlacklist	IP ranges as decimal	Fast IP blacklist checking

sopro-sodastream-core DB

Entity	Key Fields	Purpose
Campaign	IPMatchAndEngage, IPMatchAndEngageDate, SourceAndEngageJobTitleFilter	Campaign config
WidgetValidation	HasClearbit	Widget verification per client
SourceAndEngageQueuedProspect	CampaignId, CompanyId, ProspectName, ProspectEmail, Status	Prospect lifecycle
DapperPageVisitAnonymous	Company + visit metadata	Recorded anonymous page visit
GeneralSetting	AnonymousIPApiUrl, AnonymousIPApiKey	System-wide config

---

12. Configuration Reference

AnonymousIP (DB Config table)

Key	Purpose
ClearbitRevealApi.ApiKey / Url	Clearbit Reveal API credentials
ClearbitPartnershipApi.ApiKey	Clearbit Partnership API key
SoproDataApi.ApiKey / Url	SoProMasterData API config
SoproDataApi.MustIncludeSDCompanyInFinalResponse	Require SoProData match
Cache.UseCacheForXOutOfYRequests	Cache sampling ratio (e.g., "5/7")
Cache.CachedEntryExpiresAfterDays	Cache expiry in days
Cache.RefreshAfterDays	Cache refresh threshold

sopro-personalisation (appsettings)

Key	Purpose
ConnectionStrings:DefaultConnection	Personalisation DB
ConnectionStrings:SoproConnection	Main Sopro DB
AnonymousApi:ApiUrl / ApiKey	AnonymousIP service config
AzureQueue:StorageConnectionString	Azure Storage for queues

sopro-sodastream-core

Key	Source	Purpose
AnonymousIPApiUrl / ApiKey	GeneralSetting	AnonymousIP API config
WidgetPersonalisationWebSite	GeneralSetting	Widget JS URL
PersonalisationDataContext	Connection string	Direct personalisation DB access
AnonymousIPDataContext	Connection string	Direct AnonymousIP DB access (read-only)