Skip to main content



Web Intent — Developer

Web Intent / IP Match & Engage — Technical Documentation

Audience: Developers, architects, and technical support engineers. Scope: The complete pipeline, from the JavaScript widget through company identification, visit persistence, the nightly qualification automation, and prospect email finding.

1. Systems involved​

#Repository / SolutionRoleFramework
1sopro-personalisationWidget (hq.js) + IpLookup endpoint.NET Core 3.1
2soproprospecting (SoPro.sln)WebsiteWidgetController + WidgetPageVisit + Hangfire scheduler.NET Framework
3anonymous-ip (AnonymousIP.sln)Company identification (Cache → Internal → Clearbit).NET 6
4sopro-sodastream-coreQueue consumers: persistence, nightly automation, email finding.NET

2. Architecture​

3. Stage‑by‑stage detail​

Stage 1 — The client‑side widget (sopro-personalisation)​

1a. Bootstrap script — Controllers/ApiController.cs → GetWidgetCode(string Domain, int PropertyTypeId, int ClientId). Clients paste a small <script> that creates window.outbase, reads the _obid visitor id from the query string or the __outbasep cookie, and loads https://sopro-personalisation.azurewebsites.net/hq.js.

1b. Widget rendering — Controllers/ScriptController.cs → Hq(key) / HqSpa(key) → HqInternal(key, isSpa). Validates the integration token (PropertySettings), applies security checks (bot detection, IP blacklist, rate limiting), evaluates inclusion/exclusion rules, and builds an HqScriptViewModel exposing two tracking URLs:

ViewModel propertyPurpose
PageVisitUrl / PageVisitHearthBeatUrlKnown visitor (email click) tracking
AnonymousPageVisitUrl / AnonymousPageVisitHearthBeatUrlAnonymous (Web Intent) tracking
IsAnonymousIpActiveFeature flag enabling anonymous tracking

1c. Client‑side branching — Views/Script/CoreSPA.cshtml:

if (!IsBot && _obid != undefined && _obid != null) {
    // TYPE 1 — EMAIL CLICK (known prospect): POST to PageVisitUrl
} else if (!IsBot) {
    // TYPE 2 — ANONYMOUS IP (Web Intent): POST to the IpLookup endpoint
}

Data collected: page URL + query string, User‑Agent (browser/OS/device), referrer, timestamps, client IP (resolved server‑side), and _obid when present. Heartbeats measure engagement (email path ~5s; anonymous path 5s → 10s → 15s → 30s → 60s back‑off).

Stage 2A — Known visitor / email‑click path (soproprospecting)​

Controller: API/SoPro.API/Controllers/WebsiteWidgetController.cs → PageVisit(PageVisitPostViewModel model) — POST /WebsiteWidget/PageVisit.

  1. Resolve client IP from HTTP_X_FORWARDED_FOR → REMOTE_ADDR → UserHostAddress; capture User-Agent.
  2. Map the POST body into a PageVisitViewModel.
  3. Look up the prospect: _prospectMailService.GetProspectEmailByWidgetId(WidgetGuidId).
    • Found → set ProspectEmailId, geo‑locate the IP, persist via _websiteWidgetService.PageVisit(...) into WidgetPageVisit, then _prospectIntentTrackingService.TrackGoals(...).
    • Not found → return false (visit rejected).

Entity: SoProEntities/WidgetPageVisit.cs (table WidgetPageVisit). Key columns: PageVisitId (Guid, PK), VisitId, WidgetGuidId (_obid), ProspectEmailId (FK), WebSiteWidgetClientId, PageUrl/QueryString, Created/LastTimeSeen/TimeSpent/Tick, IpAddress, Browser/OS/Device/DeviceType, Referrer/ReferrerSource/ReferrerProvider, Country/Region/City, IsArchived.

The email‑click path is self‑contained and does not enter the AnonymousIP company‑detection pipeline.

Stage 2B — Anonymous visitor path (sopro-personalisation)​

Endpoint: Controllers/ScriptController.cs → IpLookup([FromBody] IPLookupModel model) — POST /Script/iplookup.

  1. Validates the page URL. If credits are exhausted (IsAfterLimitTracking), performs light tracking and returns an empty Prospect.
  2. Calls the AnonymousIP API:
GET {AnonymousIPApiUrl}/api/getCountryAndCompanyByIP
    ?apiKey={AnonymousIPApiKey}
    &ipAddress={model.IPAddress}
    &clientSideSearchId=NULL
    &clientSideClientId={model.ClientId}
  1. Deserializes into IPDetectionResponse / IPDetectionCompany; decorates with ClientId, ClientWidgetId, IpAddress, PageUrl, PageVisitId, Created.
  2. Builds the Prospect returned to the widget: if SoproDataCompanyId > 0 populate ProspectCompany; if CompanyName is empty set prospect.Company.IsUnknown = true.
  3. Enqueues the visit:
await _queueService.AddMessage("intenttrackervisit", new MessageModel {
    QueueId      = (int)SoProQueueEnum.IntentTrackerVisit,
    Message      = JsonConvert.SerializeObject(_compositeCompany),
    SearchTypeId = (int)SoProQueueSearchTypeEnum.IntentTrackerVisit
});
  1. Returns the Prospect so the widget can personalise the page in real time. AnonymousIPApiUrl/AnonymousIPApiKey come from appsettings.json/secrets.

Stage 3 — Company identification (anonymous-ip, .NET 6)​

Controller: WebAPI/Controllers/AnonymousIPController.cs, route prefix [Route("api")].

EndpointPurpose
GET /api/getCountryAndCompanyByIPSynchronous single‑IP lookup (used by IpLookup)
POST /api/createJobAsynchronous batch lookup (list of IPs → Azure queue)

Service chain (Services/MainService.cs → GetCountryAndCompanyByIpAddress):

var client = _ClientAppService.GetClientApp(clientToken);      // 1. validate token
var result = await RunSingleSearch(client, ssModel, null);     // 2. find company
result.CountryFromIPRange =
    _LocationService.GetCountryByIpAddress(ssModel.IPAddress); // 3. add country
return result;                                                  // SingleSearchCallbackResult

Provider resolution (Services/FindingService.cs → FindCompany) iterates providers from ClientApp.FindersOrder and stops at the first usable result:

foreach (var finder in clientApp.FindersOrder.Order.Split(','))
{
    var provider = _AnonymousIPProviderFactory.Create(byte.Parse(finder), price, configs);
    var providerResult = await provider.FindCompany(model, search);
    if (providerResult?.IsUsable == true) return providerResult;
}

Providers (Services/AnonymousIPProviderFactory.cs):

IdProviderDescription
0CacheFast lookup of previously resolved IPs
1InternalDataInternal IP→company DB, enriched via SoproData
2ClearbitClearbit Reveal API (https://reveal.clearbit.com/v1/companies/find?ip=) — consumes credits

Result (Models/SingleSearchCallbackResult): CompanyName, CompanyEmailDomain, CompanyWebsite, CompanySize, CompanyIndustry, LinkedinCompanyId, LinkedinHandle, SoproDataCompanyId, CountryFromIPRange, IsUsable.

Stage 4 — Persist the anonymous visit (sopro-sodastream-core)​

Consumer: SoProQueue/Consoles/SoProQueueConsumer/SoProQueueConsumer/Partials/TaskIntentTrackerVisit.cs (intenttrackervisit queue):

var obj = JsonConvert.DeserializeObject<IPDetectionCompany>(param.Message);
if (string.IsNullOrEmpty(obj.CompanyEmailDomain) && string.IsNullOrEmpty(obj.CompanyName))
    await _soProQueueIntentTrackerService.ProcessUnknownVisit(obj);   // → PageVisitAnonymousUnknown
else
    await _soProQueueIntentTrackerService.ProcessVisit(dapperPageVisit, obj); // → PageVisitAnonymous + WebIntentCompanies

Persistence: Dapper/DapperSopro/DapperSopro/DapperWebsiteWidgetService.cs → AddAnonymousPageVisit(v, obj):

  1. Look up an existing WebIntentCompanies row by (EmailDomain, ClientId, SoproDataCompanyId):
    • Exists → UPDATE WebIntentCompanies SET TotalNumberOfVisits += 1, LastVisit = @LastVisit.
    • New → INSERT, deriving LeadStatus (Not engaged / Engaged / SQL / MQL) and StageStatus (New / Active engagement / Opportunity for reengagement) from ProspectEmail history.
  2. Always INSERT a detailed PageVisitAnonymous row.

Tables:

  • PageVisitAnonymous — one row per resolved anonymous visit: PageVisitId, SessionId, ClientId, ClientWidgetId, IpAddress, Country, Company, CompanyNameClean, CompanyNameFromProvider, SoproDbCompanyId, SoproDataCompanyId, LinkedInCompanyId, Website, EmailDomain, CompanySize, Industry, PageVisited, LeadStatus, SessionDurationTotalSeconds, VisitDate, device/referrer fields, IsArchived.
  • PageVisitAnonymousUnknown — one row per visit with no company: ClientId, ClientWidgetId, IpAddress, Country, PageVisited, VisitDate, optional device/referrer fields.
  • WebIntentCompanies — aggregation of unique companies per client/widget, deduplicated on (ClientId, ClientWidgetId, Company, EmailDomain, SoproDataCompanyId). Tracks TotalNumberOfVisits, LastVisit, LeadStatus, StageStatus, plus the full Pelias geographic hierarchy. This is the source table the nightly automation reads from.

Stage 5 — Nightly qualification automation​

5a. Scheduler — soproprospecting/.../Services/HangFireJobServiceDB.cs:

// InitDatabaseProcedures()
RecurringJob.AddOrUpdate(() => _AddWebAutomationDaily(), Cron.Daily(0, 1)); // 00:01 UTC daily

_AddWebAutomationDaily(): finds every CampaignWebIntentTargeting where IsDeleted == false && TurnOnAuto == true; creates one WebIntentAutomation per targeting (LastVisit = yesterday, PerformBackDate = true); enqueues one message per record onto webintentautomation (SoProQueueEnum.WebIntentAutomation = 42) with the record Id.

5b. Consumer — SoProQueueConsumer/Partials/TaskWebIntentAutomationDaily.cs:

Company qualification — IsWiCompanyQualifiedSimple(targeting, company, allIndustries, allCompanySizes):

FilterRule
Company sizeIf targeting specifies sizes, CompanySize must be one of them (exact match).
Industry / groupQualifies if the industry matches a targeted industry OR a targeted industry group (logical OR).
LocationParsed from CompanyLocation JSON. Must match at least one Included location (name or Pelias id: country / macro‑region / region / locality / continent) and match no Excluded location (excluded match vetoes).

Additional gating: companies are skipped when StageStatus == "Active engagement" or ProspectCount == 0. Telemetry counters (CompaniesIdentified, CompaniesQualified, CompaniesQualifiedWithProspects, ProspectCount, …) are updated throughout.

Prospect assembly & verification:

  1. _dapperSoProMasterDBAdapter.GetProspects(targeting, SoproDataCompanyId) returns candidate prospects.
  2. The company is upserted via _dapperCompanyService.UpsertByCampaignIdAndEmailDomain.
  3. A DapperSourceAndEngageQueuedProspects row (status Pending) is built per prospect, assigning an EmailProfileId in round‑robin (SelectNext).
  4. Prospects are batched (20 per call) and sent to dataVerificationAdapter.IPMatchAndEngageDataVerification(IPMatchDto). Items not Verified are marked VerificationFailed.
  5. Rows are persisted; verified ids are enqueued onto ipemailfinder (SoProQueueEnum.IPEmailFinder = 43).

Key service fields: _dapperWebIntentAutomationService, _dapperIntentTrackerTargetingService, _dapperWebIntentCompanyAutomationDailyService, _dapperSoProMasterDBAdapter, _dapperCompanyService, _dapperSourceAndEngageQueuedProspectsService, _soProQueueService.

Stage 6 — Prospect email finding (sopro-sodastream-core)​

Consumer: SoProQueueConsumer/Partials/TaskIPEmailFinder.cs (ipemailfinder queue). For each SourceAndEngageQueuedProspects id with status Pending:

  1. Fetch full prospect detail from SoProMasterDB.
  2. Enforce a max of 20 prospects per company domain per campaign.
  3. Build a SimpleSearchModel and call the EmailFinder 2.0 nuget (FindEmailWithNewEmailFinder_Nuget).
  4. On success, create a DapperProspect (status New or Verify depending on WebIntentVerifyDays) and queue it for email composition (generative templates).

A formerly anonymous website visitor is now a contactable prospect — completing the IP Match and Engage loop.

Goals — real‑time client notifications (both paths)​

Separately from the nightly automation, every page visit is evaluated in real time against a set of goals. When a goal is met, Sopro can send an alert email to the client (the sales owner and/or a configured list of colleagues). The same three goal types apply to both the known/email‑click path (per prospect) and the anonymous/Web Intent path (per company).

The three goal types:

GoalFires when…Config field
Goal URL (certain pages)The visitor views a configured high‑intent pageUrls + ProspectIntentTrackingUrls
Returning Interest (repeat visitor)The visitor returns after N daysVisitorsReturningAfterDays
Multiple VisitsThe visitor makes N separate visits (≥ 4h apart)VisitorsReturningAfterTimes

URL matching supports ExactMatch / Contains / StartsWith plus Inclusion / Exclusion rules. A 12‑hour cooldown applies per URL goal and a 24‑hour throttle blocks duplicate goals per prospect/company/IP.

Configuration lives on the per‑client ProspectIntentTracking entity (soproprospecting/.../SoProEntities/ProspectIntentTracking/ProspectIntentTracking.cs), edited via the Web Intent portal (sopro-sodastream-core/.../Portal.Web/Controllers/WebSiteCatcherController.cs). Key fields: IsProspectIntentGoal, IsCompanyIntentGoal, IsEngagedCompanyGoals, IsQualifiedCompanyGoals, IsInstantAlert (vs IsDailySummary / IsWeeklySummary), Urls, VisitorsReturningAfterDays, VisitorsReturningAfterTimes, and the per‑goal IsAlertingOwner* / IsAlertingOthers* / AlertOthers* recipients. Global kill‑switch: GeneralSettings.ProspectIntentGoalEmailNotifications.

Path A — Prospect goals (known visits): WebsiteWidgetController.PageVisit() → ProspectIntentTrackingService.TrackGoals() → ProcessWebGoal() evaluates the three goals, writes a ProspectIntentTrackingGoal row, and (if IsInstantAlert + IsProspectIntentGoal) SendNotification() emails the prospect's owner + AlertOthers* using template 206 (ProspectWebVisitInstantAlert) or 182 (white‑label).

Path B — Company goals (anonymous visits): TaskIntentTrackerVisit → DapperWebsiteWidgetService.ProcessWebGoals() evaluates the same goals against the company's IP history, writes a CompanyIntentTrackingGoal row (flags IsEngagedCompanyGoal / IsQualifiedCompanyGoal), and (if IsInstantAlert + IsCompanyIntentGoal) emails the sales owners (qualified + engaged) + AlertOthers* using template 207 (EngagedCompanyWebgoal) or 208 (QualifiedCompanyWebgoal).

AspectProspect goalsCompany goals
VisitorKnown (_obid)Anonymous (IP → company)
Entry pointTrackGoalsProcessWebGoals
Goal recordProspectIntentTrackingGoalCompanyIntentTrackingGoal
Emailed toProspect owner + AlertOthers*Qualified + engaged owners + AlertOthers*
Templates206 / 182207 / 208
Master switchIsProspectIntentGoalIsCompanyIntentGoal

4. Queue reference​

Queue nameEnum (SoProQueueEnum)ProducerConsumerPayload
intenttrackervisitIntentTrackerVisitScriptController.IpLookupTaskIntentTrackerVisitSerialized IPDetectionCompany
webintentautomationWebIntentAutomation (42)HangFireJobServiceDB._AddWebAutomationDailyTaskWebIntentAutomationDailyWebIntentAutomation.Id
ipemailfinderIPEmailFinder (43)TaskWebIntentAutomationDailyTaskIPEmailFinderSourceAndEngageQueuedProspects.Id
intenttrackerexclusionIntentTrackerExclusionExclusionService.AddExclusions (when domain is excluded)TaskIntentTrackerExclusion.ProcessIntentTrackerExclusionsSerialized ExclusionQueueModel

5. Data model reference​

Table / EntityRepository (source file)Purpose
WidgetPageVisitsoproprospecting/.../SoProEntities/WidgetPageVisit.csKnown (email‑click) page visits
PageVisitAnonymoussoproprospecting/.../SoProEntities/PageVisitAnonymous.csResolved anonymous visits (company found)
PageVisitAnonymousUnknownsoproprospecting/.../SoProEntities/PageVisitAnonymousUnknown.csUnresolved anonymous visits (no company)
WebIntentCompaniessopro-sodastream-core/.../SoProEntities/WebIntentCompanies.csAggregated companies per client/widget (automation source)
WebIntentCompanyCampaignTargetingsopro-sodastream-core (join table)Links companies to campaign targeting
CampaignWebIntentTargetingsopro-sodastream-core/.../DapperModels/WebIntent/DapperCampaignWebIntentTargeting.csTargeting rules
WebIntentAutomationsopro-sodastream-core/.../DapperModels/DapperWebIntentAutomation.csOne nightly run per active targeting
WebIntentCompanyAutomationDailysopro-sodastream-core/.../DapperModels/DapperWebIntentCompanyAutomationDaily.csPer‑run snapshot + telemetry
SourceAndEngageQueuedProspectssopro-sodastream-core/.../DapperModels/DapperSourceAndEngageQueuedProspects.csProspects queued for email finding
Prospectsopro-sodastream-coreFinal contactable prospect
ProspectIntentTrackingsoproprospecting/.../SoProEntities/ProspectIntentTracking/ProspectIntentTracking.csPer‑client goal & alert configuration
ProspectIntentTrackingGoalsoproprospecting/.../SoProEntities/ProspectIntentTracking/ProspectIntentTrackingGoal.csGoal met by a known prospect visit
CompanyIntentTrackingGoalsoproprospecting/.../SoProEntities/CompanyIntentTrackingGoal.csGoal met by an anonymous company visit

6. Source file index​

ConcernFile
Bootstrap scriptsopro-personalisation/Controllers/ApiController.cs (GetWidgetCode)
Widget renderingsopro-personalisation/Controllers/ScriptController.cs (HqInternal)
Widget JSsopro-personalisation/Views/Script/CoreSPA.cshtml
Anonymous lookupsopro-personalisation/Controllers/ScriptController.cs (IpLookup)
Email‑click visitssoproprospecting/.../API/SoPro.API/Controllers/WebsiteWidgetController.cs (PageVisit)
Company detection APIanonymous-ip/WebAPI/Controllers/AnonymousIPController.cs
Detection orchestrationanonymous-ip/Services/MainService.cs, Services/FindingService.cs, Services/AnonymousIPProviderFactory.cs
Visit persistencesopro-sodastream-core/.../Partials/TaskIntentTrackerVisit.cs, Dapper/DapperSopro/DapperSopro/DapperWebsiteWidgetService.cs
Nightly schedulersoproprospecting/.../Services/HangFireJobServiceDB.cs (_AddWebAutomationDaily)
Nightly automationsopro-sodastream-core/.../Partials/TaskWebIntentAutomationDaily.cs
Email findingsopro-sodastream-core/.../Partials/TaskIPEmailFinder.cs
Prospect goals + notificationssoproprospecting/.../Services/Services/ProspectIntentTrackingService.cs (TrackGoals, SendNotification)
Company goals + notificationssopro-sodastream-core/.../DapperSopro/DapperWebsiteWidgetService.cs (ProcessWebGoals)
Goal configuration UIsopro-sodastream-core/.../Portal.Web/Controllers/WebSiteCatcherController.cs

7. Glossary​

TermMeaning
Web IntentDetecting the company behind an anonymous website visitor and evaluating it for engagement.
IP Match and EngageThe full pipeline that turns an anonymous visit into a contactable prospect.
_obidVisitor/prospect identifier carried in email links and cookies; presence marks a known visitor.
Integration tokenThe client's widget key (PropertySettings).
SoproData / SoProMasterDBMaster company & prospect data source used for enrichment and prospect sourcing.
Pelias id (Gid)Geographic identifier used for hierarchical location matching.
Lead statusNot engaged / Engaged / SQL / MQL, derived from prior engagement.
Stage statusNew / Active engagement / Opportunity for reengagement.
Round‑robin email profileSending identities are cycled across queued prospects via SelectNext.
GoalA real‑time trigger (specific page / returning after N days / N visits) that alerts the client.
Engaged vs. qualified company goalEngaged = company already has prospects in contact; qualified = company matches Web Intent targeting.

8. Bot detection​

Bots (automated crawlers, scrapers, headless browsers) are filtered out at two layers to prevent them from polluting visit data and wasting look‑up credits.

8a. Client‑side gate — Views/Script/CoreSPA.cshtml​

The widget JavaScript makes the first decision:

if (!IsBot && _obid != undefined && _obid != null) {
    // TYPE 1 — EMAIL CLICK (known prospect)
} else if (!IsBot) {
    // TYPE 2 — ANONYMOUS IP (Web Intent)
}
// If IsBot === true, neither path executes — the visit is silently dropped.

The IsBot flag is set server‑side by HqInternal, which inspects the User‑Agent and other request characteristics before the widget even renders.

8b. Server‑side bot detection — anonymous-ip/Services/Helpers/BotDetectionHelper.cs​

For anonymous visits that reach the IP‑lookup stage, a second, more precise check runs inside anonymous-ip. IPCheckingService.ProcessSingleCheck() calls BotDetectionHelper.IsBot(ipCheckerIPCheck) against the IP‑details provider response (from IPApiProvider). This check examines two concrete signals:

Rule 1 — Microsoft datacenter + Microsoft email provider

IsDatacenter == true
AND ASNOrganization contains "Microsoft"
AND EmailProvider is "office 365" or "outlook.com"
→ BotDetectionRule = MicrosoftDatacenter_MicrosoftEmailProvider (enum value 1)

Rule 2 — Google datacenter + Google email provider

IsDatacenter == true
AND ASNOrganization contains "Google"
AND EmailProvider is "gmail"
→ BotDetectionRule = GoogleDatacenter_GoogleEmailProvider (enum value 2)

If either rule matches, the IP check result is marked IsBot = true and the search is stopped immediately — no company look‑up is performed, no credits are consumed, and the visit is discarded.

8c. Additional bot check — known‑visitor path (soproprospecting)​

The email‑click path also has its own server‑side bot gate. WebsiteWidgetService.PageVisit() calls AdditionalBotCheck(ipAddress, emailDomain, pageVisitId, clientId) which hits the AnonymousIP API to verify the IP before accepting the visit. If the response returns IsBot == true:

  • The bot detection rule (BotDetectionRuleEnum) is resolved to a friendly name via .ToFriendlyName() (e.g. "Microsoft data center — Microsoft email provider").
  • A row is written to the BotVisits log table with the IP, reason, page URL, User‑Agent, and referrer.
  • The visit is rejected (return false).

8d. Bot detection enums​

Defined identically in both anonymous-ip/Utils/Enums/BotDetectionRuleEnum.cs and soproprospecting/Services/Enums/BotDetectionRuleEnum.cs:

Enum valueNameFriendly label
1MicrosoftDatacenter_MicrosoftEmailProviderMicrosoft data center — Microsoft email provider
2GoogleDatacenter_GoogleEmailProviderGoogle data center — Google email provider

8e. Data model — IPCheck entity​

The bot detection result is persisted on the IPCheck entity (Entities/IPCheck.cs):

ColumnTypeNotes
IsBotbool?true if the IP was identified as a bot
BotDetectionRulebyte?Which rule fired (1 or 2); null if not a bot

These columns also flow through to SingleSearchCallbackResult (Models/SingleIPCheckCallbackResult.cs) so that upstream callers can log or report on bot‑filtered traffic.

9. The 5‑second rule — heartbeat & short‑visit cleanup​

The widget uses a heartbeat system to measure how long a visitor stays on the page. Short, fleeting visits (likely bots, accidental clicks, or zero‑engagement bounces) are automatically purged.

9a. Heartbeat back‑off schedule​

Heartbeats are periodic pings from the browser to the PageVisitHeartBeat endpoint, reporting how many seconds have elapsed since the last ping. The anonymous‑visitor path uses a back‑off schedule to reduce load for long‑staying visitors:

Ping #Interval since last pingCumulative time
15 seconds5s
210 seconds15s
315 seconds30s
430 seconds60s
5+60 seconds60s+

The known‑visitor (email‑click) path pings every ~5 seconds with no back‑off.

Each heartbeat updates PageVisitAnonymous.SessionDurationTotalSeconds (with a cap at 3,600 seconds / 1 hour) and refreshes WebIntentCompanies.LastVisit.

9b. The 5‑second cleanup — CleanPrivacyPolicyAnonymous​

After an anonymous visit is recorded, a delayed cleanup task checks whether the visitor actually engaged. The mechanism is in DapperWebsiteWidgetService.CleanPrivacyPolicyAnonymous(string pageVisitId):

public async Task CleanPrivacyPolicyAnonymous(string id)
{
    // Wait 60 seconds before checking — gives heartbeats time to arrive
    Thread.Sleep(60000);

    var visit = await GetVisit(pageVisitId);
    if (visit != null && visit.SessionDurationTotalSeconds <= 5)
    {
        // Visit lasted 5 seconds or less → purge it
        await DeleteVisit(pageVisitId);
    }
}

Logic:

  1. A background task is scheduled when the visit is first recorded (via the PrivacyPolicyVisitsAnonymous queue).
  2. It waits 60 seconds — enough time for at least one 5‑second heartbeat to arrive and update SessionDurationTotalSeconds.
  3. After the wait, it checks SessionDurationTotalSeconds:
    • ≤ 5 seconds → the page was never open long enough for even one heartbeat to fire. The visit is treated as non‑engagement and deleted from PageVisitAnonymous.
    • > 5 seconds → at least one heartbeat fired, so the visit is kept.

This effectively means: if a visitor doesn't stay on the page for at least 5 seconds, the visit never happened (from the system's perspective). The same principle applies to the known‑visitor path via CleanPrivacyPolicy, which checks Tick <= 1 after a 12‑second wait.

9c. The "5 seconds" parameter in goal alerts​

The "DURATION" parameter in goal notification emails is hard‑coded as "5 seconds" — it describes the heartbeat interval, not a visit threshold:

var parameters = new Dictionary<string, string>()
{
    { "DURATION", "5 seconds" },
    // ...
};

This value appears in the alert email template to give recipients context about how visit duration is measured.

10. Intent Tracker Exclusions — ProcessIntentTrackerExclusions​

When a client creates a domain exclusion in the Sopro portal, the exclusion must cascade into the Web Intent data to prevent excluded companies from ever becoming prospects. This is handled by a dedicated queue consumer.

10a. Trigger — when exclusions are created​

Exclusions are entered through the portal or uploaded in bulk. Three services produce the queue message:

ServiceLocationTrigger
ExclusionService.AddExclusionssoproprospecting/Services/Services/ExclusionService.csPortal exclusion CRUD
ExclusionService.AddExclusionssopro-sodastream-core/.../Portal.Services/Implementations/Exclusions/ExclusionService.csPortal exclusion CRUD (core)
DapperExclusionService.AddExclusionssopro-sodastream-core/.../DapperSopro/DapperSopro/DapperExclusionService.csBulk/Dapper exclusion operations

If the excluded value is a domain (not an email address — checked via !exclusionEntity.Email.Contains('@')), the service enqueues:

await AddToQueueAsync(
    JsonConvert.SerializeObject(new ExclusionQueueModel {
        Domain = exclusionEntity.Email,
        CampaignId = exclusionEntity.CampaignId,
        ClientId = exclusionEntity.ClientId,
        IsCompetitorExclusion = (subType == "Competitors")
    }),
    SoProQueueEnum.IntentTrackerExclusion.ToFriendlyName()
);

Campaign sharing also triggers exclusions: CampaignsSharedService enqueues a model with SharedCampaignId to propagate exclusions between shared campaigns.

10b. The ExclusionQueueModel payload​

public class ExclusionQueueModel
{
    public string Domain { get; set; }          // The domain being excluded (e.g. "competitor.com")
    public int? CampaignId { get; set; }         // Scope: single campaign
    public int? SharedCampaignId { get; set; }   // Scope: propagate to shared campaign
    public int? ClientId { get; set; }           // Scope: all campaigns for a client
    public bool IsGlobal { get; set; }           // Scope: global (across all clients)
    public bool IsCompetitorExclusion { get; set; } // Marks as competitor (not just general)
}

10c. Consumer — TaskIntentTrackerExclusion.cs​

The queue consumer (SoProQueueConsumer/Partials/TaskIntentTrackerExclusion.cs) deserialises the model and delegates to DapperWebsiteWidgetService.ProcessIntentTrackerExclusions(model).

10d. Routing — four processing paths​

ProcessIntentTrackerExclusions routes to one of four private methods based on the model's scope fields:

Path 1 — ProcessSharedExclusions (shared campaign scope)

Copies exclusion flags from the source campaign's WebIntentCompanyCampaignTargeting rows (where IsExclusion = 1) to the shared campaign's rows for the same WebIntentCompanyIds.

Path 2 — ProcessForGlobal (global / cross‑client scope)

Finds all WebIntentCompanies rows matching the excluded domain (where SoproDataCompanyId > 0), then updates all WebIntentCompanyCampaignTargeting rows linked to those companies across all campaigns — setting IsQualified = 0, IsFlagged = 0, IsExclusion = 1.

Path 3 — ProcessForCampaign (single campaign scope, with retry)

  1. Resolves the ClientId from the campaign.
  2. Checks for shared campaigns (via CampaignsShared) that also share exclusions — builds a list of all related campaign IDs.
  3. Updates WebIntentCompanyCampaignTargeting for the excluded domain across all shared campaigns, joining through WebIntentCompanies to match by EmailDomain, ClientId, and SoproDataCompanyId > 0.
  4. Retry logic: on failure, logs the error to ApplicationsErrorLog and retries up to 3 times.

Path 4 — ProcessForClient (client‑wide scope)

Finds all WebIntentCompanies rows matching the excluded domain for the client, then updates all their WebIntentCompanyCampaignTargeting rows to exclusion status.

10e. Effect on the nightly automation​

All four paths produce the same end state on WebIntentCompanyCampaignTargeting:

IsQualified = 0
IsFlagged   = 0
IsExclusion = 1
IsCompetitorExclusion = (true if competitor subtype)

When the nightly TaskWebIntentAutomationDaily runs, its qualification loop checks IsExclusion — excluded companies are skipped, preventing them from ever producing prospects. This applies retroactively: if a company was already qualified before the exclusion was created, the exclusion queue task will mark it and it won't be processed in future nightly runs.

10f. Source files​

ConcernFile
Queue consumer entry pointsopro-sodastream-core/.../Partials/TaskIntentTrackerExclusion.cs
Queue routing (ConsoleTasks)sopro-sodastream-core/.../Partials/ConsoleTasks.cs (line ~350)
Core processing logicsopro-sodastream-core/.../DapperSopro/DapperWebsiteWidgetService.cs (ProcessIntentTrackerExclusions + 4 private methods)
Service interfacesopro-sodastream-core/.../Interface/IDapperWebsiteWidgetService.cs
Queue modelsopro-sodastream-core/.../SharedModels/SoProQueue/Exclusion/ExclusionQueueModel.cs
Portal exclusion triggersoproprospecting/.../Services/Services/ExclusionService.cs
Campaign sharing triggersoproprospecting/.../Services/Services/CampaignsSharedService.cs

11. Clearbit Credits System​

Provider Priority & Credit Consumption​

The IP→company lookup runs through providers in priority order (ClientApp.FindersOrder, e.g. "0,1,2"). Only the last provider (Clearbit, id=2) consumes credits:

Provider IdNameCostSpeedFallback
0CacheFreeInstantNext provider if miss
1InternalDataFreeFastNext provider if miss
2Clearbit1 creditAPI call (~200ms)Returns null (no result)
// FindingService.cs — FindCompany
foreach (var finder in clientApp.FindersOrder.Order.Split(','))
{
    var provider = _AnonymousIPProviderFactory.Create(byte.Parse(finder), price, configs);
    var providerResult = await provider.FindCompany(model, search);
    if (providerResult?.IsUsable == true) return providerResult;
}
return null; // All providers exhausted, no result

Credit Accounting — Account Entity​

The Account table in anonymous-ip tracks client credit balances. Credits were historically stored directly on the Account entity (Credits, CreditsSpent, CreditsLeft columns — now commented out), but the current system relies on Clearbit's own credit tracking via the ClearbitAccountId.

API endpoints for credit management:

EndpointMethodDescription
/api/updateAccountCredits?apiKey=&email=&clientId=&credits=GETUpdates the credit balance for a client's Clearbit account
/api/alignCreditBalanceWithClearbit?apiKey=&email=&clientId=GETSyncs the local credit balance with Clearbit's actual balance
/api/createClearbitAccount?apiKey=&email=&clientId=&clientName=GETCreates a new Clearbit account for a client during widget setup

Credit Exhaustion Flow​

When a client runs out of credits, the following chain executes:

ScriptController Logic — Credit Gate​

In sopro-personalisation/ScriptController.cs, the HqInternal and IpLookup methods check AreClearbitCreditsSpent:

// HqInternal — early return if no OBID and credits are spent
if (string.IsNullOrWhiteSpace(Request.Query["_obid"]) 
    && (!model.IsAnonymousIpActive || model.AreClearbitCreditsSpent))
{
    return Content(""); // Return empty — no tracking
}

// Later in HqInternal — only call AnonymousIP if credits available
if (model.IsAnonymousIpActive && !model.AreClearbitCreditsSpent)
{
    // POST to anonymousIP for company identification
}
// IpLookup — check AfterLimitTracking first
if (model.IsAfterLimitTracking)
{
    await AfterLimitTracking(model);  // Lightweight tracking, no API call
    return Json(new Prospect());       // Return empty prospect
}

// Normal flow — call AnonymousIP API (consumes credit if Clearbit hit)
var url = $"{AnonymousIPApiUrl}/api/getCountryAndCompanyByIP/...";

Email Notification​

The TurnOffAnonymousVisits method sends the ClearbitCreditsSpent (id=156) notification to:

  • CS owner (Client Service) + their team lead
  • Ops owners (all campaigns' ops owners) + their team leads

Parameters: CLIENT_NAME, CS_OWNER_EMAIL, CS_TEAM_LEAD, OPS_OWNERS, OPS_TEAM_LEADS.

12. IsAfterLimitTrackingActive — After-Limit Tracking Mode​

Purpose​

IsAfterLimitTrackingActive is a graceful degradation flag on PropertySettings. When enabled, the widget continues recording visits but skips all company identification — preserving visit data without consuming any credits.

Database​

Migration: 20250227105710_IsAfterLimitTrackingActive — adds IsAfterLimitTrackingActive (bit, default 0) to PropertySettings.

Entity: Entities/PropertySettings.cs:

public bool IsAfterLimitTrackingActive { get; set; }

ViewModel: Models/HqScriptViewModel.cs:

public bool IsAfterLimitTrackingActive { get; set; }

Request model: Models/IPLookupModel.cs:

public bool IsAfterLimitTracking { get; set; }

Client-Side Flow​

In Core.cshtml / CoreSPA.cshtml, the widget checks IsAfterLimitTrackingActive from the view model:

// If after-limit tracking is active and no prospect OBID, skip normal IP lookup
if (params.IsAfterLimitTrackingActive && !hasOBID) {
    params.IsAfterLimitTracking = true;
    postWidgetApi(visitUrl, params, ...); // Still POSTs to /iplookup
}

Server-Side Flow​

ScriptController.IpLookup() checks the flag first:

if (model.IsAfterLimitTracking)
{
    await AfterLimitTracking(model);
    return Json(new Prospect()); // Empty prospect — browser shows nothing
}

AfterLimitTracking() creates a minimal payload and enqueues directly:

private async Task AfterLimitTracking(IPLookupModel model)
{
    var compositeCompany = new IPDetectionResponse
    {
        ClientId = int.Parse(model.ClientId),
        IpAddress = model.IPAddress,
        PageUrl = model.PageUrl,
        IsAfterCreditLimit = true,  // ← marks this as after-limit
        Referrer = model.Referrer,
        DeviceType = model.DeviceType,
        OS = model.OS,
        Browser = model.Browser,
        Device = model.Device
    };
    await _queueService.AddMessage("intenttrackervisit", 
        new MessageModel { Message = JsonConvert.SerializeObject(compositeCompany) });
}

Storage — AfterLimitAnonymousRequest​

The queue consumer stores these visits in a separate table:

CREATE TABLE AfterLimitAnonymousRequest (
    Id INT PRIMARY KEY IDENTITY,
    ClientId INT NOT NULL,
    IpAddress NVARCHAR(50),
    PageVisited NVARCHAR(MAX),
    Created DATETIME2 NOT NULL,
    Referrer NVARCHAR(MAX),
    ReferrerSource NVARCHAR(MAX),
    ReferrerProvider NVARCHAR(MAX),
    DeviceType NVARCHAR(50),
    OS NVARCHAR(50),
    Browser NVARCHAR(50),
    Device NVARCHAR(50)
);

Cleanup​

Hangfire recurring job — RemoveOldUnknownAndAfterLimitVisits() runs every Sunday at 9pm:

RecurringJob.AddOrUpdate(() => RemoveOldUnknownAndAfterLimitVisits(), "0 21 * * 0");

Deletes in batches of 10,000:

  • PageVisitAnonymousUnknown older than 90 days
  • AfterLimitAnonymousRequest older than 90 days

Summary — Before vs After Limit​

AspectNormal ModeAfter-Limit Mode
TriggerIsAnonymousIpActive = true + AreClearbitCreditsSpent = falseIsAfterLimitTrackingActive = true OR model.IsAfterLimitTracking = true
IP→Company lookup✅ Cache → Internal → Clearbit❌ Skipped entirely
Credit consumptionPossible (if Clearbit hit)❌ None
Visit storagePageVisitAnonymous + WebIntentCompaniesAfterLimitAnonymousRequest
Company identified✅ Yes❌ No (only IP + page + device)
Prospect shown✅ Company details in browser❌ Empty prospect returned
Data retentionPermanent90 days (weekly cleanup)
Use caseNormal Web Intent operationCredits exhausted / manual Ops override