Real-Time Traffic Monitoring: Implementing a WebSocket Subscription API in ProxyMapService
Web scraping applications frequently require real-time tracking of all currently downloading URLs.
However, achieving this in practice is far from trivial. For instance, if the target application is a web browser, capturing network activity typically requires globally overriding the built-in fetch and XHR (XMLHttpRequest) methods. The situation becomes significantly more complex if the web page contains iframe elements: you must dynamically inject fetch and XHR overrides into them as well, adding a heavy layer of complexity and fragility to the system. Custom injection scripts can easily conflict with Content Security Policies (CSP) or simply miss requests originating from isolated contexts.
Instead of breaking frontend logic inside the browser and relying on complex script injections, a much more reliable approach is to leverage the proxy layer. To achieve this, an optional WebSocket Session API was implemented in ProxyMapService. It allows external applications to connect directly to the proxy, supply a URL filter, and instantly receive push notifications for all requested resources.
Configuration and Connection Logistics
The entire mechanism is managed by the SessionAPI section in the appsettings.json configuration file. Here is how it looks by default:
"SessionAPI": {
"Enabled": false,
"WebSocketsEnabled": false,
"Domain": ""
}
As you can see, by default, the entire SessionAPI infrastructure, including WebSockets, is turned off, and the Domain property is left empty.
Option 1. Working via Local Port (Empty Domain)
When Domain is empty, the control endpoints are accessed like a regular web server on one of the ports that ProxyMapService listens to for incoming connections. For example, if the proxy is running on port 5000:
Option 2. Working via a Dedicated Pseudo-Domain (Recommended)
To fully isolate control traffic from proxied traffic, it is recommended to enable the API and define a special internal domain:
"SessionAPI": {
"Enabled": true,
"WebSocketsEnabled": true,
"Domain": "proxymapper"
}
In this mode, proxymapper acts as a dedicated system domain. When ProxyMapService detects that a client is trying to access the proxymapper host, it intercepts the request locally and handles it as a call to the SessionAPI instead of routing it out to the external network.
Crucial Nuance for Command-Line Utilities (e.g., cURL):
If you are using curl to route traffic through the proxy, the socks4 protocol will not allow you to call the internal API at all. Furthermore, the socks5:// prefix must be replaced with socks5h:// (e.g., socks5h://127.0.0.1:5000). This ensures that curl does not attempt to resolve the proxymapper domain locally via your OS DNS, but instead delegates the name resolution directly to the proxy server itself.
With this configuration, the WebSocket Session API endpoint will be accessible via: ws://proxymapper/session/ws
How It Works in Practice
Using the subscription mechanism is straightforward for a developer:
- Establishing a Connection: The client opens a persistent WebSocket connection to ws://proxymapper/session/ws.
- Sending a Filter: Immediately after a successful connection, the client sends a text JSON message containing a subscribe action and a regular expression (Regex) matching the target URLs:
{
"action": "subscribe",
"pattern": "api\\.example\\.com/v1/.*"
}
- Subscription Confirmation and session_id: The server replies with a confirmation message acknowledging the registered filter, which includes vital execution context:
{
"status": "subscribed",
"session_id": ":5000"
}
What does session_id mean and how does traffic isolation work?
- session_id represents a unique session identifier. If your environment utilizes authentication via a Sticky Proxy, this string identifier is explicitly bound to the user upon authentication.
- If a Sticky Proxy is not used, the session_id returns a colon followed by the incoming port number (e.g., :5000). This means that notifications are strictly isolated to the specific port through which the subscription was created.
- Lifecycle Rule: The user is guaranteed to receive notifications for URLs downloaded strictly within the session that was active at the moment of subscription. If the proxy session changes, notifications for the old subscription will cease. Similarly, under port-based isolation (without sticky routing), you will never see traffic passing through another proxy port inside the WebSocket channel for port :5000.
- Receiving Notifications: When a downloaded URL matches your pattern, the proxy server dispatches a flat, structured event optimized for fast reading:
{
"event_type": "url_matched",
"timestamp": "2026-09-02T10:20:45.0328755Z",
"session_id": ":5000",
"method": "GET",
"url": "https://api.example.com/v1/users/profile"
}
Testing Environment: Proxy WebSocket Control Panel
The test web panel is hosted in a dedicated project subdirectory: tests\test-websocket.
Inside the folder lies a self-contained HTML file, index.html, which implements the diagnostic interface:
<div class="card">
<h2>Proxy WebSocket Control Panel</h2>
<div class="form-group">
<input type="text" id="wsUrl" value="ws://proxymapper/session/ws">
<button onclick="toggleConnection()">Connect</button>
</div>
<div class="form-group">
<input type="text" id="filterPattern" value="(google\.com|yandex\.ru)/.*">
<button onclick="sendSubscription()">Set Filter</button>
</div>
</div>
How to Run the Test and Verify Monitoring:
- Launch the configured ProxyMapService with WebSockets enabled and the domain set to proxymapper.
- Open the tests\test-websocket\index.html file in any modern browser whose traffic is being routed through your proxy.
- Click the Connect button, set your regex filter pattern, and watch live notifications populate the dark console UI with full request details instantly.
Conclusion
Shifting URL tracing logic out of fragile browser scripts and into the ProxyMapService layer provides 100% visibility into network requests. It no longer matters where a request originates—be it the main window, a heavy background iframe, or a hidden service worker—the proxy server reliably intercepts, filters based on the active session (or port), and safely delivers the event to the client via a WebSocket subscription without modifying a single line of original page code.