LimaCharlie Universal Sensor Protocol Adapter

USP Adapters to allow ingest from various sources into LimaCharlie.

Usage

The adapter can be used to access many different sources and many different event types. The main mechanisms specifying the source and type of events are:

1. Adapter Type: this indicates the technical source of the events, like syslog or S3 buckets.
2. Platform: the platform indicates the type of events that are acquired from that source, like text or carbon_black.

Depending on the Adapter Type specified, configurations that can be specified will change. Running the adapter with no command line arguments will list all available Adapter Types and their configurations.

Configurations can be provided to the adapter in one of three ways:

1. By specifying a configuration file.
2. By specifying the configurations via the command line in the format config-name=config-value.
3. By specifying the configurations via the environment variables in the format config-name=config-value.

All Adapter Types support the same client_options, plus type-specific configurations. You should always specify the following configurations:

• client_options.identity.oid: the LimaCharlie Organization ID (OID) this adapter is used with.
• client_options.identity.installation_key: the LimaCharlie Installation Key this adapter should use to identify with LimaCharlie.
• client_options.platform: the type of data ingested through this adapter, like text, json, gcp or carbon_black.
• client_options.sensor_seed_key: an arbitrary name for this adapter which Sensor IDs (SID) are generated from, see below.

Other common configurations that are not required include:

• client_options.mapping.parsing_re: regular expression with named capture groups. The name of each group will be used as the key in the converted JSON parsing.
• client_options.mapping.sensor_key_path: indicates which component of the events represent unique sensor identifiers.
• client_options.mapping.sensor_hostname_path: indicates which component of the event represents the hostname of the resulting Sensor in LimaCharlie.
• client_options.mapping.event_type_path: indicates which component of the event represents the Event Type of the resulting event in LimaCharlie.
• client_options.mapping.event_time_path: indicates which component of the event represents the Event Time of the resulting event in LimaCharlie.
• client_options.mapping.rename_only: if true, indicates the field mappings defined here should only be renamed fields from the original event (and not completely replacing the original event).
• client_options.mapping.mappings: a list of field remapping to be performed:
  • src_field: the source component to remap.
  • dst_field: what the SourceField should be mapped to in the final event.

All other configurations in client_options should only be set for advanced ingestion schemes.

Graceful shutdown

Adapter Close stops its producer and waits up to 60 seconds for the USP proxy to acknowledge accepted events. A producer blocked by a full buffer is unblocked after 10 seconds. Mailbox clients drain concurrently. Pub/Sub also gets up to 10 seconds to flush source acknowledgements before its RPC transport closes.

Current proxies acknowledge on receipt. An already-written idle tail can wait for their 30-second periodic ACK, so the 60-second drain timeout must remain comfortably above that interval. Proxies that acknowledge downstream acceptance provide stronger confirmation through the same protocol.

If confirmation times out, Close returns an error and OnWarning reports the unconfirmed count and sequence range without payloads. Sources already acknowledged or checkpointed cannot be replayed generically; operators should treat this as uncertain delivery. Hosts using go-uspclient directly can recover the unacknowledged messages returned by its Close method.

Logging

The adapter writes its debug (DBG) and warning (WRN) lines to stdout and its errors (ERR) to stderr. So that a failure retried in a loop (an unreachable API, a TLS error) cannot flood the host's logs, the output is deduplicated:

• The first occurrence of a message is always printed as is.
• Repeats of it within one minute are counted, then reported in a single last message repeated N more time(s) between <first> and <last>: <message> line carrying the most recent text. A message still repeating gets one such line per minute. A message that went quiet for one to two minutes is printed in full the next time it happens.
• Ephemeral ports (32768 and up) following an IP address are ignored when comparing messages, so the same error from the same host on a new connection counts as a repeat. A different address or service port is a different message.
• Each adapter has its own limits, and each level is rate limited: up to 3000 lines at once, refilled at 3000 per minute. Lines over the limit are counted and reported within seconds in a log rate limit ... reached: dropped N ... line(s) line that includes the last dropped message. A flood of debug lines never delays warnings or errors, and one adapter never uses up another's budget.
• Pending counts are reported when the adapter exits.

Set the environment variable USP_LOG_NO_DEDUP=1 to print every line unchanged.

Multi-Adapter

If running using a configuration file, it is possible to run multiple instances of the same adapter type within a single instance of the adapter process. To do this, simply make the config file multiple "documents" (in JSON or YAML format) within the same file, like:

file:
   client_options.identity.oid: XXXXX
   file_path: ./file_1
   ...

---

file:
   client_options.identity.oid: XXXXX
   file_path: ./file_2
   ...

---

...

Sensor IDs

USP Clients generate LimaCharlie Sensors at runtime. The ID of those sensors (SID) is generated based on the Organization ID (OID) and the Sensor Seed Key.

This implies that if want to re-key an IID (perhaps it was leaked), you may replace the IID with a new valid one. As long as you use the same OID and Sensor Seed Key, the generated SIDs will be stable despite the IID change.

Custom Formatting

Data sent via USP can be formatted in many different ways. Data is processed in a specific order as a pipeline:

1. Regular Expression with named capture groups parsing a string into a JSON object.
2. Built-in (in the cloud) LimaCharlie parsers that apply to specific Platform values (like carbon_black).
3. The various "extractors" defined, like EventTypePath, EventTimePath, SensorHostnamePath and SensorKeyPath.
4. Custom Mappings directives provided by the client.

Building a New Adapter Type

Is there an API or data source we don't officially support yet that you'd like to see? You can build one!

1. Modify containers/general/tool.go with the new adapter type you wish to add.

2. Add your adapter files (ex: itglue/client.go)

3. Format

   go fmt mynewsensor/client.go
   

4. Build

   go build ./containers/general
   

5. Test

   param1 and param2 represent any additional variables you need in your adapter (such as API keys, etc).

   chmod +x mynewsensor
   ./general mynewsensor client_options.identity.installation_key=$INSTALLATION_KEY client_options.identity.oid=$OID client_options.platform=json client_options.sensor_seed_key=mynewsensor param1=$param1 param2=$param2 
   

6. You should see a new sensor in your LimaCharlie org if successful

7. Submit a PR

Examples

Syslog

Start the container listening for Syslog. This example uses the syslog from a Debian box.

docker run --rm -it -p 4444:4444 usp-adapters syslog port=4444 client_options.identity.installation_key=e9a3bcdf-efa2-47ae-b6df-579a02f3a54d client_options.identity.oid=8cbe27f4-bfa1-4afb-ba19-138cd51389cd client_options.platform=text "client_options.mapping.parsing_re=(?P<date>... \d\d \d\d:\d\d:\d\d) (?P<host>.+) (?P<exe>.+?)\[(?P<pid>\d+)\]: (?P<msg>.*)" client_options.sensor_seed_key=testclient1 "client_options.mapping.mapping[0].src_field=host" "client_options.mapping.mapping[0].dst_field=syslog_hostname"

Pipe the syslog from a Debian box to the container.

journalctl -f -q | netcat 127.0.0.1 4444

S3

./general s3 client_options.identity.installation_key=e9a3bcdf-efa2-47ae-b6df-579a02f3a54d client_options.identity.oid=8cbe27f4-bfa1-4afb-ba19-138cd51389cd client_options.platform=carbon_black client_options.sensor_seed_key=tests3 bucket_name=lc-cb-test access_key=YYYYYYYYYY secret_key=XXXXXXXX  "prefix=events/org_key=NKZFDWEM/"

Stdin

./adapter stdin client_options.identity.installation_key=e9a3bcdf-efa2-47ae-b6df-579a02f3a54d client_options.identity.oid=8cbe27f4-bfa1-4afb-ba19-138cd51389cd client_options.platform=text "client_options.mapping.parsing_re=(?P<date>... \d\d \d\d:\d\d:\d\d) (?P<host>.+) (?P<exe>.+?)\[(?P<pid>\d+)\]: (?P<msg>.*)" client_options.sensor_seed_key=testclient3 client_options.mapping.event_type_path=exe

ServiceNow

Pulls audit and system logs from the ServiceNow REST Table API. By default it collects the sys_audit table (field-level change history); other tables like syslog_transaction (transaction log) and sysevent (login events) can be added purely through configuration. See servicenow/README.md.

./general servicenow client_options.identity.installation_key=e9a3bcdf-efa2-47ae-b6df-579a02f3a54d client_options.identity.oid=8cbe27f4-bfa1-4afb-ba19-138cd51389cd client_options.platform=json client_options.sensor_seed_key=servicenow instance=example username=$SERVICENOW_USERNAME password=$SERVICENOW_PASSWORD

ThreatLocker

Pulls events from the ThreatLocker Portal API. By default it collects pending Application Control approval requests; additional ThreatLocker event types can be added purely through configuration. See threatlocker/README.md.

./general threatlocker client_options.identity.installation_key=e9a3bcdf-efa2-47ae-b6df-579a02f3a54d client_options.identity.oid=8cbe27f4-bfa1-4afb-ba19-138cd51389cd client_options.platform=json client_options.sensor_seed_key=threatlocker api_key=$THREATLOCKER_API_TOKEN instance=g

Palo Alto SaaS Security (Prisma SaaS)

Pulls activity monitoring, incident, remediation, policy violation and admin audit events from the Palo Alto Networks SaaS Security (formerly Prisma SaaS) Log Events API, using an OAuth2 API client created in Strata Cloud Manager. See prisma_saas/README.md.

./general prisma_saas client_options.identity.installation_key=e9a3bcdf-efa2-47ae-b6df-579a02f3a54d client_options.identity.oid=8cbe27f4-bfa1-4afb-ba19-138cd51389cd client_options.platform=prisma_saas client_options.sensor_seed_key=prisma_saas client_id=$PRISMA_SAAS_CLIENT_ID client_secret=$PRISMA_SAAS_CLIENT_SECRET region=us

Gmail

Collects incoming email as telemetry from one or many Gmail mailboxes via the Gmail REST API. It polls users.messages.list on a rolling window (default query in:inbox), fetches each newly-seen message, and ships it as a gmail_message event. It also has opt-in Business-Email-Compromise capabilities (filters, forwarding, send-as, delegates, IMAP/POP, vacation, history). Supports a single-mailbox OAuth refresh token, or Google Workspace service-account domain-wide delegation for one mailbox, an explicit list of mailboxes, or auto-discovery of the whole domain — each mailbox shipped to its own sensor. See gmail/README.md.

./general gmail client_options.identity.installation_key=e9a3bcdf-efa2-47ae-b6df-579a02f3a54d client_options.identity.oid=8cbe27f4-bfa1-4afb-ba19-138cd51389cd client_options.platform=gmail client_options.sensor_seed_key=gmail client_id=$GMAIL_CLIENT_ID client_secret=$GMAIL_CLIENT_SECRET refresh_token=$GMAIL_REFRESH_TOKEN

Cato

Pulls events from the Cato Networks API for a given account. It polls the event feed, using a marker to page through and resume from the last position across restarts.

./general cato client_options.identity.installation_key=e9a3bcdf-efa2-47ae-b6df-579a02f3a54d client_options.identity.oid=8cbe27f4-bfa1-4afb-ba19-138cd51389cd client_options.platform=cato client_options.sensor_seed_key=cato apikey=$CATO_API_KEY accountid=$CATO_ACCOUNT_ID