CPA is a tool for generating data streams. A data stream is a sequence of measurements from a sensor or other source, over time (for example temperature data, air pressure on a car's tire, level of fuel on an aircraft, etc.).
It is meant to be used as a tool for testing the performance of systems designed to gather and process large amounts of data from different sources (e.g. IoT middleware). Additionally it can be used to complement existing datastreams, such as the ones in the CityPulse Dataset Page.
Current Features Include:
- Simple to use, command-line user interface.
- Data Generation using different distributions (Poisson, Exponential, Geometric, Pareto, Gaussian, Uniform/Random and Constant).
- Ability to specify starting date of data collection, periodicity (amount of time between subsequent measurements), prefix of URI of generated datastream.
- Uses the MUO ontology and the UCUM vocabulary for representing units of measurement in datasets, as well as the SSN ontology for representing individual observations.
- Ability to specify stream metadata in the same information model (e.g. description and geographical location),
- Uses Turtle (.ttl) notation to export the generated files.
- Data Playback support with built-in validation of .ttl using Apache Jena.
- Customizable speed of playback (real-time, accelerated, decelerated).
- JSON transmission over HTTP or UDP, or playback to the console.
- Java Development Kit (JDK) 8 or newer.
- Apache Maven, available as
mvn. - Internet access for Maven's first dependency download.
On Ubuntu, install the build tools with:
sudo apt update
sudo apt install maven default-jdkCheck the installation with java -version and mvn -version.
git clone https://github.com/CityPulse/Data-Stream-Generator.git
cd Data-Stream-Generator
mvn package
java -jar target/cpa-1.0-SNAPSHOT-jar-with-dependencies.jarmvn package runs the tests and creates the executable JAR with its
dependencies. Build output is written to target/, which Git ignores.
mvn testThe tests cover the complete generation → Turtle file → playback path:
- All seven distributions in integer and double modes, checking 6,000 observations per combination. Replayed values must match the saved values exactly and satisfy range, mean, variance, and cumulative-probability checks.
- Scripted CLI sessions for constant values and geometric means, plus unit and location metadata in generated RDF.
- Playback ordering, repeated playback, timezone offsets, millisecond precision, fractional speeds, cancellation, and a 2,000-point dataset.
- Invalid files, missing or ambiguous properties, impossible dates, unsupported modes, invalid speeds, and invalid distribution parameters.
- JSON escaping and HTTP/UDP payloads, HTTP failures and timeouts, and connection cleanup.
Distribution tests use fixed seeds and statistical tolerances, including six-standard-error bounds for sample means and cumulative probabilities. These checks detect regressions in the tested parameters; they do not prove the behavior of every possible parameter choice.
Real HTTP and UDP integration tests require local sockets. UDP tests check the production destination port and receive on an ephemeral test port to avoid needing permission to bind port 57. To run the tests that do not need sockets:
mvn '-Dtest=Playback*Test,Generated*Test,GeneratorCliTest,DistributionEdgeCaseTest' testTest reports are written to target/surefire-reports/. Log4j's
No appenders could be found messages are logging configuration warnings;
use Maven's test summary to check for failures.
The supported commands can be accessed by pressing Tab on the command prompt. The tab key can also be used for autocompletion. Currently, the following commands are implemented:
- generate: Generates a new data stream.
- playback: Plays back an existing dataset. This command has to be provided with an argument, i.e. the name of the dataset file for playback.
- help: Provides a help menu.
- clear: Clears the screen.
- exit: Exit the CPA client.
The rest of the document describes the data generation and the data playback functions.
Generation produces exactly the requested number of observations. The first observation uses the starting date; subsequent observations use the chosen periodicity. Generated timestamps preserve milliseconds.
| Choice | Distribution | Parameters and double-mode behavior |
|---|---|---|
1 |
Poisson | Mean λ > 0; nonnegative integer counts, including in double mode |
2 |
Exponential | Rate λ > 0; nonnegative values with mean 1/λ |
3 |
Geometric | Enter mean m >= 1 at the CLI; probability is p = 1/m; integer counts start at 1 |
4 |
Shifted Pareto (Lomax) | Shape α > 0; nonnegative values with CDF 1 - (1 + x)^(-α); mean 1/(α - 1) when α > 1 |
5 |
Uniform | Lower bound must be less than upper bound; samples are in [lower, upper) |
6 |
Constant | Every value equals the entered constant |
7 |
Gaussian | Finite mean and nonnegative standard deviation; zero deviation produces the mean |
Distribution parameters must be finite. Integer mode truncates continuous samples toward zero; uniform integer mode samples integers directly within the half-open bounds. Poisson and geometric counts must fit in a Java integer; unrepresentable counts report an error. Units and location metadata are stored in the Turtle file; HTTP and UDP payloads contain only the timestamp and value.
The user can generate a new dataset by typing generate in the CPA command prompt.
cpa>
generate
Subsequently the user can parameterize the data stream by typing in a number of properties. First the name of the data stream.
Stream Description (e.g. "This stream monitors temperature in Manhattan, New York") >
Temperature Stream in Rabalder Meeting Room
Then, the number of measurements to be generated:
Number of measurements [positive integer] >
10000
Subsequently the starting date of the observations:
Starting Date: To be provided in yyyy-MM-ddTHH:mm:ss format, e.g. 2014-12-12T11:11:11 >
2014-10-30T18:57:00
The periodicity between the observations is defined next:
Periodicity: Amount of time between two subsequent measurements [positive integer] >
1
Periodicity Unit of Measurement [1=minutes, 2=seconds and 3=milliseconds] >
1
The type of the generated values is set next. It is recommended to use double numbers so that values of some distributions are more accurate:
Observation Value Type: [1=integer, 2=double], double recommended >
2
The next option sets the data stream name, which is to be used inside the information model. Please use one word and avoid using special characters.
Datastream Name: (please use one word, e.g. "temperatureStream, carTraffic", etc.) >
temperatureRabalder
Then the type of the distribution used to calculate the values:
Distribution: [1=Poisson(mean lambda) 2=Exponential(rate lambda) 3=Geometric(mean 1/p) 4=Pareto(a) 5=Uniform(bounded) 6=Constant 7=Gaussian(mean, stdev)] >
5
Depending on the choice of value distribution, different parameters are asked:
Upper Bound: Upper bound for value generation in uniform distribution >
20Lower Bound: Lower bound for value generation in uniform distribution >5
The output mode should always be file, console can be used for verification/debugging:
Output Mode: [1=console 2=file] >
2Filename >temperature_stream_rabalder.ttl
As explained in the introduction, the unit of measurement is based on the UCUM vocabulary, so the user is asked here to specify a valid term (note that the citypulse project has added a few more definitions as well). The list command provides convenient access to all units of measurements:
Unit of measurement [paste URI or type 'list' for possible values]>
list
List of supported units of measurement:
[...]
[266] http://purl.oclc.org/NET/muo/citypulse/unit/traffic/vehicle-count
[267] http://purl.oclc.org/NET/muo/citypulse/unit/velocity/km-per-hour
[268] http://purl.oclc.org/NET/muo/citypulse/unit/time/seconds
[269] http://purl.oclc.org/NET/muo/citypulse/unit/events/eventData
Pick a number of measurement that better represents your data >
28
Next, the prefix of the URL for the entities defined in the generated ontology has to be defined. It is advisable to put the name of the host where the ontology will be deployed.
Datastream Prefix (URL where the stream will be placed, e.g. http://info.ee.surrey.ac.uk/CCSR/CityPulse) >
http://era-gsr.ki.sw.ericsson.se/datasets/generated/rabalder
Finally, the data stream is generated. The time spend on data stream generation depends on the number of observations set from the user before. The generated file can be found in the root folder from where the CPA CLI application was run.
Data Playback allows for simulation of data transmission from an existing data stream file (such as the one generated in the Data Generation section above).
At the cpa> prompt, enter playback /path/to/dataset.ttl, then choose a
playback speed and output mode:
| Mode | Output | Target |
|---|---|---|
1 |
JSON/HTTP | Full URL of a JSON POST receiver |
2 |
JSON/UDP | Receiver hostname or IP address; destination port is 57 |
3 |
Console | No receiver required; the target prompt is ignored |
For an initial check, use speed 1 and console mode (3). Observations
are emitted in timestamp order.
Speed must be finite and greater than zero: 0.5 is half speed, 1
is real time, and 4 is four times faster. Playback uses elapsed monotonic
time to account for time spent transmitting. A slow receiver can make
delivery late; points that are already due are sent without an extra delay.
Choose JSON/HTTP mode (1). Set
Target URI to the full URL of an HTTP endpoint that accepts JSON POST
requests, including its port and path, for example
http://localhost:8080/observations. The receiver must be running and return
200 OK for each request. The default http://localhost works only if such
a receiver is listening on port 80 at /.
Each observation is sent as a separate request with
Content-Type: application/json, for example:
{"timestamp":"2016-10-21 at 12:00:00","value":"23.5"}Output timestamps use the local system timezone and retain nonzero
milliseconds, for example 2016-10-21 at 12:00:00.125. Values are JSON
strings, including when the original RDF literal has a numeric datatype.
Each HTTP connection has a 10-second connect timeout and read timeout.
Non-200 responses and I/O failures stop playback and report an error.
The target is the receiving service's URL. For Kafka, use an HTTP adapter that accepts this payload and publishes it to Kafka; this tool does not send the Kafka broker protocol directly.
Playback reads Turtle observations by subject, so the order of triples in
the file does not matter. Each observation needs a literal sao:value and
a sao:time resource containing a literal tl:at timestamp, for example:
@prefix sao: <http://purl.oclc.org/NET/sao/> .
@prefix tl: <http://purl.org/NET/c4dm/timeline.owl#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
<http://example.org/temperature/1> a sao:Point ;
sao:value "23.5" ;
sao:time [ a tl:Instant ; tl:at "2016-10-21T12:00:00"^^xsd:dateTime ] .Stream-level time intervals are ignored. Timestamps accept seconds, optional
fractional seconds, and optional UTC offsets (Z or +02:00, for example).
Dates without an offset use the local system timezone. Playback operates at
millisecond precision.
Empty datasets, typed sao:Point records without values, missing or
nonliteral timestamps or values, multiple values/times/instants for one
observation, impossible dates, and trailing timestamp text report an error
before transmission starts.