Skip to content

About

CPA is a tool for datastream generation and playback. The CityPulse framework uses this component to run some of the demonstrators.

Resources

Stars

1 star

Watchers

11 watching

Forks

Latest commit

 

History

5 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 

Repository files navigation

CityPulse Data Stream Generator (CPA)

CPA is a tool for generating data streams. A data stream is a sequence of measurements from a sensor or other source, over time (for example temperature data, air pressure on a car's tire, level of fuel on an aircraft, etc.).

It is meant to be used as a tool for testing the performance of systems designed to gather and process large amounts of data from different sources (e.g. IoT middleware). Additionally it can be used to complement existing datastreams, such as the ones in the CityPulse Dataset Page.

Current Features Include:

  • Simple to use, command-line user interface.
  • Data Generation using different distributions (Poisson, Exponential, Geometric, Pareto, Gaussian, Uniform/Random and Constant).
  • Ability to specify starting date of data collection, periodicity (amount of time between subsequent measurements), prefix of URI of generated datastream.
  • Uses the MUO ontology and the UCUM vocabulary for representing units of measurement in datasets, as well as the SSN ontology for representing individual observations.
  • Ability to specify stream metadata in the same information model (e.g. description and geographical location),
  • Uses Turtle (.ttl) notation to export the generated files.
  • Data Playback support with built-in validation of .ttl using Apache Jena.
  • Customizable speed of playback (real-time, accelerated, decelerated).
  • JSON transmission over HTTP or UDP, or playback to the console.

Requirements

  • Java Development Kit (JDK) 8 or newer.
  • Apache Maven, available as mvn.
  • Internet access for Maven's first dependency download.

On Ubuntu, install the build tools with:

sudo apt update
sudo apt install maven default-jdk

Check the installation with java -version and mvn -version.

Build and Run

git clone https://github.com/CityPulse/Data-Stream-Generator.git
cd Data-Stream-Generator
mvn package
java -jar target/cpa-1.0-SNAPSHOT-jar-with-dependencies.jar

mvn package runs the tests and creates the executable JAR with its dependencies. Build output is written to target/, which Git ignores.

Tests

mvn test

The tests cover the complete generation → Turtle file → playback path:

  • All seven distributions in integer and double modes, checking 6,000 observations per combination. Replayed values must match the saved values exactly and satisfy range, mean, variance, and cumulative-probability checks.
  • Scripted CLI sessions for constant values and geometric means, plus unit and location metadata in generated RDF.
  • Playback ordering, repeated playback, timezone offsets, millisecond precision, fractional speeds, cancellation, and a 2,000-point dataset.
  • Invalid files, missing or ambiguous properties, impossible dates, unsupported modes, invalid speeds, and invalid distribution parameters.
  • JSON escaping and HTTP/UDP payloads, HTTP failures and timeouts, and connection cleanup.

Distribution tests use fixed seeds and statistical tolerances, including six-standard-error bounds for sample means and cumulative probabilities. These checks detect regressions in the tested parameters; they do not prove the behavior of every possible parameter choice.

Real HTTP and UDP integration tests require local sockets. UDP tests check the production destination port and receive on an ephemeral test port to avoid needing permission to bind port 57. To run the tests that do not need sockets:

mvn '-Dtest=Playback*Test,Generated*Test,GeneratorCliTest,DistributionEdgeCaseTest' test

Test reports are written to target/surefire-reports/. Log4j's No appenders could be found messages are logging configuration warnings; use Maven's test summary to check for failures.

Command Line Client

The supported commands can be accessed by pressing Tab on the command prompt. The tab key can also be used for autocompletion. Currently, the following commands are implemented:

  • generate: Generates a new data stream.
  • playback: Plays back an existing dataset. This command has to be provided with an argument, i.e. the name of the dataset file for playback.
  • help: Provides a help menu.
  • clear: Clears the screen.
  • exit: Exit the CPA client.

The rest of the document describes the data generation and the data playback functions.

Data Generation

Generation produces exactly the requested number of observations. The first observation uses the starting date; subsequent observations use the chosen periodicity. Generated timestamps preserve milliseconds.

Distributions

Choice Distribution Parameters and double-mode behavior
1 Poisson Mean λ > 0; nonnegative integer counts, including in double mode
2 Exponential Rate λ > 0; nonnegative values with mean 1/λ
3 Geometric Enter mean m >= 1 at the CLI; probability is p = 1/m; integer counts start at 1
4 Shifted Pareto (Lomax) Shape α > 0; nonnegative values with CDF 1 - (1 + x)^(-α); mean 1/(α - 1) when α > 1
5 Uniform Lower bound must be less than upper bound; samples are in [lower, upper)
6 Constant Every value equals the entered constant
7 Gaussian Finite mean and nonnegative standard deviation; zero deviation produces the mean

Distribution parameters must be finite. Integer mode truncates continuous samples toward zero; uniform integer mode samples integers directly within the half-open bounds. Poisson and geometric counts must fit in a Java integer; unrepresentable counts report an error. Units and location metadata are stored in the Turtle file; HTTP and UDP payloads contain only the timestamp and value.

Generation Prompts

The user can generate a new dataset by typing generate in the CPA command prompt.

cpa> generate

Subsequently the user can parameterize the data stream by typing in a number of properties. First the name of the data stream.

Stream Description (e.g. "This stream monitors temperature in Manhattan, New York") >Temperature Stream in Rabalder Meeting Room

Then, the number of measurements to be generated:

Number of measurements [positive integer] >10000

Subsequently the starting date of the observations:

Starting Date: To be provided in yyyy-MM-ddTHH:mm:ss format, e.g. 2014-12-12T11:11:11 >2014-10-30T18:57:00

The periodicity between the observations is defined next:

Periodicity: Amount of time between two subsequent measurements [positive integer] >1

Periodicity Unit of Measurement [1=minutes, 2=seconds and 3=milliseconds] >1

The type of the generated values is set next. It is recommended to use double numbers so that values of some distributions are more accurate:

Observation Value Type: [1=integer, 2=double], double recommended >2

The next option sets the data stream name, which is to be used inside the information model. Please use one word and avoid using special characters.

Datastream Name: (please use one word, e.g. "temperatureStream, carTraffic", etc.) >temperatureRabalder

Then the type of the distribution used to calculate the values:

Distribution: [1=Poisson(mean lambda) 2=Exponential(rate lambda) 3=Geometric(mean 1/p) 4=Pareto(a) 5=Uniform(bounded) 6=Constant 7=Gaussian(mean, stdev)] >5

Depending on the choice of value distribution, different parameters are asked:

Upper Bound: Upper bound for value generation in uniform distribution >20 Lower Bound: Lower bound for value generation in uniform distribution >5

The output mode should always be file, console can be used for verification/debugging:

Output Mode: [1=console 2=file] >2 Filename >temperature_stream_rabalder.ttl

As explained in the introduction, the unit of measurement is based on the UCUM vocabulary, so the user is asked here to specify a valid term (note that the citypulse project has added a few more definitions as well). The list command provides convenient access to all units of measurements:

Unit of measurement [paste URI or type 'list' for possible values]>list

List of supported units of measurement:

[0] http://purl.oclc.org/NET/muo/ucum/unit/length/meter

[1] http://purl.oclc.org/NET/muo/ucum/unit/time/second

[2] http://purl.oclc.org/NET/muo/ucum/unit/mass/gram

[...]

[266] http://purl.oclc.org/NET/muo/citypulse/unit/traffic/vehicle-count

[267] http://purl.oclc.org/NET/muo/citypulse/unit/velocity/km-per-hour

[268] http://purl.oclc.org/NET/muo/citypulse/unit/time/seconds

[269] http://purl.oclc.org/NET/muo/citypulse/unit/events/eventData

Pick a number of measurement that better represents your data >28

Next, the prefix of the URL for the entities defined in the generated ontology has to be defined. It is advisable to put the name of the host where the ontology will be deployed.

Datastream Prefix (URL where the stream will be placed, e.g. http://info.ee.surrey.ac.uk/CCSR/CityPulse) >http://era-gsr.ki.sw.ericsson.se/datasets/generated/rabalder

Finally, the data stream is generated. The time spend on data stream generation depends on the number of observations set from the user before. The generated file can be found in the root folder from where the CPA CLI application was run.

Data Playback

Data Playback allows for simulation of data transmission from an existing data stream file (such as the one generated in the Data Generation section above).

At the cpa> prompt, enter playback /path/to/dataset.ttl, then choose a playback speed and output mode:

Mode Output Target
1 JSON/HTTP Full URL of a JSON POST receiver
2 JSON/UDP Receiver hostname or IP address; destination port is 57
3 Console No receiver required; the target prompt is ignored

For an initial check, use speed 1 and console mode (3). Observations are emitted in timestamp order.

Speed must be finite and greater than zero: 0.5 is half speed, 1 is real time, and 4 is four times faster. Playback uses elapsed monotonic time to account for time spent transmitting. A slow receiver can make delivery late; points that are already due are sent without an extra delay.

JSON/HTTP Target

Choose JSON/HTTP mode (1). Set Target URI to the full URL of an HTTP endpoint that accepts JSON POST requests, including its port and path, for example http://localhost:8080/observations. The receiver must be running and return 200 OK for each request. The default http://localhost works only if such a receiver is listening on port 80 at /.

Each observation is sent as a separate request with Content-Type: application/json, for example:

{"timestamp":"2016-10-21 at 12:00:00","value":"23.5"}

Output timestamps use the local system timezone and retain nonzero milliseconds, for example 2016-10-21 at 12:00:00.125. Values are JSON strings, including when the original RDF literal has a numeric datatype. Each HTTP connection has a 10-second connect timeout and read timeout. Non-200 responses and I/O failures stop playback and report an error.

The target is the receiving service's URL. For Kafka, use an HTTP adapter that accepts this payload and publishes it to Kafka; this tool does not send the Kafka broker protocol directly.

Dataset Format

Playback reads Turtle observations by subject, so the order of triples in the file does not matter. Each observation needs a literal sao:value and a sao:time resource containing a literal tl:at timestamp, for example:

@prefix sao: <http://purl.oclc.org/NET/sao/> .
@prefix tl: <http://purl.org/NET/c4dm/timeline.owl#> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .

<http://example.org/temperature/1> a sao:Point ;
    sao:value "23.5" ;
    sao:time [ a tl:Instant ; tl:at "2016-10-21T12:00:00"^^xsd:dateTime ] .

Stream-level time intervals are ignored. Timestamps accept seconds, optional fractional seconds, and optional UTC offsets (Z or +02:00, for example). Dates without an offset use the local system timezone. Playback operates at millisecond precision.

Empty datasets, typed sao:Point records without values, missing or nonliteral timestamps or values, multiple values/times/instants for one observation, impossible dates, and trailing timestamp text report an error before transmission starts.

About

CPA is a tool for datastream generation and playback. The CityPulse framework uses this component to run some of the demonstrators.

Resources

Stars

1 star

Watchers

11 watching

Forks

Releases

Packages

Contributors

Languages