Skip to content

chore(binding-opcua) fix application/octet-stream encoding/decoding #… - #1449

Closed
erossignon wants to merge 1 commit into
eclipse-thingweb:masterfrom
node-opcua:feat/1400
Closed

erossignon wants to merge 1 commit into
eclipse-thingweb:masterfrom
node-opcua:feat/1400

Conversation

@erossignon

Copy link
Copy Markdown
Contributor

…1400

@codecov

codecov Bot commented Oct 15, 2025 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 63.76812% with 50 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.41%. Comparing base (3325e12) to head (92fcbd8).
⚠️ Report is 173 commits behind head on master.

Files with missing lines Patch % Lines
...ackages/binding-opcua/src/opcua-protocol-client.ts 29.57% 50 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #1449      +/-   ##
==========================================
- Coverage   77.58%   77.41%   -0.17%     
==========================================
  Files          79       84       +5     
  Lines       15331    15989     +658     
  Branches     1445     1519      +74     
==========================================
+ Hits        11894    12378     +484     
- Misses       3414     3585     +171     
- Partials       23       26       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@JKRhb JKRhb linked an issue Oct 16, 2025 that may be closed by this pull request
const variantInJson = opcuaJsonEncodeVariant(dataValue.value, false);
const content = contentSerDes.valueToContent(variantInJson, schemaDataValue, contentType);
return content;
} else if (contentType === "application/octet-stream") {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the general problem with the current code is that it right away decodes the data on the wire. It should first get the data raw and afterwards the ContentSerdes decodes the data.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Take a look at CoAP for example where the readResource call returns the raw data

public async readResource(form: CoapForm): Promise<Content> {
const req = await this.generateRequest(form, "GET");
debug(`CoapClient sending ${req.statusCode} to ${form.href}`);
return new Promise<Content>((resolve, reject) => {
req.on("response", (res: ObserveReadStream) => {
debug(`CoapClient received ${res.code} from ${form.href}`);
debug(`CoapClient received Content-Format: ${res.headers["Content-Format"]}`);
// FIXME does not work with blockwise because of node-coap
const contentType = (res.headers["Content-Format"] as string) ?? form.contentType;
resolve(new Content(contentType, Readable.from(res.payload)));
});
req.on("error", (err: Error) => reject(err));
req.end();
});
}

Afterwards the configured ContentSerdes deals with the actual content and whether the "byte stream" is JSON, CBOR or anything else..

The same should happen with OPC UA....

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the current code is that it right away decodes the data
In fact this code is where the encoding takes palce;( not decoding)

@danielpeintner

Copy link
Copy Markdown
Member

I tried to test the PR with a simple example and I think it does not work as expected. Maybe I am just not understanding it properly.

As mentioned in #1400 OPC-UA seems to work fine until I add contentType.

Note: FYI, not adding any contentType at all would mean JSON which is not the case for OPC UA I believe.

If I add
"contentType": "application/octet-stream"
in a form the codec tries to do work... which has been done already.

Also, it seems that the according code hasn't changed in that regard.

Try to add "contentType": "application/octet-stream" in a simple TD and you will get something like.

throw new Error(`application/octet-stream only supports primitive types (boolean, integer, number, string) - got ${form.type}`);
                          ^

Error: application/octet-stream only supports primitive types (boolean, integer, number, string) - got undefined

@erossignon

Copy link
Copy Markdown
Contributor Author

I tried to test the PR with a simple example and I think it does not work as expected. Maybe I am just not understanding it properly.

As mentioned in #1400 OPC-UA seems to work fine until I add contentType.

Note: FYI, not adding any contentType at all would mean JSON which is not the case for OPC UA I believe.

If I add "contentType": "application/octet-stream" in a form the codec tries to do work... which has been done already.

Also, it seems that the according code hasn't changed in that regard.

Try to add "contentType": "application/octet-stream" in a simple TD and you will get something like.

throw new Error(`application/octet-stream only supports primitive types (boolean, integer, number, string) - got ${form.type}`);
                          ^

Error: application/octet-stream only supports primitive types (boolean, integer, number, string) - got undefined

Could you provide an example of things that reproduce what you're doing. That would be easier to grasp . My understanding should be clearly available in the new unit test added. May be I missed your specific case.

@danielpeintner

Copy link
Copy Markdown
Member

@erossignon not sure what we do with this PR. I think we head into another direction ...

@erossignon

Copy link
Copy Markdown
Contributor Author

Yes, I am working on a proposal first. This one is getting unclear to follow anyway
Stay tuned

@erossignon

Copy link
Copy Markdown
Contributor Author

@danielpeintner: agreed, let's close this one. Here is where it stands, so the next person does not have to re-derive it.

What this PR does fix. On master today, putting "contentType": "application/octet-stream" on an OPC UA form fails with dataType.toLowerCase is not a function. That is core's generic octet-stream codec trying to process OPC UA content it does not understand. This PR stops that from happening, and that part works.

What it does not fix, which is the part you ran into. You reported only supports primitive types ... - got undefined. I have now measured it rather than guessed, and your report was correct.

The binding does switch (form.type) — it looks for type inside the form object. But type belongs on the affordance, outside forms:

"temperature": {
  "forms": [ { "href": "/", "opcua:nodeId": "..." } ],
  "type": "number"          // <-- here, not inside the form
}

So form.type is undefined. The TD author did nothing wrong; the code is looking in the wrong half of the structure.

I tested every combination of where type is declared, reading one Double over a real OPC UA connection:

contentType type declared on result
omitted affordance works, returns JSON
application/octet-stream affordance fails — got undefined
application/octet-stream form fails — No schema type defined
application/octet-stream both works
application/octet-stream neither fails — got undefined

So there is exactly one working configuration, and it requires declaring type twice. Nobody would write that, and it is not documented. Your test was the normal case, and the normal case is broken.

It is also not fixable by simply reading type from the affordance instead: the third row shows that declaring it on the form alone fails earlier, because the outer schema validation wants it too. Both layers want a type, from different places.

Why close rather than rework. Two reasons.

  1. This PR predates the codec split that landed in Opcua binding refactor #1451, so it would need substantial rework against src/codecs/ regardless.
  2. More importantly, the remaining problem is a design question, not a patch: should OPC UA content go through WoT's generic content negotiation at all? You raised the same question with Avoid external contentType processing for OPC UA #1402. That deserves to be settled on the issue before anyone writes code, rather than inside a pull request that is now 173 commits behind.

I am writing that up on #1400 with the full measurements. The codec-collision fix from this PR will come back as part of it — it is correct, it just is not sufficient on its own.

Thanks for testing it back in October and for saying plainly that it did not work. That turned out to be the useful part.

@erossignon erossignon closed this Sep 21, 2026
erossignon added a commit to node-opcua/node-wot that referenced this pull request Sep 25, 2026
…eb#1400

Reads one Double over a real OPC UA connection once per combination of
contentType and schema-type placement, and prints the result table.

Assertions pin the current master behaviour so the test goes red when
eclipse-thingweb#1400 is fixed: application/octet-stream fails in every form with the
same OctetstreamCodec error Daniel reported when he opened the issue.

Run against PR eclipse-thingweb#1449 the matrix differs - only the row declaring type
on BOTH the affordance and the form passes, which is why his report
against that PR was correct.
erossignon added a commit to node-opcua/node-wot that referenced this pull request Oct 23, 2026
…eb#1400

Reads one Double over a real OPC UA connection once per combination of
contentType and schema-type placement, and prints the result table.

Assertions pin the current master behaviour so the test goes red when
eclipse-thingweb#1400 is fixed: application/octet-stream fails in every form with the
same OctetstreamCodec error Daniel reported when he opened the issue.

Run against PR eclipse-thingweb#1449 the matrix differs - only the row declaring type
on BOTH the affordance and the form passes, which is why his report
against that PR was correct.
danielpeintner pushed a commit that referenced this pull request Nov 5, 2026
…) (#1574)

* test(binding-opcua): end-to-end contentType matrix for #1400

Reads one Double over a real OPC UA connection once per combination of
contentType and schema-type placement, and prints the result table.

Assertions pin the current master behaviour so the test goes red when
#1400 is fixed: application/octet-stream fails in every form with the
same OctetstreamCodec error Daniel reported when he opened the issue.

Run against PR #1449 the matrix differs - only the row declaring type
on BOTH the affordance and the form passes, which is why his report
against that PR was correct.

* test(binding-opcua): contentType x data shape matrix

Twelve OPC UA shapes - scalars, arrays, LocalizedText, Variant,
ExtensionObject - crossed with six contentType spellings. Self-contained:
registers its own namespace and variables on the fixture server, so the
same file drops into any checkout.

Also records the JS type value() returns and the raw wire bytes, which is
how the float32 narrowing under application/octet-stream was found.

* feat(binding-opcua): binding-scoped content negotiation (POC)

OPC UA payloads no longer go through the global ContentSerdes registry,
where application/octet-stream belongs to Modbus and silently narrows a
64-bit Double to a big-endian float32.

opcua-content-negotiation.ts is now the single place a form's contentType
becomes a representation decision: bare value, Variant, DataValue, OPC UA
Binary, or raw bytes for a genuine ByteString. Anything else is refused
with a message naming the form, the actual OPC UA dataType, and a usable
alternative, instead of failing inside a codec that knows nothing about
OPC UA.

The default stays application/json, which W3C WoT TD 1.1 5.3.4.2 fixes
normatively and a binding may not redefine.

153 passing, no regressions.

* test(binding-opcua): cover ByteString and true 64-bit Int64

ByteString is the one shape for which application/octet-stream is
legitimate, and it was untested - the only legal use of that media type
had no coverage.

Int64 now carries 2^53+1 as an explicit [high, low] pair with
arrayType Scalar. A plain JS number literal silently rounds to 2^53
before reaching the server, so the previous input proved nothing; and
node-opcua throws without an explicit arrayType, which is what hung the
earlier run inside before().

* feat(binding-opcua): route the write path through content negotiation

Write now resolves contentType the same way read does, so the two agree
on what a flavour means, and adds a write round-trip test per flavour.

The test found two real bugs:

- OpcuaJSONCodec.valueToBytes encoded with "ascii" in three places, so
  any non-ASCII string was corrupted on write - "hello woerld" came back
  with a replacement character. Pre-existing on master, unrelated to this
  work, now utf-8.
- Int64 needs an explicit arrayType: coerceInt64 yields a [high, low]
  pair that node-opcua cannot distinguish from a two-element array.

It also pinned down the limit of what a binding can fix alone.
ConsumedThing.writeProperty calls ContentSerdes.valueToContent before the
binding is reached, so on write core has already chosen a codec by media
type. application/octet-stream writes therefore still land in the
Modbus-oriented codec and cannot be intercepted. That is recorded as a
failing-by-design expectation rather than papered over, and needs the
core change discussed in Dec 2025.

174 passing.

* test(binding-opcua): extend shape matrix to full OPC UA type space

Adds the remaining numeric scalars, the non-numeric built-ins (DateTime,
Guid, NodeId, QualifiedName, StatusCode, XmlElement), enumerations, the
null value, more array shapes and 2-dimensional matrices.

Matrices are the reason for the arrayDimensions field: OPC UA sends a
flat array plus a separate Dimensions member, which no value-shaped
contentType can carry.

* feat(binding-opcua): own the octet-stream codec for opc.tcp forms

A ByteString read as application/octet-stream came back as a garbled
string, and such a write never reached the binding at all: core used the
generic OctetstreamCodec, which packs Modbus registers. With core #1572
the binding registers its own codec for the opc.tcp scheme, so both
directions stay OPC UA aware and every other binding is untouched.

The value is base64, as the JSON flavours already report for a
ByteString, and arrayBuffer() still returns the raw bytes. An array of
ByteStrings is refused: a raw octet stream cannot frame several.

* docs(binding-opcua): add design notes under doc/

Explains what the binding does and why: the two converters a value
crosses, what each contentType produces, the measured behaviour per OPC
UA data type, the essence of OPC UA JSON, and the decisions with their
costs. Diagrams are Mermaid inline, with PlantUML sources beside them.

Written for reviewers and for whoever maintains this next: the choices
that differ from OPC 10101 or from Part 6 Table 80 are stated as such,
with the reason and what it costs.

* refactor(binding-opcua)!: stop exposing node-opcua internal shapes

schemaDataValue described what DataValue.toJSON() happens to produce.
That is a node-opcua internal, not a specified format, so nothing in a
Thing Description should ever be validated against it or receive it.

Only the OPC UA JSON encodings remain: 1.04 (Part 6 Annex H) and 1.05
(Part 6 5.4). The tests that validated the internal shape are removed
rather than rewritten; the payload assertions next to them already cover
what a consumer actually sees.

* feat(binding-opcua): support the OPC UA 1.05 JSON encoding

A form selects the edition with application/opcua+json;version=1.05,
and 1.05 takes ;mode=compact|verbose. The default stays 1.04, so no
existing Thing Description changes meaning.

1.05 renames the Variant fields to UaType/Value, flattens the DataValue
onto them and writes NodeIds as strings. It has no bare-value encoding,
so ;type=Value takes the payload out of the envelope, per the rule in
Part 14 7.2.5.4.

Decoding accepts either edition whatever the form declares: the field
names tell them apart, as Part 6 Annex H prescribes.

* test(binding-opcua): print the contentType tables only on request

The matrices are documentation, but they printed on every run, with full
error messages in unaligned columns: unreadable, and noise in a pipeline.

They are aligned tables with short verdicts now, and print only under
BINDING_OPCUA_TEST_VERBOSE. "npm run test:verbose" sets it through a
require file, which behaves the same on every platform.

* test(binding-opcua): actually exercise the octet-stream refusal

The non-ByteString test matched either the binding's target-type refusal
or the codec's "must be a Buffer" message. Writing 1.5 always hit the
second, so the refusal the binding is meant to own was never reached -
coverage showed the throw unvisited.

Split it: a base64 payload to a Double now reaches the target-type
refusal and asserts it names the type, and a non-bytes value keeps the
codec message. Adds the Uint8Array input branch, and drops describeFormat,
which nothing called.

* docs(binding-opcua): move the diagrams to mermaid

The docs carried mermaid inline and PlantUML sources of the same flows,
and they had already drifted - the write diagram showed ContentSerdes in
one and JsonCodec in the other. GitHub renders neither .puml nor the
images, so reviewers never saw those four.

Converts the two diagrams that had no mermaid equivalent, the
octet-stream flow and the opcua+json read with its two schema errors,
and drops the rest as duplicates. The big aligned note was already in
the page as the "Payloads by example" tables.

* docs(binding-opcua): plan for application/octet-stream as OPC UA Binary

Answers the review request on PR 1574: application/octet-stream on
opc.tcp forms carries OPC UA Binary, and value() returns what
application/json returns.

The plan names the part of each OPC UA message that is the payload
(DataValue for properties, KeyValuePair[] for action arguments) and
lists each choice as a question with a proposal, so the reviewers can
change one without reopening the rest. Structures in writes and action
inputs are given as OPC UA JSON 1.05 and encoded by the binding.

* feat(binding-opcua): application/octet-stream carries OPC UA Binary

As asked on PR 1574: on opc.tcp forms the payload is a UA Binary
DataValue (OPC 10101 6.5.3, OPC 10000-6 Table 80), and value() returns
what application/json returns, by calling the same function.

The codec runs in core without a session, so it encodes values of
built-in types and hands anything else to the binding as JSON; a
DataValue mask never sets bit 0x40, which tells the two apart. The
binding names the session whose definitions decode a companion
structure in the Content type (;uaconn=). Action arguments travel as a
DataValue of KeyValuePair[]; structures in writes and inputs are
refused until plan question A3 is settled.

* docs(binding-opcua): octet-stream is OPC UA Binary (D7, D4 withdrawn)

decisions.md records D7 and withdraws D4, the raw ByteString meaning
the reviewers asked to drop; content-negotiation.md shows the new
payloads and the write and read path through the UA Binary codec.

The plan now matches what was built: action payloads are a DataValue
like every other octet-stream payload, the output also carries
;uapart=arguments, and scenario C (type on the form only) is refused by
core before decoding, as with application/json.

* fix(binding-opcua): accept Part 6 field names Dimensions and Status

node-opcua is aligning its OPC UA JSON on Part 6: 1.05 matrices carry
Dimensions instead of UaDimensions, and the DataValue status is Status
in both editions instead of StatusCode (1.04) / UaStatus (1.05). The
AJV schemas set additionalProperties false, so such payloads were
refused with "Invalid JSON dataValue".

The schemas now accept both spellings. Edition detection is untouched
for Dimensions and Status, since 1.04 uses the same keys and UaType
already marks 1.05; UaStatus is dropped from the detection markers as
no Part 6 edition defines it.

(cherry picked from commit f817739f869ab9005df15ac942afe342a07b4a0a)

* chore(binding-opcua): move to node-opcua 2.187.10

npx node-opcua-versions bump -w -a. The release contains node-opcua
PR 1835, which aligns OPC UA JSON on Part 6 1.04 and 1.05.

The action tests called the methods of MySensor's MethodSet on
MySensor itself. node-opcua 2.187 applies OPC 10000-4 5.12.2: the
method must be a component of the object or of one of its types, so
the server answered BadMethodInvalid. The forms now name MethodSet.

* fix(binding-opcua): accept the 1.05 StatusCode object in a DataValue

A 1.05 StatusCode is {"Code":...}, plus "Symbol" in Verbose (Part 6
5.4.2.12). The 1.05 schema allowed only an integer, so any 1.05
DataValue with a status other than Good was refused with "Invalid JSON
dataValue". node-opcua 2.186 and earlier wrote that field as
StatusCode, never UaStatus, so StatusCode replaces UaStatus.

* feat(binding-opcua): pass the session's namespace table to OPC UA JSON

With node-opcua 2.187.10 the 1.05 encodings name a namespace by its
URI (nsu=) when given the table, so a payload no longer depends on the
server's namespace indexes. Every encoder and decoder of the binding
now gets the table of its session.

The codecs that decode in core have no session: opcua+json and
octet-stream reads now carry ;uaconn= to the session's context, which
holds the table and the structure definitions. On write, opcua+json
passes the consumer's JSON through instead of decoding it with no
table, which turned an nsu= NodeId into a String NodeId of namespace 0.

* docs(binding-opcua): JSON shapes of node-opcua 2.187.10

The plan's appendix lists what the move to 2.187.10 did in the
binding: the namespace table, the schemas, the 1.05 status object and
the stricter Call check. The JSON pages show the new matrix shapes and
a structure's fields beside UaTypeId instead of UaBody.

* docs(binding-opcua): track structure writes (question A3) in issue 1583

The plan's question A3 and the error a structure write gets now point
at eclipse-thingweb/node-wot issue 1583, which carries the proposal
past this PR: structures given as OPC UA JSON 1.05, encoded by the
binding with its session.

* feat(binding-opcua)!: opc.tcp forms take application/opcua+uabinary

The media type of OPC 10000-6 Table 80, planned for WoT OPC UA
Binding 1.1, as asked on PR 1574. The payload stays the UA Binary
DataValue; a type parameter picks what value() returns: Value (the
default), Variant or DataValue, as OPC UA JSON, so the timestamps,
status, type and Union choice the value alone loses stay reachable.

application/json, opcua+json, opcua+octet-stream, octet-stream and
an omitted contentType are refused, each naming its replacement.
The codec needs no URI scheme; OpcuaJSONCodec, OpcuaBinaryCodec and
the JSON schemas are removed.

BREAKING CHANGE: an opc.tcp form needs application/opcua+uabinary.

* fix(examples): point the opcua scripts at the files that exist

bindings:opcua:1 and :2 still referred to opcua-demo1.js and
opcua-demo2.js, renamed to demo-opcua1.js and demo-opcua2.js by the
kebab-case rename; both scripts failed with "Cannot find module".

Adds bindings:opcua:secure for the security demo, which had no script.

* fix(examples): call coffee machine methods on their MethodSet

brewCoffee and fillTank passed the CoffeeMachineA object as the OPC UA
objectId, but MakeCoffee and FillTank are components of its 2:MethodSet,
not of the machine. The server rejected the pairing with
BadMethodInvalid, after resolving the method node itself just fine.

Point opcua:nodeId at 2:MethodSet in both action forms.

* fix(examples): give the opcua demo temperature in °C, not m/s

The temperature property copied the unit of pumpSpeed, and demo-opcua2
printed it. The demo server reports values around 27, i.e. degrees.

* docs(examples): explain each OPC UA example in the README

Describes what each of the four demos shows, how to run it (including
the test server the secure demo needs), and the two TD idioms readers
trip on: browse paths, and the method owner in an action form.
Output samples come from real runs; images are hand-made SVGs.
Written in ASD-STE100 Simplified Technical English.

* docs(examples): OPC UA forms use application/opcua+uabinary

The 15 opc.tcp forms of the OPC UA examples: 14 had no contentType
and one said application/json, which the binding now refuses.

* docs(binding-opcua): decision D8 and the contentType pages

D8 replaces D2, D3, D5, D6 and D7, which stay for the record. The
content negotiation and JSON pages describe the three views; the
octet-stream plan is marked superseded.

* refactor(binding-opcua): each view fixes its JSON edition

The views of application/opcua+uabinary each use one OPC UA JSON
edition, so the edition and mode fields of ContentFormat, and the
1.04 Reversible and 1.05 Verbose encoders behind them, were unreachable.

Value stays 1.04 NonReversible: 1.05 has no bare value and flattens a
matrix. Variant and DataValue are 1.05 Compact. Writes still accept
either edition. D8 and the encoding page say so.

* fix(binding-opcua): refuse what a write cannot convert, and refuse early

Review of PR 1574. A write is converted to the target's DataType or
refused, never written approximately: integers out of range or with a
fraction, texts that are not numbers or dates, nested arrays in the
Value view, and Variant/DataValue JSON that decodes to no value were
written as wrapped, truncated, NaN or Null values. DateTime texts now
write back, and an abstract DataType takes the type of the value.

A refused contentType now fails before Read, Call or CreateMonitoredItems,
and a payload is read as JSON first: a JSON number looked like UA Binary.
The connection context keeps its structure definitions across calls.

* docs(binding-opcua): keep only the current contentType rules

Review of PR 1574. The old contentTypes are refused with the generic
"unsupported contentType" error instead of naming a replacement:
application/opcua+uabinary does not replace JSON, they were never right.

doc/ is removed: it recorded decisions and plans that no longer hold.
What still holds moves to the README contentType section: the three
views, writes, actions, what the Value view loses, structure writes
(issue 1583). Comments pointing at doc/ or at plan questions now say
the rule itself. The removed pages stay readable at the tag
archive/pr-1574-design-notes of node-opcua/node-wot.

* docs(binding-opcua): list type=Value in the contentType table

Review of PR 1574. The code accepts the explicit form, and only the
README left it out.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ContentType seems to confuse OPC UA

2 participants