Repository navigation
chore(binding-opcua) fix application/octet-stream encoding/decoding #… - #1449
erossignon wants to merge 1 commit into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1449 +/- ##
==========================================
- Coverage 77.58% 77.41% -0.17%
==========================================
Files 79 84 +5
Lines 15331 15989 +658
Branches 1445 1519 +74
==========================================
+ Hits 11894 12378 +484
- Misses 3414 3585 +171
- Partials 23 26 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
| const variantInJson = opcuaJsonEncodeVariant(dataValue.value, false); | ||
| const content = contentSerDes.valueToContent(variantInJson, schemaDataValue, contentType); | ||
| return content; | ||
| } else if (contentType === "application/octet-stream") { |
There was a problem hiding this comment.
I think the general problem with the current code is that it right away decodes the data on the wire. It should first get the data raw and afterwards the ContentSerdes decodes the data.
There was a problem hiding this comment.
Take a look at CoAP for example where the readResource call returns the raw data
node-wot/packages/binding-coap/src/coap-client.ts
Lines 59 to 75 in 8dc6ed8
Afterwards the configured ContentSerdes deals with the actual content and whether the "byte stream" is JSON, CBOR or anything else..
The same should happen with OPC UA....
There was a problem hiding this comment.
the current code is that it right away decodes the data
In fact this code is where the encoding takes palce;( not decoding)
|
I tried to test the PR with a simple example and I think it does not work as expected. Maybe I am just not understanding it properly. As mentioned in #1400 OPC-UA seems to work fine until I add Note: FYI, not adding any contentType at all would mean JSON which is not the case for OPC UA I believe. If I add Also, it seems that the according code hasn't changed in that regard. Try to add |
Could you provide an example of things that reproduce what you're doing. That would be easier to grasp . My understanding should be clearly available in the new unit test added. May be I missed your specific case. |
|
@erossignon not sure what we do with this PR. I think we head into another direction ... |
|
Yes, I am working on a proposal first. This one is getting unclear to follow anyway |
|
@danielpeintner: agreed, let's close this one. Here is where it stands, so the next person does not have to re-derive it. What this PR does fix. On master today, putting What it does not fix, which is the part you ran into. You reported The binding does "temperature": {
"forms": [ { "href": "/", "opcua:nodeId": "..." } ],
"type": "number" // <-- here, not inside the form
}So I tested every combination of where
So there is exactly one working configuration, and it requires declaring It is also not fixable by simply reading Why close rather than rework. Two reasons.
I am writing that up on #1400 with the full measurements. The codec-collision fix from this PR will come back as part of it — it is correct, it just is not sufficient on its own. Thanks for testing it back in October and for saying plainly that it did not work. That turned out to be the useful part. |
…eb#1400 Reads one Double over a real OPC UA connection once per combination of contentType and schema-type placement, and prints the result table. Assertions pin the current master behaviour so the test goes red when eclipse-thingweb#1400 is fixed: application/octet-stream fails in every form with the same OctetstreamCodec error Daniel reported when he opened the issue. Run against PR eclipse-thingweb#1449 the matrix differs - only the row declaring type on BOTH the affordance and the form passes, which is why his report against that PR was correct.
…eb#1400 Reads one Double over a real OPC UA connection once per combination of contentType and schema-type placement, and prints the result table. Assertions pin the current master behaviour so the test goes red when eclipse-thingweb#1400 is fixed: application/octet-stream fails in every form with the same OctetstreamCodec error Daniel reported when he opened the issue. Run against PR eclipse-thingweb#1449 the matrix differs - only the row declaring type on BOTH the affordance and the form passes, which is why his report against that PR was correct.
…) (#1574) * test(binding-opcua): end-to-end contentType matrix for #1400 Reads one Double over a real OPC UA connection once per combination of contentType and schema-type placement, and prints the result table. Assertions pin the current master behaviour so the test goes red when #1400 is fixed: application/octet-stream fails in every form with the same OctetstreamCodec error Daniel reported when he opened the issue. Run against PR #1449 the matrix differs - only the row declaring type on BOTH the affordance and the form passes, which is why his report against that PR was correct. * test(binding-opcua): contentType x data shape matrix Twelve OPC UA shapes - scalars, arrays, LocalizedText, Variant, ExtensionObject - crossed with six contentType spellings. Self-contained: registers its own namespace and variables on the fixture server, so the same file drops into any checkout. Also records the JS type value() returns and the raw wire bytes, which is how the float32 narrowing under application/octet-stream was found. * feat(binding-opcua): binding-scoped content negotiation (POC) OPC UA payloads no longer go through the global ContentSerdes registry, where application/octet-stream belongs to Modbus and silently narrows a 64-bit Double to a big-endian float32. opcua-content-negotiation.ts is now the single place a form's contentType becomes a representation decision: bare value, Variant, DataValue, OPC UA Binary, or raw bytes for a genuine ByteString. Anything else is refused with a message naming the form, the actual OPC UA dataType, and a usable alternative, instead of failing inside a codec that knows nothing about OPC UA. The default stays application/json, which W3C WoT TD 1.1 5.3.4.2 fixes normatively and a binding may not redefine. 153 passing, no regressions. * test(binding-opcua): cover ByteString and true 64-bit Int64 ByteString is the one shape for which application/octet-stream is legitimate, and it was untested - the only legal use of that media type had no coverage. Int64 now carries 2^53+1 as an explicit [high, low] pair with arrayType Scalar. A plain JS number literal silently rounds to 2^53 before reaching the server, so the previous input proved nothing; and node-opcua throws without an explicit arrayType, which is what hung the earlier run inside before(). * feat(binding-opcua): route the write path through content negotiation Write now resolves contentType the same way read does, so the two agree on what a flavour means, and adds a write round-trip test per flavour. The test found two real bugs: - OpcuaJSONCodec.valueToBytes encoded with "ascii" in three places, so any non-ASCII string was corrupted on write - "hello woerld" came back with a replacement character. Pre-existing on master, unrelated to this work, now utf-8. - Int64 needs an explicit arrayType: coerceInt64 yields a [high, low] pair that node-opcua cannot distinguish from a two-element array. It also pinned down the limit of what a binding can fix alone. ConsumedThing.writeProperty calls ContentSerdes.valueToContent before the binding is reached, so on write core has already chosen a codec by media type. application/octet-stream writes therefore still land in the Modbus-oriented codec and cannot be intercepted. That is recorded as a failing-by-design expectation rather than papered over, and needs the core change discussed in Dec 2025. 174 passing. * test(binding-opcua): extend shape matrix to full OPC UA type space Adds the remaining numeric scalars, the non-numeric built-ins (DateTime, Guid, NodeId, QualifiedName, StatusCode, XmlElement), enumerations, the null value, more array shapes and 2-dimensional matrices. Matrices are the reason for the arrayDimensions field: OPC UA sends a flat array plus a separate Dimensions member, which no value-shaped contentType can carry. * feat(binding-opcua): own the octet-stream codec for opc.tcp forms A ByteString read as application/octet-stream came back as a garbled string, and such a write never reached the binding at all: core used the generic OctetstreamCodec, which packs Modbus registers. With core #1572 the binding registers its own codec for the opc.tcp scheme, so both directions stay OPC UA aware and every other binding is untouched. The value is base64, as the JSON flavours already report for a ByteString, and arrayBuffer() still returns the raw bytes. An array of ByteStrings is refused: a raw octet stream cannot frame several. * docs(binding-opcua): add design notes under doc/ Explains what the binding does and why: the two converters a value crosses, what each contentType produces, the measured behaviour per OPC UA data type, the essence of OPC UA JSON, and the decisions with their costs. Diagrams are Mermaid inline, with PlantUML sources beside them. Written for reviewers and for whoever maintains this next: the choices that differ from OPC 10101 or from Part 6 Table 80 are stated as such, with the reason and what it costs. * refactor(binding-opcua)!: stop exposing node-opcua internal shapes schemaDataValue described what DataValue.toJSON() happens to produce. That is a node-opcua internal, not a specified format, so nothing in a Thing Description should ever be validated against it or receive it. Only the OPC UA JSON encodings remain: 1.04 (Part 6 Annex H) and 1.05 (Part 6 5.4). The tests that validated the internal shape are removed rather than rewritten; the payload assertions next to them already cover what a consumer actually sees. * feat(binding-opcua): support the OPC UA 1.05 JSON encoding A form selects the edition with application/opcua+json;version=1.05, and 1.05 takes ;mode=compact|verbose. The default stays 1.04, so no existing Thing Description changes meaning. 1.05 renames the Variant fields to UaType/Value, flattens the DataValue onto them and writes NodeIds as strings. It has no bare-value encoding, so ;type=Value takes the payload out of the envelope, per the rule in Part 14 7.2.5.4. Decoding accepts either edition whatever the form declares: the field names tell them apart, as Part 6 Annex H prescribes. * test(binding-opcua): print the contentType tables only on request The matrices are documentation, but they printed on every run, with full error messages in unaligned columns: unreadable, and noise in a pipeline. They are aligned tables with short verdicts now, and print only under BINDING_OPCUA_TEST_VERBOSE. "npm run test:verbose" sets it through a require file, which behaves the same on every platform. * test(binding-opcua): actually exercise the octet-stream refusal The non-ByteString test matched either the binding's target-type refusal or the codec's "must be a Buffer" message. Writing 1.5 always hit the second, so the refusal the binding is meant to own was never reached - coverage showed the throw unvisited. Split it: a base64 payload to a Double now reaches the target-type refusal and asserts it names the type, and a non-bytes value keeps the codec message. Adds the Uint8Array input branch, and drops describeFormat, which nothing called. * docs(binding-opcua): move the diagrams to mermaid The docs carried mermaid inline and PlantUML sources of the same flows, and they had already drifted - the write diagram showed ContentSerdes in one and JsonCodec in the other. GitHub renders neither .puml nor the images, so reviewers never saw those four. Converts the two diagrams that had no mermaid equivalent, the octet-stream flow and the opcua+json read with its two schema errors, and drops the rest as duplicates. The big aligned note was already in the page as the "Payloads by example" tables. * docs(binding-opcua): plan for application/octet-stream as OPC UA Binary Answers the review request on PR 1574: application/octet-stream on opc.tcp forms carries OPC UA Binary, and value() returns what application/json returns. The plan names the part of each OPC UA message that is the payload (DataValue for properties, KeyValuePair[] for action arguments) and lists each choice as a question with a proposal, so the reviewers can change one without reopening the rest. Structures in writes and action inputs are given as OPC UA JSON 1.05 and encoded by the binding. * feat(binding-opcua): application/octet-stream carries OPC UA Binary As asked on PR 1574: on opc.tcp forms the payload is a UA Binary DataValue (OPC 10101 6.5.3, OPC 10000-6 Table 80), and value() returns what application/json returns, by calling the same function. The codec runs in core without a session, so it encodes values of built-in types and hands anything else to the binding as JSON; a DataValue mask never sets bit 0x40, which tells the two apart. The binding names the session whose definitions decode a companion structure in the Content type (;uaconn=). Action arguments travel as a DataValue of KeyValuePair[]; structures in writes and inputs are refused until plan question A3 is settled. * docs(binding-opcua): octet-stream is OPC UA Binary (D7, D4 withdrawn) decisions.md records D7 and withdraws D4, the raw ByteString meaning the reviewers asked to drop; content-negotiation.md shows the new payloads and the write and read path through the UA Binary codec. The plan now matches what was built: action payloads are a DataValue like every other octet-stream payload, the output also carries ;uapart=arguments, and scenario C (type on the form only) is refused by core before decoding, as with application/json. * fix(binding-opcua): accept Part 6 field names Dimensions and Status node-opcua is aligning its OPC UA JSON on Part 6: 1.05 matrices carry Dimensions instead of UaDimensions, and the DataValue status is Status in both editions instead of StatusCode (1.04) / UaStatus (1.05). The AJV schemas set additionalProperties false, so such payloads were refused with "Invalid JSON dataValue". The schemas now accept both spellings. Edition detection is untouched for Dimensions and Status, since 1.04 uses the same keys and UaType already marks 1.05; UaStatus is dropped from the detection markers as no Part 6 edition defines it. (cherry picked from commit f817739f869ab9005df15ac942afe342a07b4a0a) * chore(binding-opcua): move to node-opcua 2.187.10 npx node-opcua-versions bump -w -a. The release contains node-opcua PR 1835, which aligns OPC UA JSON on Part 6 1.04 and 1.05. The action tests called the methods of MySensor's MethodSet on MySensor itself. node-opcua 2.187 applies OPC 10000-4 5.12.2: the method must be a component of the object or of one of its types, so the server answered BadMethodInvalid. The forms now name MethodSet. * fix(binding-opcua): accept the 1.05 StatusCode object in a DataValue A 1.05 StatusCode is {"Code":...}, plus "Symbol" in Verbose (Part 6 5.4.2.12). The 1.05 schema allowed only an integer, so any 1.05 DataValue with a status other than Good was refused with "Invalid JSON dataValue". node-opcua 2.186 and earlier wrote that field as StatusCode, never UaStatus, so StatusCode replaces UaStatus. * feat(binding-opcua): pass the session's namespace table to OPC UA JSON With node-opcua 2.187.10 the 1.05 encodings name a namespace by its URI (nsu=) when given the table, so a payload no longer depends on the server's namespace indexes. Every encoder and decoder of the binding now gets the table of its session. The codecs that decode in core have no session: opcua+json and octet-stream reads now carry ;uaconn= to the session's context, which holds the table and the structure definitions. On write, opcua+json passes the consumer's JSON through instead of decoding it with no table, which turned an nsu= NodeId into a String NodeId of namespace 0. * docs(binding-opcua): JSON shapes of node-opcua 2.187.10 The plan's appendix lists what the move to 2.187.10 did in the binding: the namespace table, the schemas, the 1.05 status object and the stricter Call check. The JSON pages show the new matrix shapes and a structure's fields beside UaTypeId instead of UaBody. * docs(binding-opcua): track structure writes (question A3) in issue 1583 The plan's question A3 and the error a structure write gets now point at eclipse-thingweb/node-wot issue 1583, which carries the proposal past this PR: structures given as OPC UA JSON 1.05, encoded by the binding with its session. * feat(binding-opcua)!: opc.tcp forms take application/opcua+uabinary The media type of OPC 10000-6 Table 80, planned for WoT OPC UA Binding 1.1, as asked on PR 1574. The payload stays the UA Binary DataValue; a type parameter picks what value() returns: Value (the default), Variant or DataValue, as OPC UA JSON, so the timestamps, status, type and Union choice the value alone loses stay reachable. application/json, opcua+json, opcua+octet-stream, octet-stream and an omitted contentType are refused, each naming its replacement. The codec needs no URI scheme; OpcuaJSONCodec, OpcuaBinaryCodec and the JSON schemas are removed. BREAKING CHANGE: an opc.tcp form needs application/opcua+uabinary. * fix(examples): point the opcua scripts at the files that exist bindings:opcua:1 and :2 still referred to opcua-demo1.js and opcua-demo2.js, renamed to demo-opcua1.js and demo-opcua2.js by the kebab-case rename; both scripts failed with "Cannot find module". Adds bindings:opcua:secure for the security demo, which had no script. * fix(examples): call coffee machine methods on their MethodSet brewCoffee and fillTank passed the CoffeeMachineA object as the OPC UA objectId, but MakeCoffee and FillTank are components of its 2:MethodSet, not of the machine. The server rejected the pairing with BadMethodInvalid, after resolving the method node itself just fine. Point opcua:nodeId at 2:MethodSet in both action forms. * fix(examples): give the opcua demo temperature in °C, not m/s The temperature property copied the unit of pumpSpeed, and demo-opcua2 printed it. The demo server reports values around 27, i.e. degrees. * docs(examples): explain each OPC UA example in the README Describes what each of the four demos shows, how to run it (including the test server the secure demo needs), and the two TD idioms readers trip on: browse paths, and the method owner in an action form. Output samples come from real runs; images are hand-made SVGs. Written in ASD-STE100 Simplified Technical English. * docs(examples): OPC UA forms use application/opcua+uabinary The 15 opc.tcp forms of the OPC UA examples: 14 had no contentType and one said application/json, which the binding now refuses. * docs(binding-opcua): decision D8 and the contentType pages D8 replaces D2, D3, D5, D6 and D7, which stay for the record. The content negotiation and JSON pages describe the three views; the octet-stream plan is marked superseded. * refactor(binding-opcua): each view fixes its JSON edition The views of application/opcua+uabinary each use one OPC UA JSON edition, so the edition and mode fields of ContentFormat, and the 1.04 Reversible and 1.05 Verbose encoders behind them, were unreachable. Value stays 1.04 NonReversible: 1.05 has no bare value and flattens a matrix. Variant and DataValue are 1.05 Compact. Writes still accept either edition. D8 and the encoding page say so. * fix(binding-opcua): refuse what a write cannot convert, and refuse early Review of PR 1574. A write is converted to the target's DataType or refused, never written approximately: integers out of range or with a fraction, texts that are not numbers or dates, nested arrays in the Value view, and Variant/DataValue JSON that decodes to no value were written as wrapped, truncated, NaN or Null values. DateTime texts now write back, and an abstract DataType takes the type of the value. A refused contentType now fails before Read, Call or CreateMonitoredItems, and a payload is read as JSON first: a JSON number looked like UA Binary. The connection context keeps its structure definitions across calls. * docs(binding-opcua): keep only the current contentType rules Review of PR 1574. The old contentTypes are refused with the generic "unsupported contentType" error instead of naming a replacement: application/opcua+uabinary does not replace JSON, they were never right. doc/ is removed: it recorded decisions and plans that no longer hold. What still holds moves to the README contentType section: the three views, writes, actions, what the Value view loses, structure writes (issue 1583). Comments pointing at doc/ or at plan questions now say the rule itself. The removed pages stay readable at the tag archive/pr-1574-design-notes of node-opcua/node-wot. * docs(binding-opcua): list type=Value in the contentType table Review of PR 1574. The code accepts the explicit form, and only the README left it out.
…1400