[ZEPPELIN-6733] Add AI assistant chat to notebooks (server-side)

[seung-00]

What is this PR for?

This PR adds the server-side implementation of an AI assistant for notebooks.

Motivation

AI can help with coding and data analysis, but using it with Zeppelin requires users to move code and results between the notebook and a separate chat. A built-in assistant can reduce this effort. It can also make notebook operations and other tasks easier through tools.

Decisions

Model adaptor

• The assistant currently supports OpenAI models.
• It uses the official OpenAI Java SDK to keep up with API changes.

Data structure

Notebook 1 ── N Conversation
Conversation 1 ── N Message
Message (assistant) 1 ── N ToolCall

File path
$ZEPPELIN_HOME/assistant/<noteId>/<conversationId>.json

Permissions

• Anyone who can read a notebook can read its conversations.
• Only the user who created a conversation can mutate it.

See the proposal for more details.

Direction

• The assistant is disabled by default and can be enabled through zeppelin.notebook.assistant.enable. Enabling it by default is planned once the feature is stable.
• The initial tool, list_paragraphs, just returns paragraph metadata. More tools are needed to make the assistant more capable. Contributions are welcome.

What type of PR is it?

Feature

What is the Jira issue?

• https://issues.apache.org/jira/browse/ZEPPELIN-6733

How should this be tested?

• Run the assistant unit tests

./mvnw test -pl zeppelin-server \
  -Dtest='org.apache.zeppelin.service.assistant.*Test'

• Verify conversation management and message history through the REST APIs
• Verify streaming responses and tool calls through WebSocket

Questions:

• Does the license files need to update? Y
• Is there breaking changes for older versions? N
• Does this needs documentation? Y

[voidmatcha]

I found two issues that appear to be causing the CI failures.

First, the transitive dependencies from openai-java conflict with existing dependency versions, causing Maven dependency convergence checks to fail. Second, ASSISTANT_SEND_MESSAGE and ASSISTANT_EVENT, which were added to the Java side, are missing from the TypeScript SDK, so the WebSocket contract check stops the E2E job before Playwright starts.

I prepared a reference implementation covering both issues:

voidmatcha@3f7c15d

With this approach, Maven dependency convergence, the WebSocket contract check, the SDK contract typecheck, and all 28 assistant-related server tests pass locally. Please feel free to use it as a reference.

[jongyoul]

👍

[jongyoul]

It would be better to locate it on AssistantService.

[seung-00]

@jongyoul Agreed, I’ve moved it.

[jongyoul]

LGTM. Thank you for this complicated job. I left comments but all are not blockers to merge this PR. We could handle it with another PR.

[voidmatcha]

Merged into master (de1d7a9).