1 | remctl Streaming Protocol Draft |
---|
2 | |
---|
3 | Introduction |
---|
4 | |
---|
5 | This is a draft of what would become version three of the remctl |
---|
6 | protocol. It adds optional support for bidirectional streaming, |
---|
7 | allowing the server and client to exchange arbitrary unsequenced data |
---|
8 | while a command is running with coordinated termination of the |
---|
9 | command. |
---|
10 | |
---|
11 | This draft should not be used for implementation yet. The details of |
---|
12 | the protocol may change substantially before it is added to remctl. |
---|
13 | |
---|
14 | Client library API changes are not discussed in this draft, only |
---|
15 | protocol issues. |
---|
16 | |
---|
17 | Streaming Overview |
---|
18 | |
---|
19 | A streaming command is a client-initiated operation. The client |
---|
20 | requests streaming when it sends the command. The server either |
---|
21 | accepts that request or rejects it. If a particular command requires |
---|
22 | streaming, the server may reject a non-streaming version of that |
---|
23 | command. |
---|
24 | |
---|
25 | Once the server accepts the streaming command, it runs the command, |
---|
26 | passing as arguments the arguments provided in the initial command |
---|
27 | token as before. From that point on, all command output is |
---|
28 | immediately sent back to the client, and the client may also send |
---|
29 | data at any time which is passed as input to the command. This |
---|
30 | continues without ordering and sequencing until one side or the other |
---|
31 | wants to start shutting things down. |
---|
32 | |
---|
33 | The server can indicate the end of an output stream with a token |
---|
34 | dedicated to that purpose. This communicates an EOF on that stream to |
---|
35 | the client without terminating the command processing. The client can |
---|
36 | similarly tell the server that it is done providing input tokens by |
---|
37 | sending an end of stream token to the server, after which it is not |
---|
38 | permitted to send any more input tokens until the server indicates the |
---|
39 | command is finished. |
---|
40 | |
---|
41 | When the command is over, the server sends a token indicating that |
---|
42 | fact. This token will also include the exit status of the command. |
---|
43 | This token implies end of stream on all output streams. The server is |
---|
44 | solely responsible for determining when the command is over. The most |
---|
45 | the client can do is indicate the end of the input stream, but it must |
---|
46 | still wait for the server to indicate the end of the command. |
---|
47 | |
---|
48 | If the server indicates the end of the command before the client |
---|
49 | closes the input stream, the server will continue to consume and |
---|
50 | discard input tokens from the client until the client sends an end of |
---|
51 | stream token. |
---|
52 | |
---|
53 | There are therefore two ways for a streaming command to end: |
---|
54 | |
---|
55 | * Client sends end of stream token. Server sends end of streaming |
---|
56 | command token, possibly preceeded by an arbitrary number of |
---|
57 | streaming output or end of stream tokens. |
---|
58 | |
---|
59 | * Server sends end of streaming command token. Client sends end of |
---|
60 | stream token, possibly preceeded by an arbitrary number of streaming |
---|
61 | input tokens. |
---|
62 | |
---|
63 | The streaming command is not over until the server has sent an end of |
---|
64 | command token and the client has sent an end of input stream token. |
---|
65 | Only after both of those tokens have been sent do both sides return to |
---|
66 | normal processing of remctl commands. |
---|
67 | |
---|
68 | New Tokens |
---|
69 | |
---|
70 | MESSAGE_COMMAND_STREAM |
---|
71 | |
---|
72 | Identical to MESSAGE_COMMAND but starts a streaming command instead of |
---|
73 | a regular command. A separate token is used for this purpose to avoid |
---|
74 | changing the format of the MESSAGE_COMMAND token to add an additional |
---|
75 | flag. |
---|
76 | |
---|
77 | MESSAGE_STREAM_DATA |
---|
78 | |
---|
79 | Used by both the client and the server to send data during a streaming |
---|
80 | command. The format is the same as a MESSAGE_OUTPUT token. For |
---|
81 | tokens from the client, the stream is required to be 1. Other streams |
---|
82 | are reserved for future versions of the protocol. |
---|
83 | |
---|
84 | MESSAGE_STREAM_END |
---|
85 | |
---|
86 | Indicates an end of data on a stream, equivalent to an EOF condition. |
---|
87 | The only content of this token is the stream number. After this token |
---|
88 | is sent, no further MESSAGE_STREAM_DATA tokens with that stream number |
---|
89 | will be sent as part of this command. When sent by the client, this |
---|
90 | indicates the end of streaming data from the client and promises that |
---|
91 | the client will send no further data until the server sends the end of |
---|
92 | command token. |
---|
93 | |
---|
94 | MESSAGE_COMMAND_END |
---|
95 | |
---|
96 | Indicates the end of a streaming command. The format is identical to |
---|
97 | the MESSAGE_STATUS token. A separate token is used rather than |
---|
98 | reusing the MESSAGE_STATUS token only because the expected client |
---|
99 | response is different (the client may need to send a final |
---|
100 | MESSAGE_STREAM_END token) and it may be useful for the client to |
---|
101 | easily distinguish. |
---|
102 | |
---|
103 | Implementation Issues |
---|
104 | |
---|
105 | The remctl server will be responsible for translating the network |
---|
106 | packets from the client into data on standard input for the command |
---|
107 | and output from the command into network packets. To do so safely |
---|
108 | without making assumptions about ordering of data, it will need to |
---|
109 | poll the running command and the client network connection and do |
---|
110 | non-blocking reads and writes. However, it may need to block the |
---|
111 | client until the command can process data, or block the command while |
---|
112 | waiting for client data. |
---|
113 | |
---|
114 | I think the best way of handling this is to add buffering code to the |
---|
115 | server with an upper limit on the allowed amount of data buffering. |
---|
116 | The remctl server will then accept data from either the command or the |
---|
117 | client up to the buffer size even if it can't yet write that data to |
---|
118 | its destination. Once the buffer is full, the server will stop |
---|
119 | polling that connection until the buffer has been drained, forcing the |
---|
120 | other end of that connection to block. |
---|
121 | |
---|
122 | One challenge for such buffering is that the amount of data contained |
---|
123 | in a single GSS-API token may vary, and ideally the server should not |
---|
124 | partially consume that data. If the server accepts a token, it should |
---|
125 | read the entire token. This is not strictly necessary, but I think |
---|
126 | partially consuming an output token would significantly increase the |
---|
127 | risk of deadlock. |
---|
128 | |
---|
129 | Limitations |
---|
130 | |
---|
131 | This protocol does not address deadlock and instead requires that the |
---|
132 | client and server be aware of deadlock issues and find ways of |
---|
133 | avoiding deadlock within the implementation of a streaming command. |
---|