Stop MAC servers in parallel

[DomGarguilo]

Looking through code paths that could help speed up the ITs and found this one.

In the old code, MiniAccumuloClusterControl.stop() stops tservers, sservers and compactors one at a time and waits after stopping each. The new code now sends the SIGTERM to all processes of that server type first, then waits on each. This allows them all to start shutting down in parallel before the wait step.

I did some rough testing by timing stop() and found that this speeds things up by ~350ms per server which is negligible on a single run but the time savings multiply.

[dlmarion]

It looks like we are waiting up to 30 seconds per server even with your change.

            for (Process tserver : tabletServerProcesses) {
              try {
                cluster.stopProcessWithTimeout(tserver, 30, TimeUnit.SECONDS);
              } catch (ExecutionException | TimeoutException e) {
             ...
           }

I wonder if instead we change the signature of cluster.stopProcessWithTimeout to accept all processes, incorporate your change in the that method, and wait a total of 30s for all processes to stop instead of one.

Thoughts?

[DomGarguilo]

Yea I think there are some options here. The changes here keep the same order in which servers are shut down. We could try shutting all processes down at once but that would obviously not guarantee shutdown order which might be needed for some reason in certain tests.

Its worth mentioning that the 30s wait is just a worst case scenario but it is per process so if all processes fail to stop properly we would be waiting 30s for each.

[dlmarion]

The changes here keep the same order in which servers are shut down. We could try shutting all processes down at once but that would obviously not guarantee shutdown order which might be needed for some reason in certain tests.

I'm not sure I'm following w/r/t the order. The changes are in a switch statement based on the server method parameter. So, in this method, we are only shutting down one type of server. I'm just suggesting that instead of this method calling cluster.stopProcessWithTimeout(tserver, 30, TimeUnit.SECONDS); it calls cluster.stopProcessWithTimeout(tabletServerProcesses, 30, TimeUnit.SECONDS); instead and inside of stopProcessWithTimeout we shut them all down in parallel.

[DomGarguilo]

I'm not sure I'm following w/r/t the order. The changes are in a switch statement based on the server method parameter. So, in this method, we are only shutting down one type of server. I'm just suggesting that instead of this method calling cluster.stopProcessWithTimeout(tserver, 30, TimeUnit.SECONDS); it calls cluster.stopProcessWithTimeout(tabletServerProcesses, 30, TimeUnit.SECONDS); instead and inside of stopProcessWithTimeout we shut them all down in parallel.

Yea you're right. I was conflating this with stopping all server types at once. Within a single stop() call, order shouldnt matter. It looks like that is how it is in main already where stopProcessesWithTimeout() waits 30s per server type in total which makes more sense.

[DomGarguilo]

In a7f6e98 I added a new method that accepts a list of processes, starts to shut them all down, then waits for them all under a single timer.

This should be in line with what you mentioned above @dlmarion

[dlmarion]

Looks good

[dlmarion]

Kicked off a full IT build

[dlmarion]

Full IT build successful, looks like it shaved about 20 minutes off the time. Great job!