Several servers had been provisioned for an industrial data platform. The design described buffered ingestion, logical service endpoints, redundant nodes, shared state, and separate roles for data, application, and visualization. From that, the platform was taken to be highly available.
The assumption
The intended architecture treated the platform as one service. Clients and interface nodes would use a logical endpoint. Ingestion would be buffered. Application and data nodes would be redundant and would perform the same role. State the service depends on would be shared. Data, the application framework, and the visualization tier would be separate. Failover would not require clients to be reconfigured.
- Clients Interfaces Aimed at a service, not at a chosen server.
- Ingestion Buffered intake Holds data while the service is briefly unreachable.
Both use one logical service endpoint.
- Same role on each Redundant data nodes Failover does not ask clients to pick the active node.
Application and visualization are separate tiers.
- Separate tier Application framework Not combined with the data tier.
- Separate tier Visualization Not combined with the data tier.
The question
Does the running system behave the way that design says it should?
The review traced the running system: data paths, installed services, application roles, storage, where databases lived, configuration, identities and dependencies, which state was local and which was shared, and what failover would require of a client.
What the system was doing
What we found did not match the intended high-availability design.
- Ingestion was concentrated on one active node.
- Clients and interfaces depended on specific server identities.
- Important state remained local to individual nodes.
- Nominally redundant nodes did not have the same configuration or the same functionality.
- Other infrastructure nodes existed and were unused, or did not serve the role they appeared to have.
- Data, application, visualization, and supporting services were mixed on the same nodes.
In that arrangement, the nodes did not fail over as one service.
- Clients Bound to one identity Interfaces target a specific server, not a logical endpoint.
- Ingestion One active node Intake is concentrated there.
Clients reach the active node directly.
- Local Node-local state The database and other state the service needs stay on the node.
- Not equivalent Nominal peer Present, with different configuration and functionality.
- Not in the path Other infrastructure Provisioned, unused, or not serving the role it appeared to have.
Why it matters
A second node does not make a service redundant if clients are aimed at the first node and the state they need never leaves that node. Losing the active node does not move the service. A second machine does not take its place merely because it exists.
Redundant infrastructure is not the same thing as a redundant service.
The architecture direction
The review produced a remediation design. It did not rebuild the environment.
The design puts a logical service endpoint in front of redundant nodes that run the same role. Ingestion to that endpoint is buffered. State the service depends on is shared, rather than kept in a database local to whichever node last answered. Data, the application framework, and visualization are separated. Failover does not require a client to learn which physical node is active.
- Not established in this review Upstream interface layer Reviewed at a high level only. Further analysis is required before changes there.
The specified design starts below that layer.
- Ingestion Buffered intake Writes to the logical endpoint, not to one server name.
- Clients Logical service endpoint Failover does not require client reconfiguration.
Shared state sits behind the endpoint.
- Consistent roles Redundant data nodes Each node can serve the same role. Configuration matches.
Application and visualization are separate tiers.
- Separate tier Application framework Not the home of the data tier’s state.
- Separate tier Visualization Reads the service. Does not decide which physical node is active.
What this review did not establish
Don’t validate the diagram
Validate the system. Trace the path a client actually uses, where the state actually lives, and whether another node can continue that service without the client being retargeted. A drawing of redundant servers cannot answer that.