Data Retrieval & Queries

The Data Retrieval & Queries service provides a robust, abstracted interface for interacting with the underlying search and analytics engine (Elasticsearch/OpenSearch). It handles everything from basic document indexing and index management to complex filtered searches, aggregations, SQL queries, and specialized compliance evaluations.

Under the hood, the service utilizes an OpenSearch-compatible client. Exceptions are typically caught, logged via the eventService, and re-thrown as RuntimeException or UtmElasticsearchException to ensure the application layer can handle failures gracefully.

Index Management

Before interacting with documents, you often need to manage the indices that store them. The service provides methods to check existence, index new documents, and perform bulk deletions.

Use indexExist(String index) to check if an index or index pattern exists before attempting operations that might fail on missing indices.

Use index(String index, T document) to insert or update a document. The method is generic, allowing you to pass any strongly-typed Java object.

Use deleteIndex(List<String> indices) to perform bulk removals of obsolete indices.

The deleteIndex method performs a bulk deletion. Always validate the indices list before execution to prevent accidental data loss. If the list is empty, the method safely returns without action.

Search Operations

The service provides multiple ways to retrieve documents, ranging from simple existence checks to paginated, filtered searches.

Standard Search

The primary search method takes a list of filters, pagination parameters, and a target class type. It automatically builds the underlying SearchRequest.

// Example usage of the search method
public <T> SearchResponse<T> search(
    List<FilterType> filters, 
    Integer top, 
    String indexPattern,
    Pageable pageable, 
    Class<T> type
)

Utility Search Methods

For optimized querying when you don't need the actual document payloads, use the following utility methods:

The buildQuery helper method constructs the SearchRequest. It applies the target index pattern, converts the provided List<FilterType> into a native query using SearchUtil.toQuery(), and applies pagination and sorting rules via SearchUtil.applyPaginationAndSort().

Aggregations and Updates

Field Value Aggregation

To generate analytics or populate filter dropdowns in the UI, use getFieldValuesWithCount. This method aggregates all unique values for a specific field and returns the document count for each.

Map<String, Long> getFieldValuesWithCount(
    String field, 
    String index, 
    List<FilterType> filters, 
    Integer top,
    boolean orderByCount, 
    boolean sortAsc
)

Use orderByCount = true and sortAsc = false to quickly retrieve the "Top N" most frequent values for a given field.

Update By Query

To modify multiple documents that match specific criteria without retrieving them to the application layer, use updateByQuery.

public void updateByQuery(Query query, String index, String script)

This method executes a Painless script against all documents matching the provided Query in the specified index.

Advanced Querying

SQL Query Support

The service supports executing raw SQL queries against the search engine via the searchBySql method. This is particularly useful for complex joins or aggregations that are easier to express in SQL than in the native Query DSL.

public <T> SearchSqlResponse<T> searchBySql(SqlQueryRequest request, Class<T> responseType)

Compliance Evaluation History

The service includes specialized methods for retrieving compliance control evaluations. These methods query a specific, hardcoded index: v11-log-compliance-evaluation.

Retrieving Evaluations

There are two primary methods for fetching compliance data for a specific controlId:

getControlEvaluations(Long controlId):
- Retrieves the last 30 evaluations.
- Sorted by timestamp in descending order.
- Maps the raw hits to UtmComplianceControlEvaluationHistoryDto.
getLatestControlEvaluation(Long controlId):
- Highly optimized query that sets size(1).
- Returns only the single most recent evaluation or null if none exist.
- Maps the result using UtmComplianceControlLatestEvaluationMapper.