Replacing OR/AND/NOT filters with bool filter creates a hard-to-understand query with too many levels?

Refresh

April 2019

Views

386 time

1

I have the following filter in a filtered query. As seen, it has many OR/AND/NOT filters at different levels. I was advised to replace them with bool filters for performance reasons, and I am going to do that.

"filter" : {
                "or" : [

                    {
                        "and" : [
                            { "range" : { "start" : { "lte": 201407292300 } } },
                            { "range" : { "end" : { "gte": 201407292300 } } },
                            { "term" : { "condtion1" : false } },
                            {
                                "or" : [
                                    {
                                        "and" : [
                                            { "term" : { "condtion2" : false } },
                                            {
                                                "or": [
                                                    {
                                                        "and" : [
                                                            { "missing" : { "field" : "condtion6" } },
                                                            { "missing" : { "field" : "condtion7" } }
                                                        ]
                                                    },
                                                    { "term" : { "condtion6" : "nop" } }
                                                    { "term" : { "condtion7" : "rst" } }
                                                ]
                                            }
                                        ]
                                    },
                                    {
                                        "and" : [
                                            { "term" : { "condtion2" : true } },
                                            {
                                                "or": [
                                                    {
                                                        "and" : [
                                                            { "missing" : { "field" : "condtion3" } },
                                                            { "missing" : { "field" : "condtion4" } },
                                                            { "missing" : { "field" : "condtion5" } },
                                                            { "missing" : { "field" : "condtion6" } },
                                                            { "missing" : { "field" : "condtion7" } }
                                                        ]
                                                    },
                                                    { "term" : { "condtion3" : "abc" } },
                                                    { "term" : { "condtion4" : "def" } },
                                                    { "term" : { "condtion5" : "ghj" } },
                                                    { "term" : { "condtion6" : "nop" } },
                                                    { "term" : { "condtion7" : "rst" } }
                                                ]
                                            }                                       
                                        ]
                                    }
                                ]
                            }

                        ]
                    },

                    {
                        "and" : [
                            { 
                                "term": { "condtion8" : "TIME_POINT_1" } 
                            },
                            {  "range" : { "start" : { "lte": 201407302300 } } },
                            { 
                                "or": [
                                    { "term" : { "condtion9" : "GROUP_B" } }, 
                                    {
                                        "and" : [
                                            { "term" : { "condtion9" : "GROUP_A" } },
                                            { "ids" : { values: [100, 10] } }
                                        ] 
                                    }
                                ]
                            }
                        ]
                    },

                    {
                        "and" : [
                            { 
                                "term": { "condtion8" : "TIME_POINT_2" } 
                            },
                            { "ids" : { values: [100, 10] } }
                        ]
                    },                  

                    {
                        "and" : [
                            { 
                                "term": { "condtion8" : "TIME_POINT_3" } 
                            },
                            { 
                                "or": [
                                    { "term" : { "condtion1" : true } },
                                    { "range" : { "end" : { "lt": 201407302300 } } }
                                ]
                            },
                            { 
                                "or": [
                                    { "term" : { "condtion9" : "GROUP_B" } }, 
                                    {
                                        "and" : [
                                            { "term" : { "condtion9" : "GROUP_A" } },
                                            { "ids" : { values: [100, 10] } }
                                        ] 
                                    }
                                ]
                            }
                        ]
                    }

                ]
            }

However, I feel replacing these OR/AND/NOT filters would create a query that has too many levels and is hard to understand. For example, replacing

"or": [
   ....
]

I have to have:

"bool" {
   "should": [
    ]
}

Am I right that replacing OR/AND/NOT with bool filter in my case is at the expense of sacrificing understandability?

A related question

If I have to replace OR/AND/NOT filters for performance, should I replace ALL of these OR/AND/NOT filters, or just some of them such as the one at the top for example?

Thanks and regards.

1 answers

1

You should replace all of them except geo/script/range filters. Having said that understanding the possible impact of each filter can help you also. For example if one of the filter is going to filter out say 90% of the result then you may want to put that in an and filter at the starting. Since and/or filters are executed sequentially the rest of the filters will have lesser documents to process. In case of bool filters all the filters are combined in a single bitset operation. You might have already read about it.

I don't think you will be sacrificing understability by replacing OR/AND/NOT with bool filter. As the example you have given, for a single or filter converting to should filter looks like an increase in the query structure but in an overall combination the structure would be almost similar.