- [Chander] So let's talk about querying and indexing. Let's say we have a document, and when I say document, consider that as a row in a SQL Server table. In this case, we are talking about search, so this is going to be a flat document which means it's really a document in a no SQL database like Azure's database. In the document, we have certain properties. The Name is my name, and then we have Company: Cazton. Then, you can see we have quite a few skills.
Next, you have an Index Writer, and this is exactly how a search engine works. There is an index writer that looks at the document, analyzes the document. What is Analysis mean? Well, analyzer is something that we can either create on our own or reuse one of the existing ones that are part of a search engine. For example, if we want to use a stop word analyzer, we can take the entire data and say remove all the stop words that come as part of it.
A lot of times you'll notice people will remove a, an, and the because they're more like articles, and they may not add any benefit to the entire data as it is. They might also use an analyzer that might also forget the casing in general and just go with the lowercase because a lot of times, even if you have angular with a capital, your user might choose to search it with lowercase. So a lot of these analyzers could be created, and then they can analyze the data we have which is the raw data we have, to create the index as we just did in the last example, in the last slide.
Then we have the querying part of it where the user is going to query your index, not the actual data. When we query, we have the index searcher or the reader which is smart enough to use the same analyzer which was used to create that index, and then present that data in a format that's better for the user to understand. This query could be done by a client which could also be a client like ASP.NET Core, no-js or anything like that, and it talks to the next searcher, and then adds its own flavor to it.
For example, if the query was done from a web page, and you have a certain format you want to display that web page in, you might want to use certain kind of CSS layout for it. Maybe what you could do is grab the data, and have your .net application talk to the index searcher once you have the data in hand, you might want to convert it over to JSON, and then use your own CSS or anything else, and that could be an Intermediary Layer. This is a very simplified diagram that explains to you how indexing works and then how querying for the same index works.
One of the things to remember is that the retrieval from the index is going to be a set of matching documents. A lot of times, you're still going to have some kind of ranking algorithm to return to the user what you want to return. Let's take an example where you get 100 thousand records back. You have to have some way of ordering it or presenting it to the user in a fashion that makes sense for him. Now you could have a UI where you say order by some kind of category, or you can also have some logic in your application based on the users history of interacting with their application, and you can recommend different things on the first page or the first few pages rather than showing all the 100 thousand records because that is simply not feasible.
- Querying and indexing
- Creating a search service
- Using APIs during searching
- Importing JSON data
- Handling synonyms
- Working with suggestors and facets