AI-Driven WordPress Search by Crocoblock

It’s well-known that the native WordPress search is not convenient or flexible, and it’s especially obvious for big websites; that’s why it’s so common to use search plugins to extend this functionality. 

The JetSearch plugin has been there for a while, improving the user experience by adding AJAX functionality and quick, customizable result previews, sorting, etc. But now is the perfect time to spice it up with new AI functionality and make it even more powerful and user-oriented. 

In this article, I will talk about our experiments in implementing Open AI to WordPress search, the pitfalls, and the successes we achieved.

Table of Contents

• Experiments with WordPress and Open AI
  • Open AI Embeddings for WordPress search
• WordPress Native vs. AI-Driven Search
  • Typos in search
  • No typos in the search
• Drawbacks of WordPress AI-Powered Search
• AI Seach Experimental Plugin
  • Advanced usage: search by custom fields
• FAQ
• Takeaway

Experiments with WordPress and Open AI

Our first implementation of AI into one of our plugins was adding it to the Query Builder tool of JetEngine to generate SQL queries. But this is a job for generative AI – when it creates things from scratch – while for handling search, we need it to work with existing data for handling searches. 

In our case, we initially wanted to improve search results for our Help Center (Knowledge Base) because there are tons of information, and it would be great if users could always find exactly what they need without making much effort. Our clients sometimes have to ask support agents about things that already exist in documentation; thus, there’s a space for improvement. 

We planned to release the AI search functionality and offer it to our clients as soon as we implemented and tested it. We’ve actually succeeded and are now in a phase of active testing of the experimental plugin. 

I will talk you through our development process while creating this plugin. 

Open AI Embeddings for WordPress search

Open AI official documentation suggests using Embeddings for search. It allows you to analyze the text that you give it and generate a mathematical vector – which is a sequence of numbers. Hence, we can give this model a large volume of information, get its representation in vector format, save it, and then compare specific vectors. 

This was the general explanation. Now, I will break it down into points and tell you what exactly we have done and how this logic works. 

1. Splitting the Knowledge Base content into fragments.
2. Then, using the Embeddings API to convert them into vectors.
3. Saving the vectors in the website database.
4. Creating a search interface where users will type their search requests.  
5. Converting the user’s request into a vector as well (using Embeddings API).
6. Comparing the users’ request vector with the existing vectors of the text fragments from our database. At this point, we sort all of these fragments using mathematical methods to find the most similar matches. 

Hence, the most similar vectors will be the text fragments most relevant to the user’s request. 

All that remains is to deliver a predefined number of these fragments to the front end in the order of relevance, with the links leading to our website’s corresponding section. In simple words, display the search results. 

WordPress Native vs. AI-Driven Search 

AI improves WordPress search, and I will demonstrate this in the upcoming section. 

Typos in the search 

If there’s a typo in WordPress native search, it will simply not show any results. For example, I’ve made a typo in the word “sweater” and typed “seater with voluminous sleeves.” These are the results of the native search. As you can see, nothing was found.

Now, I’ve activated the experimental Jet AI Search plugin and used the same search request with the same typo. Look at the difference – all the existing sweaters with voluminous sleeves were found, plus a blouse with voluminous sleeves. Quite a difference, isn’t it?

This experiment shows the difference in how the native and AI-powered search works:

• WordPress native search tries to find the exact match of the search phrase in the website’s database. It means you will get zero results if you make the slightest spelling mistake in one of the words.
• AI-powered search can understand the context because it works with vectors, so it’s not sensitive to typos.

Obviously, it’s a huge advantage.

No typos in the search

Now, let’s check how native and AI-powered searches perform if there are no typos in the request. I’ve used the same phrase, “sweater with voluminous sleeves.”

The screenshot below shows that it found two relevant products but then offered completely irrelevant home pages.

Now, look what AI-search did: it suggested the same two completely relevant products, plus all the sweaters and everything with voluminous sleeves, which is great for good customer experience and upsales. 

To sum it up, it would be fair to say that the results of a native WordPress search without a typo are worse than the results of an AI-powered search, even if there’s a typo in the request. Pretty impressive!

Drawbacks of WordPress AI-Powered Search

We face a few challenges when implementing AI into a WordPress search. Mainly, it’s all about resources. 

Open AI Embeddings API generates quite big vectors, and it requires resources to process them, especially if we are talking about PHP, which is not well-adapted to such tasks. This leads to a reduction in the speed when working with large volumes of content.

So, if your site is quite small, our AI search tool will work at a normal speed. But on large volumes of content, it slows down exponentially. 

The second challenge is operational logic. To get relevant results, you should break content into fragments properly. The fragments should not be too long (because the search requests are not long, and they should match). 

At the same time, if the fragments are too small, they might lose any meaning – and the search results will not be relevant because they will not reflect the meaning but just match a set of letters. 

There are two potential solutions: the first is to use another tool that generates smaller vectors than Embeddings. However, the search accuracy will be lower because of this. 

Another solution is to use a SaaS model where data storage and request processing will be carried out by more suitable tools for users while the web interface works as an API for indexing site content and searching for that indexed content. 

As a result, we will get the same results but faster and by using fewer resources. 

💡 If you want to learn more details, watch the talk of Andrew Shevchenko, Crocoblock’s CTO, about the development of the AI search. Also, check out the live chat discussion, where people shared their thoughts and ideas on this topic.

AI Seach Experimental Plugin

You can download the plugin from our GitHub repository, test it, and even use it on the production site. All you need to have is some money on the Open AI account to perform searches. 

The interface is pretty straightforward; its settings are on Dashboard > Settings > AI Search. 

You can leave default Working Mode settings, but you definitely need to set the post types you want to be auto-fetched and fetch content for the first time in the Fetch Content tab. After that, you will see the posts being added to the Fetched Content Statistics. 

If you want to try how this plugin works on a live site, check out the JetFormBuilder site and how it performs search in the documentation. It’s strongly recommended to use the Crocoblock JetSearch plugin powered by AJAX and see results on the same page, right under the search field.

Advanced use: search by custom fields

This code snippet allows you to add values from meta fields to parsed content fragments. By using it, you can perform the AI search in the selected custom fields, not only posts content or excerpts.
Don’t forget to change the names to match the names on your website. After adding this code, you need to re-fetch the corresponding post type’s content to ensure new fragments are stored in the database.

add_filter( 'jet-ai-search/post-fragments', function( $fragments, $post, $parser ) {

	// Optional part - making sure we work with posts of need type
	if ( 'product' !== $post->post_type ) {
		return $fragments;
	}

	// Getting field value
	$field_name = 'description-for-the-search';
	$custom_description = get_post_meta( $post->ID, $field_name, true );

	// If value is empty - nothing to do here
	if ( ! $custom_description ) {
		return $fragments;
	}

	// Reset previously parsed stack and store new fragment
	$parser->reset_results();

	$parser->set_stack_defaults( [
		'post_id'    => $post->ID,
		'post_url'   => $post->guid,
		'post_title' => $post->post_title,
		'source'     => $post->post_type,
	] );

	// Optional - store post title with field value for better context
	$title = $parser->prepare_heading( $post->post_title );

	$fragment = $parser->prepare_fragment( $custom_description );

	$parser->stack_result( [
		'fragment' => $title . $fragment
	], true );

	// Merge stored fragment with all previous fragments
	$fragments = array_merge( $fragments, $parser->get_result() );

	return $fragments;
	
}, 10, 3 );

FAQ

Takeaway

In this article, I’ve talked you through the process of developing the experimental plugin for an AI-powered search for WordPress websites, showed how it works, and why such enhancements are so important for almost any project.  

Feel free to test the plugin on your website and give feedback on GitHub or in the comments below.