My company’s docs are all hosted as HTML documents at. The Qdrant service will run inside a Docker container.

Install the qdrant-client Python package and launch a Qdrant server via Docker: you will use Qdrant to create a locally hosted vector index for the docs, against which queries will be run.

Install the openai Python package and create an account: you will use this account to send your docs and queries to an inference endpoint, which will return an embedding vector for each piece of text.īetter yet, if you want to implement semantic search for your own website using this method, you can follow along! Here are the ingredients you’ll need: You can find all the code for this post in the voxel51/fiftyone-docs-search repo, and it’s easy to install the package locally in edit mode with pip install -e. Wrapped it all in a user-friendly command line interface and Python API.Generated a vector index from these embedding.Split docs into blocks and added some automated cleanup.Converted all of the docs to a unified format.So, here’s how I turned our docs into a semantically searchable vector database: Semantically search your company’s docs from the command line. I was not going to let this fly… so I built this in my spare time: As a user myself, I sometimes find that given the sheer quantity of documentation, finding precisely what I’m looking for requires more time than I’d like. We have a lot of documentation, which has its pros and cons. This put us in a curious position: it was now possible for people using open source FiftyOne to readily search datasets with natural language queries, but using our documentation still required traditional keyword search. As a machine learning engineer and developer evangelist, my job is to listen to our open source community and bring them what they need - new features, integrations, tutorials, workshops, you name it.Ī few weeks ago, we added native support for vector search engines and text similarity queries to FiftyOne, so that users can find the most relevant images in their (often massive - containing millions or tens of millions of samples) datasets, via simple natural language queries. For the past six months, I’ve been working at series A startup Voxel51, a and creator of the open source computer vision toolkit FiftyOne.

Triple your membership toolkit download