Purpose

• In this video, we’ll be uploading our internal data to our vector database, Pinecone.
• We’ll introduce Document Loaders from Langchain and ingest some documents
  • A folder of PDFs
  • GitHub Folder
• In the later videos, we’ll integrate this functionality for Websites and YouTube videos, so stay tuned.

What are Document Loaders?

Tasks

1. Update our dependencies.

   1. Run this in the root directory

      pip install -r requirements_mimir.txt
      

2. Add our Pinecone Environment Key.

   1. Go to your Pinecone console https://app.pinecone.io/
   2. Go to Manage / API keys
   3. Add to your .env file

3. Add our GitHub repository key.

   1. Go to Settings/Developer settings https://github.com/settings/tokens?type=beta
   2. Click Generate new token
   3. Paste it to your .env file

4. Upload our documents to the Pinecone index.

   1. GitHub
   2. PDFs

5. View our documents in Pinecone console.

6. Run our first similarity search.

Conclusion

• Awesome! So we’ve now populated our vector database with a bunch of differnet data sources. In the next section, we’ll put this section to work, by giving our chatbot function, the ability to call our vector database and actually retrieve some of these documents.