# Batch Get Chunks Source: https://morphik.ai/docs/api-reference/batch-get-chunks https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /batch/chunks Retrieve specific chunks by their document ID and chunk number in a single batch operation. # Batch Get Documents Source: https://morphik.ai/docs/api-reference/batch-get-documents https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /batch/documents Retrieve multiple documents by their IDs in a single batch operation. # Creating Apps Source: https://morphik.ai/docs/api-reference/creating-apps Provision isolated Morphik apps and generate connection URIs. Morphik apps are isolated data environments. Each app has its own documents, embeddings, and auth token, so data stays separated even when apps live on the same cluster. Think of an app as a separate Morphik instance with a shared control plane. Common uses: * Create one app per customer or tenant to keep data segregated. * Split environments (prod, staging, sandbox) without running multiple clusters. * Separate projects with different data retention or access policies. ## Create a new app (cloud) **POST** `/cloud/generate_uri` This endpoint creates an app and returns a Morphik URI that clients use to connect to it. ### Authentication Provide a Bearer token in `Authorization: Bearer `. Use an existing Morphik API token to create apps and mint new URIs programmatically. ### Request Body Optional client-generated app id (recommended: UUID). If omitted, the server generates one. Human-friendly app name. Used in the Morphik URI. Days until the token expires (default: 3650). ### Example request ```bash theme={null} curl -X POST \ https://api.morphik.ai/cloud/generate_uri \ -H 'Authorization: Bearer YOUR_JWT_TOKEN' \ -H 'Content-Type: application/json' \ -d '{ "name": "customer-acme" }' ``` ### Response Connection URI in the format `morphik://name:token@host`. The app id associated with the URI. **Example response:** ```json theme={null} { "uri": "morphik://customer-acme:eyJhbGciOi...@api.morphik.ai", "app_id": "f5c5e51a-7a1b-4c8d-8d7e-3c5ed3c6c7b2" } ``` ### Notes * The response always contains a newly minted token for the app. * If `app_id` is omitted, the server generates one. * `name` is required. * App names must be unique per owner or org; duplicates return 409. * If the account tier has reached its app limit, the API returns 403. # Delete Cloud App Source: https://morphik.ai/docs/api-reference/delete-cloud-app https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml delete /apps Delete all resources associated with a given cloud application. # Delete Document Source: https://morphik.ai/docs/api-reference/documents/delete-document https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml delete /documents/{document_id} Delete a document and all associated data. This endpoint deletes a document and all its associated data, including: - Document metadata - Document content in storage - Document chunks and embeddings in vector store # Download Document File Source: https://morphik.ai/docs/api-reference/documents/download-document-file https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /documents/{document_id}/file Download the actual file content for a document. This endpoint is used for local storage when file:// URLs cannot be accessed by browsers. # Extract Document Pages Source: https://morphik.ai/docs/api-reference/documents/extract-document-pages https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /documents/pages Extract specific pages from a document (PDF, PowerPoint, or Word) as base64-encoded images or URLs. When output_format="url", pages that fail URL generation fall back to base64 data URIs (mixed output possible). # Get Document Source: https://morphik.ai/docs/api-reference/documents/get-document https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /documents/{document_id} Retrieve a single document by its external identifier. Returns the :class:`Document` metadata if found or raises 404. # Get Document By Filename Source: https://morphik.ai/docs/api-reference/documents/get-document-by-filename https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /documents/filename/{filename} Get document by filename. # Get Document Download Url Source: https://morphik.ai/docs/api-reference/documents/get-document-download-url https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /documents/{document_id}/download_url Get a download URL for a specific document. # Get Document Status Source: https://morphik.ai/docs/api-reference/documents/get-document-status https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /documents/{document_id}/status Get the processing status of a document. # Get Document Summary Source: https://morphik.ai/docs/api-reference/documents/get-document-summary https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /documents/{document_id}/summary Retrieve the latest summary for a document. # List Docs Source: https://morphik.ai/docs/api-reference/documents/list-docs https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /documents/list_docs Flexible document listing with aggregates, projections, and advanced pagination. Alias: `/documents` and `/documents/list_docs` share this handler. **Supported operators**: `$and`, `$or`, `$nor`, `$not`, `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$in`, `$nin`, `$exists`, `$type`, `$regex`, `$contains`. **Implicit equality** (backwards compatible, JSONB containment): ```json {"status": "active"} ``` **Explicit operators** (typed comparisons for number, decimal, datetime, date): ```json {"priority": {"$gte": 40}, "end_date": {"$lt": "2025-01-01"}} ``` Use `document_filters` with a `filename` key to filter the filename column: ```json {"filename": {"$regex": {"pattern": "^report_.*\.pdf$", "flags": "i"}}} ``` Use `folder_name` and `end_user_id` query parameters to scope system metadata. # List Docs Source: https://morphik.ai/docs/api-reference/documents/list-docs-1 https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /documents Flexible document listing with aggregates, projections, and advanced pagination. Alias: `/documents` and `/documents/list_docs` share this handler. **Supported operators**: `$and`, `$or`, `$nor`, `$not`, `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$in`, `$nin`, `$exists`, `$type`, `$regex`, `$contains`. **Implicit equality** (backwards compatible, JSONB containment): ```json {"status": "active"} ``` **Explicit operators** (typed comparisons for number, decimal, datetime, date): ```json {"priority": {"$gte": 40}, "end_date": {"$lt": "2025-01-01"}} ``` Use `document_filters` with a `filename` key to filter the filename column: ```json {"filename": {"$regex": {"pattern": "^report_.*\.pdf$", "flags": "i"}}} ``` Use `folder_name` and `end_user_id` query parameters to scope system metadata. # Update Document File Source: https://morphik.ai/docs/api-reference/documents/update-document-file https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /documents/{document_id}/update_file Update a document by replacing its content with a new file and queueing re-ingestion. # Update Document Metadata Source: https://morphik.ai/docs/api-reference/documents/update-document-metadata https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /documents/{document_id}/update_metadata Update only a document's metadata. # Update Document Text Source: https://morphik.ai/docs/api-reference/documents/update-document-text https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /documents/{document_id}/update_text Update a document by replacing its text content and queueing re-ingestion. # Upsert Document Summary Source: https://morphik.ai/docs/api-reference/documents/upsert-document-summary https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml put /documents/{document_id}/summary Create or update a document summary with optional versioning. # Connector Oauth Callback Source: https://morphik.ai/docs/api-reference/ee--connectors/connector-oauth-callback https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /ee/connectors/{connector_type}/oauth2callback Handles the OAuth 2.0 callback from the authentication provider. Validates state, finalizes authentication, and stores credentials. # Disconnect Source: https://morphik.ai/docs/api-reference/ee--connectors/disconnect https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /ee/connectors/disconnect Disconnect from a connector and remove credentials. # Finalize Auth Source: https://morphik.ai/docs/api-reference/ee--connectors/finalize-auth https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /ee/connectors/finalize-auth Finalize the OAuth flow and exchange the code for a token. # Finalize Manual Auth Source: https://morphik.ai/docs/api-reference/ee--connectors/finalize-manual-auth https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /ee/connectors/{connector_type}/auth/finalize Finalize authentication using manual credentials. This endpoint is used for connectors that require manual credential input (like Zotero) instead of OAuth flows. # Get Auth Status For Connector Source: https://morphik.ai/docs/api-reference/ee--connectors/get-auth-status-for-connector https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /ee/connectors/{connector_type}/auth_status Checks the current authentication status for the given connector type. # Get Initiate Auth Url Source: https://morphik.ai/docs/api-reference/ee--connectors/get-initiate-auth-url https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /ee/connectors/{connector_type}/auth/initiate_url Return the provider's *authorization_url* for the given connector. The method mirrors the logic of the `/auth/initiate` endpoint but sends a JSON payload instead of a redirect so that browsers can stay on the same origin until they intentionally navigate away. For OAuth-based connectors, this returns authorization_url. For manual credential connectors, this returns the credential form specification. # Get Status Source: https://morphik.ai/docs/api-reference/ee--connectors/get-status https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /ee/connectors/status Get the authentication status for a connector. # Ingest File Source: https://morphik.ai/docs/api-reference/ee--connectors/ingest-file https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /ee/connectors/{connector_type}/ingest Ingest a single file from a connector. # Ingest Repository Source: https://morphik.ai/docs/api-reference/ee--connectors/ingest-repository https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /ee/connectors/{connector_type}/ingest-repository Ingest an entire GitHub repository. # Initiate Auth Source: https://morphik.ai/docs/api-reference/ee--connectors/initiate-auth https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /ee/connectors/initiate-auth Initiate the OAuth flow for a connector. # List Files Source: https://morphik.ai/docs/api-reference/ee--connectors/list-files https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /ee/connectors/list-files List files from a connector. # List Files For Connector Source: https://morphik.ai/docs/api-reference/ee--connectors/list-files-for-connector https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /ee/connectors/{connector_type}/files Lists files and folders from the specified connector. # Add Document To Folder Source: https://morphik.ai/docs/api-reference/folders/add-document-to-folder https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /folders/{folder_id_or_name}/documents/{document_id} Add a document to a folder. # Create Folder Source: https://morphik.ai/docs/api-reference/folders/create-folder https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /folders Create a new folder. # Delete Folder Source: https://morphik.ai/docs/api-reference/folders/delete-folder https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml delete /folders/{folder_id_or_name} Delete a folder and all associated documents. # Folder Details Source: https://morphik.ai/docs/api-reference/folders/folder-details https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /folders/details Retrieve folder metadata with optional document statistics and projections. # Get Folder Source: https://morphik.ai/docs/api-reference/folders/get-folder https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /folders/{folder_id_or_name} Get a folder by ID or name. # Get Folder Summary Source: https://morphik.ai/docs/api-reference/folders/get-folder-summary https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /folders/{folder_id_or_name}/summary Retrieve the latest summary for a folder. # List Folder Summaries Source: https://morphik.ai/docs/api-reference/folders/list-folder-summaries https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /folders/summary Return compact folder list (id, name, doc_count, updated_at). # List Folders Source: https://morphik.ai/docs/api-reference/folders/list-folders https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /folders List all folders the user has access to. # Remove Document From Folder Source: https://morphik.ai/docs/api-reference/folders/remove-document-from-folder https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml delete /folders/{folder_id_or_name}/documents/{document_id} Remove a document from a folder. # Upsert Folder Summary Source: https://morphik.ai/docs/api-reference/folders/upsert-folder-summary https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml put /folders/{folder_id_or_name}/summary Create or update a folder summary with optional versioning. # Generate Cloud Uri Source: https://morphik.ai/docs/api-reference/generate-cloud-uri https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /cloud/generate_uri Generate an authenticated URI for a cloud-hosted Morphik application. # Get Available Models Source: https://morphik.ai/docs/api-reference/get-available-models https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /models Get list of available models from configuration. Returns models grouped by type (chat, embedding, etc.) with their metadata. # Get Available Models For Selection Source: https://morphik.ai/docs/api-reference/get-available-models-for-selection https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /models/available Get list of available models for UI selection. Returns a list of models that can be used for queries. Each model includes: - id: Model identifier to use in llm_config - name: Display name for the model - provider: The LLM provider (e.g., openai, anthropic, ollama) - description: Optional description of the model # Get Chat History Source: https://morphik.ai/docs/api-reference/get-chat-history https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /chat/{chat_id} Retrieve the message history for a chat conversation. # Getting Started with Morphik API Source: https://morphik.ai/docs/api-reference/getting-started Quick guide to start using the Morphik API ## Get Your Credentials Copy URI from Morphik Dashboard From the Morphik dashboard: * **Python SDK**: Click "Copy URI" * **TypeScript/API**: Click "Copy Token" ## Installation ```bash theme={null} npm install morphik ``` ```bash theme={null} pip install morphik ``` ## Ingest a Document ```python theme={null} from morphik import Morphik # Initialize with your URI client = Morphik("YOUR_COPIED_URI") # Ingest a file with open('document.pdf', 'rb') as f: doc = client.ingest_file(f) print(f"Document ID: {doc.id}") ``` ```typescript theme={null} import Morphik from 'morphik'; import * as fs from 'fs'; // Initialize with your token const client = new Morphik({ apiKey: 'YOUR_COPIED_TOKEN' }); // Ingest a file const file = fs.createReadStream('document.pdf'); const doc = await client.ingest.ingestFile({ file }); console.log('Document ID:', doc.external_id); ``` ```bash theme={null} curl -X POST https://api.morphik.ai/ingest/file \ -H "Authorization: Bearer YOUR_COPIED_TOKEN" \ -H "Content-Type: multipart/form-data" \ -F "file=@document.pdf" ``` ## Query Your Documents ```python theme={null} from morphik import Morphik # Initialize with your URI client = Morphik("YOUR_COPIED_URI") # Query with RAG response = client.query( "What are the key points in this document?", k=5, use_colpali=True ) print(response.answer) ``` ```typescript theme={null} import Morphik from 'morphik'; const client = new Morphik({ apiKey: 'YOUR_COPIED_TOKEN' }); // Query with RAG const response = await client.query.generateCompletion({ query: 'What are the key points in this document?', k: 5, use_colpali: true }); console.log(response.completion); ``` ```bash theme={null} curl -X POST https://api.morphik.ai/query \ -H "Authorization: Bearer YOUR_COPIED_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "query": "What are the key points in this document?", "k": 5, "use_colpali": true }' ``` # Health Check Source: https://morphik.ai/docs/api-reference/health/health-check https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /health Comprehensive health check endpoint that queries all underlying services. Checks the following services: - PostgreSQL database - Redis - PGVector store - Storage service (Local/S3) - ColPali vector store (if enabled) # Ping Health Source: https://morphik.ai/docs/api-reference/health/ping-health https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /ping Simple health check endpoint that returns 200 OK. # Batch Ingest Files Source: https://morphik.ai/docs/api-reference/ingestion/batch-ingest-files https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /ingest/files Batch ingest **multiple files** (async). Each file is treated the same as :func:`ingest_file` but sharing the same request avoids many round-trips. All heavy work is still delegated to the background worker pool. # Ingest File Source: https://morphik.ai/docs/api-reference/ingestion/ingest-file https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /ingest/file Ingest a **file** asynchronously. The file is uploaded to object storage, a *Document* stub is persisted with ``status='processing'`` and a background worker picks up the heavy parsing / chunking work. # Ingest Text Source: https://morphik.ai/docs/api-reference/ingestion/ingest-text https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /ingest/text Ingest a **text** document asynchronously (queued like /ingest/file). # Query Document Source: https://morphik.ai/docs/api-reference/ingestion/query-document https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /ingest/document/query Execute a one-off analysis for a document using Morphik On-the-Fly, optionally enforcing structured output and scheduling a follow-up ingestion. `ingestion_options` is a JSON string controlling post-analysis ingestion behaviour via keys such as `ingest`, `metadata`, `use_colpali`, `folder_name`, and `end_user_id`. Additional keys are ignored. A :class:`DocumentQueryResponse` describing the inline analysis and any queued ingestion is returned. # Requeue Ingest Jobs Source: https://morphik.ai/docs/api-reference/ingestion/requeue-ingest-jobs https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /ingest/requeue Requeue ingestion jobs for documents stuck in processing or marked as failed. # List Chat Conversations Source: https://morphik.ai/docs/api-reference/list-chat-conversations https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /chats List chat conversations available to the current user. # List Cloud Apps Source: https://morphik.ai/docs/api-reference/list-cloud-apps https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /apps List provisioned apps for the specified organization/user. # Get Logs Source: https://morphik.ai/docs/api-reference/logs/get-logs https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /logs/ Return recent logs for the authenticated user (scoped by app_id). Args: hours: Number of hours of history to retrieve (default 4) - <= 4 hours: reads from local files - > 4 hours: queries logs.morphik.ai proxy (requires proxy endpoint) # Delete Model Source: https://morphik.ai/docs/api-reference/models/delete-model https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml delete /models/{model_id} Delete a custom model. # List Api Keys Source: https://morphik.ai/docs/api-reference/models/list-api-keys https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /api-keys List all configured API keys (sanitized). # List Custom Models Source: https://morphik.ai/docs/api-reference/models/list-custom-models https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /models/custom List all custom models for the authenticated user. # Save Api Key Source: https://morphik.ai/docs/api-reference/models/save-api-key https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /api-keys Save API key for a provider. # Save Model Source: https://morphik.ai/docs/api-reference/models/save-model https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /models Save a custom model configuration. # Query Completion Source: https://morphik.ai/docs/api-reference/query-completion https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /query Generate completion using relevant chunks as context. # Rename Cloud App Source: https://morphik.ai/docs/api-reference/rename-cloud-app https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml patch /apps/rename Rename an existing cloud application. # Retrieve Chunks Source: https://morphik.ai/docs/api-reference/retrieve-chunks https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /retrieve/chunks Retrieve relevant chunks. The optional `request.filters` payload accepts equality checks (which also match scalars inside JSON arrays) plus the logical operators `$and`, `$or`, `$nor`, and `$not`. Field-level predicates include `$eq`, `$ne`, `$in`, `$nin`, `$exists`, `$type`, `$regex`, `$contains`, and the comparison operators `$gt`, `$gte`, `$lt`, and `$lte`. Comparison clauses evaluate typed metadata (`number`, `decimal`, `datetime`, or `date`) and raise detailed validation errors when operands cannot be coerced. Regex filters allow the optional `i` flag for case-insensitive matching, while `$contains` performs substring checks (case-insensitive by default, configurable via `case_sensitive`). Filters can be nested freely, for example: ```json { "$and": [ {"category": "policy"}, {"$or": [{"region": "emea"}, {"priority": {"$in": ["p0", "p1"]}}]} ] } ``` Returns a list of `ChunkResult` objects ordered by relevance. # Retrieve Chunks Grouped Source: https://morphik.ai/docs/api-reference/retrieve-chunks-grouped https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /retrieve/chunks/grouped Retrieve relevant chunks with grouped response format. Uses the same filter operators as `/retrieve/chunks` (equality, `$eq/$ne`, `$gt/$gte/$lt/$lte`, `$in/$nin`, `$exists`, `$type`, `$regex`, `$contains`, and the logical `$and/$or/$nor/$not`), with arbitrary nesting supported inside `request.filters`. Returns both flat results (for backward compatibility) and grouped results (for UI). When padding > 0, groups chunks by main matches and their padding chunks. # Retrieve Documents Source: https://morphik.ai/docs/api-reference/retrieve-documents https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /retrieve/docs Retrieve relevant documents. `request.filters` supports equality checks (including scalar-to-array matches) and the same operator set as `/retrieve/chunks`: logical composition via `$and`, `$or`, `$nor`, `$not`, plus field predicates `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$in`, `$nin`, `$exists`, `$type`, `$regex`, and `$contains`. Use the same JSON structure as `/retrieve/chunks` when expressing complex logic. Comparison operators require metadata typed as `number`, `decimal`, `datetime`, or `date`. # Rotate App Token Source: https://morphik.ai/docs/api-reference/rotate-app-token https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /apps/rotate_token Rotate the token for an existing application. # Search Documents By Name Source: https://morphik.ai/docs/api-reference/search-documents-by-name https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml post /search/documents Search documents by filename using full-text search. `request.filters` accepts the same operator set as `/retrieve/chunks`: `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`, `$in`, `$nin`, `$exists`, `$type`, `$regex` (with optional `i` flag), `$contains`, and the logical operators `$and`, `$or`, `$nor`, `$not`. Comparison clauses honor typed metadata (`number`, `decimal`, `datetime`, `date`). # Update Chat Title Source: https://morphik.ai/docs/api-reference/update-chat-title https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml patch /chats/{chat_id}/title Update the title of a chat conversation. # Get App Storage Usage Source: https://morphik.ai/docs/api-reference/usage/get-app-storage-usage https://app.stainless.com/api/spec/documented/morphik/openapi.documented.yml get /usage/app-storage # Cloud Architecture Source: https://morphik.ai/docs/concepts/cloud-architecture How Morphik Cloud UI and Morphik Core interact in production deployments. Morphik Cloud is split into two services that communicate over HTTPS: * **Morphik Cloud UI**: The control plane that handles users, orgs, billing, and app provisioning. * **Morphik Core**: The data plane that stores documents and embeddings, runs ingestion, and serves retrieval and chat. This separation keeps your application management in the UI while all document data lives in the core API. ## Components at a glance | Component | Primary role | Typical hosting | | ------------------------ | --------------------------------------------- | ------------------------- | | Cloud UI | Auth, orgs, billing, app metadata, dashboards | Vercel (or your web host) | | Morphik Core | Ingestion, storage, retrieval, search, chat | EC2 or Kubernetes | | Embedding GPU (optional) | Multimodal embeddings (ColPali API mode) | Lambda GPU, on-prem GPU | | Postgres + pgvector | Documents, embeddings, app isolation | Neon or any Postgres | | Object storage | Raw files and chunk payloads | S3 or local disk | | Redis + worker | Async ingestion pipeline | Same VPC as core | ## Morphik URI: the contract between UI and Core When you create an app, Morphik Core returns a Morphik URI: ``` morphik://:@ ``` The Cloud UI parses this URI, extracts the token, and uses it for API calls: ``` Authorization: Bearer ``` The token contains the `app_id`, and Morphik Core uses that to isolate data per app. ## Provisioning flow (control plane) App creation is a control-plane operation that provisions an app and returns a Morphik URI. ```mermaid theme={null} sequenceDiagram participant Browser participant UI as Cloud UI participant Core as Morphik Core participant DB as Core Postgres Browser->>UI: Create app UI->>Core: POST /cloud/generate_uri Core->>DB: Create app + token Core-->>UI: morphik://... URI UI-->>Browser: App created ``` In a dedicated-cluster setup, the UI can call the cluster directly (instead of the shared API) and pass an admin secret to mint the URI. ## Runtime flow (data plane) Once an app exists, the UI talks to Morphik Core directly from the browser. Core verifies the token and scopes all reads and writes by `app_id`. ### Ingestion ```mermaid theme={null} sequenceDiagram participant Browser participant Core as Morphik Core participant Redis participant Worker participant GPU as Embedding API participant DB as Postgres + pgvector Browser->>Core: POST /ingest/file (Bearer token) Core->>DB: Create document record Core->>Redis: Enqueue ingestion job Worker->>GPU: Embed (ColPali API mode) Worker->>DB: Store embeddings + metadata Core-->>Browser: Ingest accepted ``` If you run in local embedding mode, the worker generates embeddings on the core instance instead of calling the external GPU. ### Retrieval and chat ```mermaid theme={null} sequenceDiagram participant Browser participant Core as Morphik Core participant DB as Postgres + pgvector Browser->>Core: POST /query or /retrieve/chunks Core->>DB: Vector search scoped by app_id Core-->>Browser: Results and citations ``` ## Agent mode (server-side) Agent mode runs in a server route (Cloud UI) so it can call your LLM provider securely. The agent uses Morphik Core as a tool: * The UI calls `/api/agent/chat` on the Cloud UI. * The server route calls Morphik Core for retrieval (using the app token). * The server route streams the LLM response back to the browser. # Retrieving Images Source: https://morphik.ai/docs/concepts/colpali Using Late-interaction and Contrastive learning to achieve state-of-the-art performance in visual retrieval ## Introduction Upto now, we've seen RAG techniques that **i)** parse a given document, **ii)** convert it to text, and **iii)** embed the text for retrieval. These techniques have been particularly text-heavy. Embedding models expect text in, and parsers break down when provided with documents that aren't text-dominant. This motivates the question: > When was the last time you looked at a document and only saw text? Most business documents, research papers, reports, and presentations we encounter daily are rich visual experiences: tables organizing crucial data, charts illuminating trends, infographics explaining complex concepts, and visual layouts that guide our understanding. These visual elements aren't just decorative—they're fundamental to how information is communicated. However, most RAG systems treat these elements as second-class citizens. They are either ignored or captioned and embedded as text. This leads to poor retrieval performance - especially for tasks that require visual reasoning. In this guide, we'll explore a series of models, starting with *ColPali* that are built from the ground up to help retrieve images with the same fidelity as text. ## What is ColPali? The core idea behind ColPali is simple: the core bottleneck in retrieval is not the performance of the embedding model, but **prior data ingestion pipeline**. As a result, this new technique proposes doing away with any data preprocessing - embedding the entire document as a list of images instead. ColPali Architecture The diagram above shows the ColPali pipeline when compared with traditional layout-detection based data ingestion pipelines. Directly ingesting the document as a list of images significantly speeds up the time taken to ingest each document, while also ensuring higher retrieval quality. ## How does it work? ### Embedding Process The embedding process for ColPali borrows heavily from models like CLIP. That is, the vision encoder part of the model (as seen in the diagram above) is trained via a technique called **Contrastive Learning**. As we've discussed in previous explainers, an encoder is a function (usually a neural network or a transformer) that maps a given input to a fixed-length vector. Contrastive learning is a technique that allows us to train two encoders of different input types (such as image and text) to produce vectors in the "same embedding space". That is, the embedding of the word "dog" would be very close to the embedding of the image of a dog. The way we can achieve this is simple in theory: 1. Take a large dataset of image and text pairs. 2. Pass the image and text through the vision and text encoders respectively. 3. Compute the dot product of the image embeddings and text embeddings. 4. Penalize the encoders for embeddings that are not close to each other (i.e. a low dot product). Over time, the encoders will learn to produce embeddings that project to the same space. As we scale this up, as researchers did with [SigLIP](https://arxiv.org/abs/2303.15343), we see that instead of just matching images of objects to their corresponding word, we can also match images of the handwritten version of the word to their corresponding word in text. So, we have a system that, given an image, can provide a vector embedding that lies in the same space as a text embedding. ### Retrieval Process The retrieval process for ColPali borrows from late-interaction based reranking techniques such as [ColBERT](https://arxiv.org/abs/2004.12832). The idea is that instead of directly embedding an image or an entire block of text, we can embed individual patches or tokens instead. Then, instead of using the regular dot product or the cosine similarity, we can employ a slightly different scoring function. This scoring function looks at the most similar patches and tokens, and then sums those similarities up to obtain a final score. ColBERT Architecture In order to speed up the retrieval process, Morphik uses a technique that computes the [hamming distance](https://en.wikipedia.org/wiki/Hamming_distance) between individual embeddings instead of the dot product. This is because the hamming distance is a much faster operation than the dot product, and helps scale the retrieval process to millions of documents. This technique is borrows from the amazing team at [Vespa](https://blog.vespa.ai/scaling-colpali-to-billions/). ## How to use ColPali? With Morphik, using ColPali is as simple as adding a single `true/false` parameter to the `ingest_file` function and the query function. Here is what an example ingestion pathway looks like: ```python theme={null} from morphik import Morphik db = Morphik("YOUR-URI-HERE") db.ingest_file("report_with_images_and_charts.pdf", use_colpali=True) ``` Here is an example query pathway: ```python theme={null} db.query("At what time-step did we see the highest GDP growth rate?", use_colpali=True) ``` So instead of having to implement the ColPali pipeline from scratch, you can use Morphik to do it for you in a single line of code! ## Controlling Output Format When retrieving ColPali chunks (which are page images), you can control how the images are returned using the `output_format` parameter: ```python theme={null} # Return as base64-encoded data (default) chunks = db.retrieve_chunks("quarterly results", use_colpali=True) # Return as presigned URLs (useful for web UIs) chunks = db.retrieve_chunks("quarterly results", use_colpali=True, output_format="url") # Convert images to markdown text via OCR chunks = db.retrieve_chunks("quarterly results", use_colpali=True, output_format="text") ``` The three output formats are: * **`"base64"`** (default): Returns base64-encoded image data * **`"url"`**: Returns presigned HTTPS URLs, convenient for LLMs and UIs that accept remote image URLs * **`"text"`**: Converts page images to markdown text via OCR ### Choosing Between Formats **base64 vs url**: Both formats pass images to LLMs for visual understanding and produce similar inference results. However, `url` is lighter on network transfer since only the URL is sent to your application (the LLM fetches the image directly). This can result in faster response times, especially when working with multiple images. **When to use text**: Passing images to LLMs for inference can be slow and consume significant context tokens. Use `output_format="text"` when: * You need **faster inference** speeds * Your documents are **primarily text-based** (reports, articles, contracts) * You're hitting **context length limits** If you're experiencing context limit issues with image-based retrieval, it may be because images aren't being passed correctly to the model. See [Generating Completions with Retrieved Chunks](/cookbooks/generating-completions-with-retrieved-chunks) for examples of properly passing images (both base64 and URLs) to vision-capable models like GPT-4o. # Metadata Filtering Source: https://morphik.ai/docs/concepts/metadata-filtering Canonical reference for Morphik’s metadata filter DSL and typed comparisons. Morphik lets you filter documents and chunks directly in the database using a concise JSON filter syntax. The same structure powers the REST API, Python SDK (sync + async), folder helpers, and `UserScope`, so you can define a filter once and reuse it everywhere. Prefer server-side filters over client-side post-processing. You’ll reduce bandwidth, improve performance, and keep behavior consistent between endpoints. ## Where Filters Apply You can pass `filters` (or `document_filters`) to: * Retrieval endpoints: [`retrieve_chunks`](/python-sdk/retrieve_chunks), [`retrieve_docs`](/python-sdk/retrieve_docs), [`query`](/python-sdk/query), [`query_document`](/python-sdk/query_document) ingestion options. * Listing/management: [`list_documents`](/python-sdk/list_documents), document/folder analytics, chat history, and anywhere an SDK method exposes a `filters` argument. ## Quick Start ```python theme={null} from datetime import datetime from morphik import Morphik db = Morphik() filters = { "$and": [ {"department": {"$eq": "research"}}, {"priority": {"$gte": 40}}, {"start_date": {"$lte": datetime.now().isoformat()}}, {"tags": {"$contains": {"value": "contract"}}} ] } chunks = db.retrieve_chunks("project delta highlights", filters=filters, k=6) ``` ### Typed Metadata Typed comparisons (numbers, decimals, dates, datetimes) rely on `metadata_types`. Supply the per-field hints during ingest or metadata updates: ```python theme={null} doc = db.ingest_text( content="SOW for Delta", metadata={ "priority": 42, "start_date": "2024-01-15T12:30:00Z", "end_date": "2024-12-31", "cost": "1234.56" }, metadata_types={ "priority": "number", "start_date": "datetime", "end_date": "date", "cost": "decimal" } ) ``` If you omit a hint, Morphik infers one automatically for simple scalars, but explicitly declaring types is recommended for reliable range queries. ### DateTime and Timezone Behavior Morphik preserves your timezone format exactly as provided: | Input | Stored As | Notes | | ----------------------------------- | ----------------------------- | ----------------------------- | | `datetime(2024, 1, 15)` (naive) | `"2024-01-15T00:00:00"` | No timezone added | | `datetime(2024, 1, 15, tzinfo=UTC)` | `"2024-01-15T00:00:00+00:00"` | Timezone preserved | | `"2024-01-15T12:00:00Z"` (string) | `"2024-01-15T12:00:00+00:00"` | Z converted to +00:00 | | `1705312800` (UNIX timestamp) | `"2024-01-15T10:00:00+00:00"` | Timestamps are inherently UTC | **SDK Type Reconstruction:** When you retrieve a `Document` via the Python SDK, datetime/date/decimal values in `metadata` are automatically reconstructed to their Python types using the `metadata_types` hints. This means you get back what you put in: ```python theme={null} from datetime import datetime # Ingest with naive datetime doc = db.ingest_text("...", metadata={"created": datetime(2024, 1, 15)}) # Retrieve - metadata["created"] is a datetime object, not a string retrieved = db.get_document(doc.external_id) print(type(retrieved.metadata["created"])) # print(retrieved.metadata["created"].tzinfo) # None (still naive) ``` ### Mixed Timezone Formats **Morphik handles mixed formats correctly** - filtering and comparisons work even if some documents have naive datetimes and others have timezone-aware ones: ```python theme={null} from datetime import datetime, UTC # Mixed formats across documents - Morphik handles this fine db.ingest_text("Doc A", metadata={"ts": datetime(2024, 1, 15)}) # naive db.ingest_text("Doc B", metadata={"ts": datetime(2024, 6, 15, tzinfo=UTC)}) # aware # Filtering works correctly results = db.list_documents(filters={"ts": {"$gte": "2024-05-01"}}) # Returns Doc B ``` **Python comparisons fail with mixed formats.** If you retrieve mixed-format datetimes and compare them locally, Python raises `TypeError`: ```python theme={null} sorted([naive_dt, aware_dt]) # TypeError: can't compare offset-naive and offset-aware ``` **Recommendation:** Stay consistent - pick one format (preferably timezone-aware with UTC) and use it throughout. Let Morphik handle filtering rather than sorting in Python. ## Implicit vs Explicit Syntax * **Implicit equality** – Bare key/value pairs (`{"status": "active"}`) use JSON containment and are ideal for simple matching. They also check whether an array contains the value. * **Explicit operators** – Wrap a field in an operator object to unlock typed comparisons, set logic, regex, substring checks, etc. (`{"status": {"$ne": "archived"}}`). ## Operator Reference ### Equality & Comparison | Operator | Description | Example | | ---------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- | | `$eq` / implicit value | Equality (also matches scalars in arrays). | `{"status": {"$eq": "completed"}}` | | `$ne` | Not equal. | `{"status": {"$ne": "archived"}}` | | `$gt`, `$gte`, `$lt`, `$lte` | Greater/less-than comparisons for numbers, decimals, dates, datetimes, and strings (`$eq/$ne` only). Requires correct `metadata_types`. | `{"priority": {"$gte": 40}}`, `{"end_date": {"$lt": "2025-01-01"}}` | ### Set Membership | Operator | Description | Example | | -------- | -------------------------------------------- | -------------------------------------------------- | | `$in` | Matches any operand in the provided list. | `{"status": {"$in": ["completed", "processing"]}}` | | `$nin` | Matches when the value is *not* in the list. | `{"region": {"$nin": ["EU", "LATAM"]}}` | ### Type & Existence | Operator | Description | Example | | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------- | | `$exists` | Field must (or must not) exist. Accepts booleans or truthy strings. | `{"external_id": {"$exists": true}}` | | `$type` | Field must have one of the supported metadata types (`string`, `number`, `decimal`, `datetime`, `date`, `boolean`, `array`, `object`, `null`). | `{"start_date": {"$type": "datetime"}}` | ### String & Pattern Matching | Operator | Description | Example | | ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- | | `$contains` | Case-insensitive substring match by default; accepts `{ "value": "...", "case_sensitive": bool }`. Works on scalars and array entries. | `{"title": {"$contains": "Q4 Summary"}}` | | `$regex` | PostgreSQL regex match. Accepts a raw string pattern or `{ "pattern": "...", "flags": "i" }` (only the `i` flag is supported). Works on scalars and arrays. | `{"folder": {"$regex": {"pattern": "^fin", "flags": "i"}}}` | ### Logical Composition | Operator | Description | | -------- | ------------------------------------------------------ | | `$and` | All nested clauses must match (non-empty list). | | `$or` | At least one nested clause must match. | | `$nor` | None of the nested clauses may match (`NOT (A OR B)`). | | `$not` | Inverts a single clause. | Mix logical operators freely with field-level operators for complex expressions. ## Common Patterns ### Current Window Between Start/End ```json theme={null} { "$and": [ {"start_date": {"$lte": "2024-06-01T00:00:00Z"}}, {"end_date": {"$gte": "2024-06-01T00:00:00Z"}} ] } ``` ### Folder/User Scope plus Metadata ```python theme={null} folder = db.get_folder("legal") scoped = folder.signin("user-42") filters = {"priority": {"$gte": 50}} response = scoped.list_documents(filters=filters, include_total_count=True) ``` ### Array Membership & Substring ```json theme={null} { "$and": [ {"tags": {"$contains": {"value": "contract"}}}, {"tags": {"$regex": {"pattern": "quarter", "flags": "i"}}} ] } ``` ## Troubleshooting * **“Unsupported metadata filter operator …”** – Double-check spelling and operand type (lists for `$in`, non-empty arrays for `$and`, etc.). * **“Metadata field … expects type …”** – The server couldn’t coerce the operand to the declared type. Ensure numbers/dates are valid JSON scalars or native Python types before serialization. * **Range query returns nothing** – Confirm the target documents were ingested/updated with the corresponding `metadata_types`. Re-ingest or call `update_document_metadata` with the proper type hints if necessary. Still stuck? Share your filter payload and endpoint at `founders@morphik.ai` or on [Discord](https://discord.com/invite/BwMtv3Zaju). # Introduction to RAG Source: https://morphik.ai/docs/concepts/naive-rag An overview of Retrieval Augmented Generation with Vector Similarity Search ## What is RAG? RAG stands for **R**etrieval **A**ugmented **G**eneration. It is a set of tools and techniques that allow us to provide additional context to an LLM given a query. For example, let's say you're creating an app that helps users assemble furniture. If the user is stuck on a particular step, they may want to upload a picture of their current progress and ask follow-up questions to a chatbot. RAG would allow you to take that picture, as well as the user query, and then search over a pre-ingested knowledge base - a set of user manuals for different types of furniture, for instance - and *augment* the query with context from that knowledge base. So, instead of the response being > "oh, it looks like you've screwed the rear leg backwards" it would be something like > "seems like you're assembling chair CX-184. You may have skipped step 8 in the assembly process, since the rear leg is screwed backwards. Here is a step-by-step solution from the assembly guide: ...". Note how both answers recognized the issue correctly, but since the LLM had additional context in the second answer, it was also able to provide a solution and more specific details. That's the gist of RAG - LLMs provide **higher-quality responses** when provided with **more context** surrounding a query. While the core concept itself is quite obvious, the complexity arises in *how* we can effectively retrieve the correct information. In the following sections, we explain one way to effectively perform RAG based on the concept of vector embeddings and similarity search (we'll explain what these mean!). In reality, Morphik uses a combination of different RAG techniques to achieve the best solution. In this explainer, we’ll restrict ourselves to single vector-search based retrieval. ## How does RAG work? RAG roughly consists of 4 actions: i) ingesting knowledge - such as code documentation, textbooks, or product catalogues - ii) *retrieving* relevant chunks from said knowledge, iii) using the retrieved information to *augment* the user query into a better prompt, and iv) *generating* a model response from the prompt. ### Ingest In order to help add context to a prompt, we first need that context to exist. This is what ingestion helps with. Ingestion is the process of converting your knowledge base into a format that's optimized for retrieval. This typically involves three key steps: chunking, embedding, and indexing. **Chunking** involves breaking down documents into smaller, manageable pieces. While LLMs have context windows that can handle thousands of tokens, we want to retrieve only the most relevant information for a given query. Chunking strategies vary based on the content type - code documentation might be chunked by function or class, while textbooks might be chunked by section or paragraph. The ideal chunk size balances granularity (smaller chunks for precise retrieval) with context preservation (larger chunks for maintaining semantic meaning). **Embedding** transforms these text chunks into vector representations - essentially converting semantic meaning into mathematical space. This is done using embedding models that distill the essence of text into dense vectors. The [math and ML behind embeddings](https://www.3blue1brown.com/lessons/gpt#embedding) is really interesting. They have a [long history](https://en.wikipedia.org/wiki/Word_embedding) of development - with origins as old as 1957. Over time, models that produce word embeddings have gone through multiple iterations - different domains, novel neural network architectures, as well as different training paradigms. Here's a gif we made using [Manim](https://www.manim.community/) to explain word embeddings: Animation showing how word embeddings work Now, if the embeddings space encodes meaning, it is reasonable to assume that words or text that mean similar things have embeddings that are close to each other. That is, *the closer the embeddings are to each other, the more similar the "embedees"*. This idea is the foundation of retrieval. Once we have these embeddings, we can save them in a vector store. When a query comes in, we can embed it and find the chunks that are the closest to it. ### Retrieve A natural question is, how do we quantify "closeness"? This ends up being the cosine distance. There's more information about this - i.e. why we use cosine distance - in the dropdown below. We could consider the distance between two embeddings. However, if we want direct similarities scores, we'd have to perform some kind of inverse transformation (if the distance is small, they are actually more similar). More importantly, regular distance computation suffers from the [curse of dimensionality](https://en.wikipedia.org/wiki/Curse_of_dimensionality). Instead, we could look at the *dot product* of two vectors instead. The closer two vectors are to each other, the higher their dot product value. However, since the dot product of two vectors is proportional to the length of the vectors, this also means that a larger vector (i.e. a vector whose magnitude is high) will be the most similar to other vectors. As a result, we can divide the dot product with the lengths of both vectors, obtaining the cosine distance. Computing cosine distance is expensive. So, in practice, most embedding models provide *normalized* embeddings (i.e. embeddings with length 1). Then, computing the dot product is the same as computing the cosine distance between two vectors. So, upto this point, we've taken documents, separated them into chunks, and embedded each chunk into a vector. We've also discussed how to compute vectors that are close to each other. As a result ...we now have a practical way to retrieve relevant context. When a user query arrives, it's first transformed into an embedding vector. Next, the retrieval system performs a similarity search in our vector database to find the most relevant chunks—these are the vectors closest in semantic meaning to the query. The number of retrieved chunks can vary, typically between three to ten, depending on the application's requirements and the desired comprehensiveness of context. ### Augment With relevant chunks in hand, we now move to the augmentation step. Here, the original user query is combined with the retrieved chunks to construct an enriched prompt. This augmented prompt provides the LLM with additional context, guiding it toward more precise, accurate, and relevant responses. For example, if our furniture-assembly app receives the query: > "Why does the chair feel unstable?" The retrieval might return chunks like: * "Ensure that the screws (B3) from step 5 are tightened completely." * "Instability may result if the cross-bar (part D) from step 7 is incorrectly positioned." The augmented prompt passed to the LLM could be: > "User query: 'Why does the chair feel unstable?' Context from manual: 'Ensure screws (B3) from step 5 are tightened completely. Instability may result if the cross-bar (part D) from step 7 is incorrectly positioned.'" This detailed context enables the LLM to generate a highly informed, actionable response. ### Generate Finally, the augmented prompt is passed to the LLM to generate the final answer. Because the LLM has access to highly relevant context, its responses are significantly more informative and actionable. Continuing our previous example, the response could be: > "The chair's instability is likely caused by loose screws (B3) from step 5 or an incorrectly positioned cross-bar (part D). Verify these areas, tightening screws fully and checking that part D matches the orientation shown in step 7 of the assembly manual." This demonstrates how RAG improves the response quality by leveraging external knowledge effectively. ## Putting it all together To summarize, RAG leverages four key steps—ingestion, retrieval, augmentation, and generation—to significantly enhance the quality of LLM-generated responses. By converting knowledge bases into vector embeddings, performing efficient similarity searches, and providing detailed context to LLMs, RAG allows applications to deliver precise, context-rich interactions. In future articles, we'll delve deeper into specific RAG techniques, discuss optimization strategies for vector search, and explore how combining multiple retrieval methods can further enhance application performance. # User and Folder Scoping Source: https://morphik.ai/docs/concepts/user-folder-scoping Organizing data with user and folder scoping in Morphik Morphik provides powerful mechanisms to organize and isolate your data through **user scoping** and **folder scoping**. These features allow you to create logical boundaries for different projects or user groups while maintaining a unified database. ## Folder Scoping Folders in Morphik allow you to organize documents into logical groups, similar to directories in a file system, but for your unstructured data. Operations performed within a folder scope only affect documents within that folder. ### When to Use Folder Scoping * **Project Organization**: Separate documents by project, department, or purpose * **Data Categorization**: Group similar documents together * **Access Control**: Create logical boundaries for different document sets ### Creating and Using Folders ```python theme={null} from morphik import Morphik with Morphik() as db: # Create or get a folder folder = db.create_folder("project_x") # or folder = db.get_folder("project_x") # Operations are scoped to this folder doc = folder.ingest_text("This document belongs to Project X") chunks = folder.retrieve_chunks("project") docs = folder.list_documents() # Only lists documents in this folder ``` ## User Scoping User scoping allows multi-tenant applications to isolate data per end user, ensuring each user only sees their own documents. This is particularly useful for applications where privacy between users is important. ### When to Use User Scoping * **Multi-tenant Applications**: Keep each user's data separate * **Privacy Requirements**: Ensure users can only access their own documents * **Personal Data Spaces**: Create user-specific knowledge bases ### Creating and Using User Scopes ```python theme={null} from morphik import Morphik with Morphik() as db: # Create a user scope user_scope = db.signin("user123") # Operations are scoped to this user doc = user_scope.ingest_text("This belongs to user123 only") docs = user_scope.list_documents() # Only lists documents for this user completion = user_scope.query("What documents do I have?") # Only searches user123's documents ``` ## Combined User and Folder Scoping For the maximum level of organization, you can combine both scopes to organize documents by both user and folder. ```python theme={null} from morphik import Morphik with Morphik() as db: # First get a folder folder = db.get_folder("project_x") # Then scope to a specific user within that folder user_folder_scope = folder.signin("user123") # Operations are scoped to both the folder and user doc = user_folder_scope.ingest_text("This belongs to user123 in project_x") chunks = user_folder_scope.retrieve_chunks("project") docs = user_folder_scope.list_documents() # Only lists user123's documents in project_x ``` ## Asynchronous Usage All scoping features are also available with the asynchronous client: ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: # Create folder and user scopes folder = db.get_folder("project_x") user_scope = db.signin("user123") combined_scope = folder.signin("user123") # Use scoped operations asynchronously doc = await combined_scope.ingest_text("Document for user123 in project_x") results = await combined_scope.retrieve_docs("project") ``` ## Important Considerations * All methods available on the main Morphik client are also available on folder and user scopes * Operations performed within a scope are isolated to that scope * Documents created within a scope are only accessible within that scope, unless explicitly queried with appropriate filters * Scopes can be used for both reading and writing operations * User and folder information is stored as metadata with the documents, so you can still filter across scopes with explicit filter parameters if needed ## Filtering by Folder Name in Metadata When you ingest documents in a folder with the `folder_name` parameter, that value is automatically available in the document's metadata for filtering. This enables powerful cross-folder queries using Morphik's metadata filter operators: ```python theme={null} from morphik import Morphik db = Morphik() # Filter documents from multiple folders filters = { "folder_name": {"$in": ["legal", "hr", "finance"]} } docs = db.list_documents(filters=filters) # Combine folder filtering with other metadata filters = { "$and": [ {"folder_name": {"$regex": {"pattern": "^project_", "flags": "i"}}}, {"status": "active"}, {"priority": {"$gte": 70}} ] } response = db.query("What are the high-priority project updates?", filters=filters) # Exclude specific folders filters = { "$and": [ {"folder_name": {"$nin": ["archived", "drafts"]}}, {"created_date": {"$gte": "2024-01-01"}} ] } chunks = db.retrieve_chunks("quarterly report", filters=filters, k=10) ``` This approach is useful when you need to: * Query across multiple folders simultaneously * Use pattern matching on folder names * Combine folder filters with complex metadata conditions * Build dynamic queries where folder selection isn't known at scope creation time For more filtering examples, see the [Complex Metadata Filtering](/cookbooks/complex-metadata-filtering) cookbook and [Metadata Filtering](/concepts/metadata-filtering) reference. ## Use Cases ### Multi-Project Research Team A research team working on multiple projects can use folder scoping to keep document sets separate: ```python theme={null} # Project A research project_a = db.get_folder("project_a") doc = project_a.ingest_file("project_a_results.pdf") # Project B research (completely separate) project_b = db.get_folder("project_b") doc = project_b.ingest_file("project_b_results.pdf") ``` ### Multi-tenant Application An application serving multiple end users can use user scoping to keep each user's data private: ```python theme={null} # User 1's personal data user1 = db.signin("user1") doc = user1.ingest_text("My private notes") # User 2's personal data (completely separate) user2 = db.signin("user2") doc = user2.ingest_text("My confidential information") ``` ### Enterprise Knowledge Management For complex enterprise setups, combine both scopes to organize by both department and individual: ```python theme={null} # Marketing department marketing = db.get_folder("marketing") # Individual marketers' workspaces alice_marketing = marketing.signin("alice") bob_marketing = marketing.signin("bob") # Engineering department engineering = db.get_folder("engineering") # Individual engineers' workspaces charlie_engineering = engineering.signin("charlie") dave_engineering = engineering.signin("dave") ``` # Configure Morphik Source: https://morphik.ai/docs/configuration Exploring the tunable knobs via `morphik.toml` Morphik uses the `morphik.toml` configuration file to control all aspects of the system. ## Model Configuration Morphik uses [LiteLLM](https://litellm.ai) to route to 100+ LLM providers with a unified interface. This means you can use models from OpenAI, Anthropic, Google, AWS Bedrock, Azure, Hugging Face, and many more - all with the same simple configuration format. ### Example Configurations In your `morphik.toml`, define models in the LiteLLM format: ```toml theme={null} [registered_models] # OpenAI openai_gpt4 = { model_name = "gpt-4" } openai_gpt4_mini = { model_name = "gpt-4-0125-preview" } # Anthropic claude_3_opus = { model_name = "claude-3-opus-20240229" } claude_3_sonnet = { model_name = "claude-3-sonnet-20240229" } # Google gemini_pro = { model_name = "gemini/gemini-pro" } gemini_flash = { model_name = "gemini/gemini-1.5-flash" } # Azure OpenAI azure_gpt4 = { model_name = "azure/gpt-4", api_base = "YOUR_AZURE_URL", api_key = "YOUR_KEY" } # AWS Bedrock bedrock_claude = { model_name = "bedrock/anthropic.claude-v2" } # And 100+ more providers... ``` Then reference these models throughout your configuration: ```toml theme={null} [completion] model = "claude_3_opus" # Use any registered model [embedding] model = "openai_embedding" # Use any registered embedding model ``` ## Local LLMs Morphik can also run entirely with local LLMs. We directly integrate with two major local LLM servers: ### Ollama Ollama [Ollama](https://ollama.ai) - Run Llama, Mistral, Gemma and other models locally. ```toml theme={null} [registered_models] # Ollama models ollama_llama = { model_name = "ollama_chat/llama3.2", api_base = "http://localhost:11434" } ollama_qwen_vision = { model_name = "ollama_chat/qwen2.5:72b", api_base = "http://localhost:11434", vision = true } ollama_embedding = { model_name = "ollama/nomic-embed-text", api_base = "http://localhost:11434" } ``` ### 🍋 Lemonade [Lemonade Server](https://lemonade-server.ai/) - Optimized local LLM server for AMD GPUs and NPUs. ```toml theme={null} [registered_models] # Lemonade models lemonade_qwen = { model_name = "openai/Qwen2.5-VL-7B-Instruct-GGUF", api_base = "http://localhost:8020/api/v1", vision = true } lemonade_embedding = { model_name = "openai/nomic-embed-text-v1-GGUF", api_base = "http://localhost:8020/api/v1" } ``` ### Docker Deployments When running Morphik in Docker: * Local services: Use `http://host.docker.internal:PORT` * Both in Docker: Use container names (e.g., `http://ollama:11434`) ## Need Help? 1. Join our [Discord community](https://discord.com/invite/BwMtv3Zaju) 2. Check [GitHub](https://github.com/morphik-org/morphik-core) for issues # Agent Workflows Source: https://morphik.ai/docs/cookbooks/agent-workflows Practical examples of using the Morphik Agent for complex analysis tasks This page focuses on a single pattern: wiring Morphik retrieval tools into your own agent loop. ## Build a Retrieval Agent with Morphik Tools Use Morphik's retrieval APIs as function-calling tools when you want a lightweight agent loop you fully control. The example below uses OpenAI's Responses API, `list_documents` for discovery, and `retrieve_chunks` (with ColPali image URLs) for grounded answers. > Prerequisites: `pip install morphik openai`, set `MORPHIK_URI` and `OPENAI_API_KEY`, and update the metadata fields in `SYSTEM_INSTRUCTIONS` to match your corpus. Giving the LLM direct access to retrieval tools lets it decide when to explore vs. fetch, narrow scope with filters, and request only the needed pages. This example exposes two tools, but you can add more (e.g., page-range or download helpers) from the [API reference](/api-reference/getting-started) to expand the agent’s toolkit while keeping guardrails tight. You can use any major LLM with tool-calling support (OpenAI, Anthropic, Gemini, etc.); we show OpenAI for familiarity and will add provider-specific snippets soon—the core loop remains the same. Imports used in the snippets below: ```python theme={null} import argparse import json import os from morphik import Morphik from openai import OpenAI ``` ### 1) System prompt Use this to govern the agent’s behavior, metadata awareness, and citation discipline; it is the primary control surface for how the loop plans and answers. ```python theme={null} SYSTEM_INSTRUCTIONS = """You are a helpful assistant with access to a curated knowledge base. Available metadata fields (customize to your schema): - doc_id (string) - doc_type (string) - title (string) - category (string) - status (string) - effective_date (datetime) - expires_at (datetime) - tags (array) - region (string) Filter operators: - $eq, $gt, $gte, $lt, $lte, $and, $contains, $regex Strategy: 1) Call list_documents to see available material and metadata. 2) Apply filters before retrieval when date, status, region, or tags matter. 3) Use retrieve_chunks with use_colpali=True and output_format="url" to pull text and images. 4) Cite sources with doc_id (or your ID field), filename, and page numbers. """ ``` ### 2) Tool schemas (OpenAI Responses API) Expose only the tools you want the model to call. The schema format may differ slightly by provider, but the tool list and arguments stay consistent. ```python theme={null} TOOLS = [ { "type": "function", "name": "list_documents", "description": "List documents in the knowledge base with optional metadata filters.", "parameters": { "type": "object", "properties": { "filters": { "type": ["string", "null"], "description": "Metadata filters as a JSON string. Pass null or empty for no filters." }, "limit": { "type": ["integer", "null"], "description": "Maximum number of documents to return (default: 10)" } }, "required": ["filters", "limit"], "additionalProperties": False }, "strict": True }, { "type": "function", "name": "retrieve_chunks", "description": "Retrieve relevant document chunks for a query using semantic search.", "parameters": { "type": "object", "properties": { "query": { "type": "string", "description": "Search query or user question" }, "filters": { "type": ["string", "null"], "description": "Optional metadata filters as a JSON string" }, "k": { "type": ["integer", "null"], "description": "Number of chunks to retrieve (default: 3)" } }, "required": ["query", "filters", "k"], "additionalProperties": False }, "strict": True } ] ``` ### 3) Tool implementations (Morphik Python SDK) These Python functions bridge the LLM’s tool calls to Morphik APIs and normalize outputs (metadata, previews, image URLs) for reuse in the loop. ```python theme={null} def list_documents(morphik: Morphik, filters=None, limit=None): if isinstance(filters, str) and filters.strip(): filters = json.loads(filters) response = morphik.list_documents(filters=filters, limit=limit or 10) docs = response.documents if hasattr(response, "documents") else response normalized = [] for doc in docs: meta = doc.metadata or {} normalized.append({ "external_id": doc.external_id, "filename": doc.filename, "doc_id": meta.get("doc_id"), "title": meta.get("title"), "doc_type": meta.get("doc_type"), "category": meta.get("category"), "status": meta.get("status"), "effective_date": str(meta.get("effective_date")) if meta.get("effective_date") else None, "expires_at": str(meta.get("expires_at")) if meta.get("expires_at") else None, "tags": meta.get("tags") or [], }) return {"documents": normalized, "count": len(normalized)} def retrieve_chunks(morphik: Morphik, query, filters=None, k=None): if isinstance(filters, str) and filters.strip(): filters = json.loads(filters) chunks = morphik.retrieve_chunks( query=query, filters=filters, k=k or 3, use_colpali=True, output_format="url" ) results = [] image_urls = [] for chunk in chunks: meta = chunk.metadata or {} is_image_url = isinstance(chunk.content, str) and chunk.content.startswith("http") if is_image_url: image_urls.append(chunk.content) preview = chunk.content if isinstance(preview, str) and len(preview) > 300: preview = f"{preview[:300]}..." if is_image_url: preview = f"[Image URL: {chunk.content}]" results.append({ "score": round(chunk.score, 3), "content_preview": preview, "filename": meta.get("filename") or chunk.filename or "unknown", "doc_id": meta.get("doc_id"), "doc_type": meta.get("doc_type"), "title": meta.get("title"), "expires_at": str(meta.get("expires_at")) if meta.get("expires_at") else None, "tags": meta.get("tags") or [], }) return { "chunks": results, "count": len(results), "image_urls": image_urls } ``` ### 4) Agent runner (OpenAI Responses loop) Handles the model call, executes tool invocations, and re-feeds results until the model returns a final answer. Swap the client/model for any tool-calling LLM and keep the structure. ```python theme={null} def run_agent(morphik: Morphik, openai_client: OpenAI, user_query: str, model: str): conversation = [{"role": "user", "content": user_query}] for _ in range(5): response = openai_client.responses.create( model=model, instructions=SYSTEM_INSTRUCTIONS, tools=TOOLS, input=conversation ) conversation += response.output calls = [item for item in response.output if item.type == "function_call"] if not calls: print(response.output_text) return for call in calls: args = json.loads(call.arguments) if call.name == "list_documents": result = list_documents( morphik, filters=args.get("filters"), limit=args.get("limit") ) image_urls = [] elif call.name == "retrieve_chunks": result = retrieve_chunks( morphik, query=args["query"], filters=args.get("filters"), k=args.get("k") ) image_urls = result.get("image_urls", []) else: result = {"error": f"Unknown function: {call.name}"} image_urls = [] conversation.append({ "type": "function_call_output", "call_id": call.call_id, "output": json.dumps(result) }) if image_urls: image_content = [{"type": "input_text", "text": "Here are the retrieved pages:"}] for url in image_urls: image_content.append({"type": "input_image", "image_url": url}) conversation.append({"role": "user", "content": image_content}) print("[Warning] Reached maximum iterations without a final answer") ``` ### 5) Optional CLI entry point ```python theme={null} def main(): parser = argparse.ArgumentParser(description="Retrieval agent with Morphik tools") parser.add_argument("--uri", default=os.getenv("MORPHIK_URI")) parser.add_argument("--query", default="What are the latest policy updates published this year?") parser.add_argument("--openai-key", default=os.getenv("OPENAI_API_KEY")) parser.add_argument("--model", default="gpt-4o-mini") args = parser.parse_args() if not args.uri: raise SystemExit("MORPHIK_URI not set. Use --uri or set MORPHIK_URI.") if not args.openai_key: raise SystemExit("OPENAI_API_KEY not set. Use --openai-key or set OPENAI_API_KEY.") morphik = Morphik(args.uri, timeout=30) openai_client = OpenAI(api_key=args.openai_key) run_agent(morphik, openai_client, args.query, args.model) if __name__ == "__main__": main() ``` Run it with your own question: ```bash theme={null} export MORPHIK_URI="morphik://your-token@your-host" export OPENAI_API_KEY="your-openai-key" python retrieval_agent.py --query "Where can I find the most recent safety policies?" ``` To extend this loop, add more Morphik tools (e.g., page-range retrieval or download helpers) from the [API reference](/api-reference/getting-started) into `TOOLS` and implement matching handlers in the runner. This retrieval agent pattern shows how to keep full control over tool usage while grounding responses with Morphik. The agent's ability to plan, execute tools, and remember context makes it ideal for sophisticated business intelligence and research workflows. # API Basic Operations Source: https://morphik.ai/docs/cookbooks/api-basic-operations End-to-end walkthrough of ingestion, retrieval, and LLM integration using the Morphik REST API. This cookbook walks through the core Morphik workflow using the REST API—multimodal ingestion for high-accuracy retrieval, text ingestion for OCR-driven chunks, and integrating with your own LLM. > **Prerequisites** > > * Morphik API endpoint (e.g., `http://localhost:8000` or `https://api.morphik.ai`) > * API credentials via `MORPHIK_API_KEY` (if authentication is enabled) > * For the optional OpenAI example, set `OPENAI_API_KEY` ## 1. Environment setup ```bash theme={null} export MORPHIK_API_URL="https://api.morphik.ai" export MORPHIK_API_KEY="your-api-key" export QUESTION="What are the key takeaways from the uploaded document?" ``` ## 2. Multimodal ingestion (ColPali) This path indexes the original file contents directly, yielding higher accuracy for scanned pages, tables, and images. ```bash theme={null} # Ingest a document with ColPali (multimodal) curl -X POST "${MORPHIK_API_URL}/ingest/file" \ -H "Authorization: Bearer ${MORPHIK_API_KEY}" \ -F "file=@/path/to/document.pdf" \ -F 'metadata={"demo_variant": "multimodal"}' \ -F "use_colpali=true" ``` **Response:** ```json theme={null} { "external_id": "doc-123abc", "filename": "document.pdf", "status": "processing", "metadata": {"demo_variant": "multimodal"} } ``` ### Wait for processing to complete ```bash theme={null} # Check document status curl -X GET "${MORPHIK_API_URL}/documents/doc-123abc/status" \ -H "Authorization: Bearer ${MORPHIK_API_KEY}" ``` **Response when ready:** ```json theme={null} { "status": "completed", "document_id": "doc-123abc", "chunks_created": 15 } ``` ## 3. Text ingestion (OCR + chunking) This path OCRs the document before chunking it, making the text immediately available for retrieval. ```bash theme={null} # Ingest without ColPali (standard text extraction) curl -X POST "${MORPHIK_API_URL}/ingest/file" \ -H "Authorization: Bearer ${MORPHIK_API_KEY}" \ -F "file=@/path/to/document.pdf" \ -F 'metadata={"demo_variant": "standard"}' \ -F "use_colpali=false" ``` ## 4. Query with Morphik completion Generate a completion directly using Morphik's configured LLM. ### Multimodal query (ColPali) ```bash theme={null} curl -X POST "${MORPHIK_API_URL}/query" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${MORPHIK_API_KEY}" \ -d '{ "query": "What are the key takeaways from the uploaded document?", "use_colpali": true, "k": 4, "filters": {"demo_variant": "multimodal"} }' ``` **Response:** ```json theme={null} { "completion": "The key takeaways from the document are...", "usage": { "prompt_tokens": 1250, "completion_tokens": 150, "total_tokens": 1400 }, "sources": [ { "document_id": "doc-123abc", "chunk_number": 3, "score": 0.89 } ] } ``` ### Text query (standard) ```bash theme={null} curl -X POST "${MORPHIK_API_URL}/query" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${MORPHIK_API_KEY}" \ -d '{ "query": "What are the key takeaways?", "use_colpali": false, "k": 4, "filters": {"demo_variant": "standard"} }' ``` ## 5. Query with system prompt override Override the default system prompt to customize the LLM's behavior and response style. ```bash theme={null} curl -X POST "${MORPHIK_API_URL}/query" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${MORPHIK_API_KEY}" \ -d '{ "query": "What are the key takeaways?", "k": 4, "filters": {"demo_variant": "multimodal"}, "prompt_overrides": { "query": { "system_prompt": "You are a legal expert. Analyze the document and provide insights in formal legal language. Always cite specific sections when making claims." } } }' ``` **Example with different persona:** ```bash theme={null} # Pirate assistant example curl -X POST "${MORPHIK_API_URL}/query" \ -H "Content-Type: application/json" \ -d '{ "query": "What is this document about?", "k": 3, "prompt_overrides": { "query": { "system_prompt": "You are a pirate assistant. Always respond in pirate speak with arrr and matey! Be entertaining while staying accurate to the document content." } } }' ``` **Response:** ```json theme={null} { "completion": "Arrr, me hearty! This here document be talkin' about...", "usage": {"prompt_tokens": 1100, "completion_tokens": 85, "total_tokens": 1185} } ``` ## 6. Retrieve chunks for your own LLM For production workloads, retrieve Morphik's curated chunks and forward them to your preferred LLM. This gives you full control over prompts, orchestration, and rate limits. ### Step 1: Retrieve relevant chunks ```bash theme={null} curl -X POST "${MORPHIK_API_URL}/retrieve/chunks" \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${MORPHIK_API_KEY}" \ -d '{ "query": "What are the key takeaways?", "use_colpali": true, "k": 4, "padding": 1, "filters": {"demo_variant": "multimodal"} }' ``` **Response:** ```json theme={null} [ { "document_id": "doc-123abc", "chunk_number": 3, "content": "The key findings indicate...", "score": 0.89, "download_url": "https://storage.morphik.ai/chunks/doc-123abc-3.png", "content_type": "image/png" }, { "document_id": "doc-123abc", "chunk_number": 7, "content": "Additional analysis shows...", "score": 0.82, "download_url": null, "content_type": "text/plain" } ] ``` ### Step 2: Forward to your LLM (OpenAI example) #### Text-only chunks with OpenAI ```bash theme={null} # Extract text content from chunks TEXT_CONTEXT=$(curl -s -X POST "${MORPHIK_API_URL}/retrieve/chunks" \ -H "Content-Type: application/json" \ -d '{ "query": "What are the key takeaways?", "use_colpali": false, "k": 4, "filters": {"demo_variant": "standard"} }' | jq -r '.[] | "Source #\(.chunk_number):\n\(.content)"' | tr '\n' ' ') # Send to OpenAI curl https://api.openai.com/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer ${OPENAI_API_KEY}" \ -d "{ \"model\": \"gpt-4o-mini\", \"messages\": [ { \"role\": \"system\", \"content\": \"You are a helpful assistant. Use only the provided context to answer questions.\" }, { \"role\": \"user\", \"content\": \"Context: ${TEXT_CONTEXT}\n\nQuestion: ${QUESTION}\" } ] }" ``` #### Multimodal chunks with OpenAI (vision) ```bash theme={null} # Retrieve multimodal chunks with images CHUNKS=$(curl -s -X POST "${MORPHIK_API_URL}/retrieve/chunks" \ -H "Content-Type: application/json" \ -d '{ "query": "What are the key takeaways?", "use_colpali": true, "k": 4, "filters": {"demo_variant": "multimodal"} }') # Build multimodal content (requires jq processing for proper JSON) # For simplicity, here's a Python example: cat > query_openai.py << 'EOF' import os import json import requests chunks = requests.post( f"{os.environ['MORPHIK_API_URL']}/retrieve/chunks", json={ "query": os.environ["QUESTION"], "use_colpali": True, "k": 4, "filters": {"demo_variant": "multimodal"} } ).json() # Build multimodal message content content = [{"type": "text", "text": f"Answer using these sources.\n\nQuestion: {os.environ['QUESTION']}"}] for chunk in chunks: if chunk.get("download_url") and chunk.get("content_type", "").startswith("image/"): content.append({ "type": "image_url", "image_url": {"url": chunk["download_url"]} }) # Send to OpenAI response = requests.post( "https://api.openai.com/v1/chat/completions", headers={"Authorization": f"Bearer {os.environ['OPENAI_API_KEY']}"}, json={ "model": "gpt-4o", "messages": [{"role": "user", "content": content}] } ) print(response.json()["choices"][0]["message"]["content"]) EOF python query_openai.py ``` ### Step 3: Using other LLM providers The same pattern works with any LLM provider. Morphik handles retrieval and chunking; you control the completion: **Anthropic Claude:** ```bash theme={null} curl https://api.anthropic.com/v1/messages \ -H "x-api-key: ${ANTHROPIC_API_KEY}" \ -H "anthropic-version: 2023-06-01" \ -H "content-type: application/json" \ -d "{ \"model\": \"claude-3-5-sonnet-20241022\", \"max_tokens\": 1024, \"messages\": [{ \"role\": \"user\", \"content\": \"Context: ${TEXT_CONTEXT}\n\nQuestion: ${QUESTION}\" }] }" ``` **Google Gemini:** ```bash theme={null} curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-pro:generateContent?key=${GOOGLE_API_KEY}" \ -H "Content-Type: application/json" \ -d "{ \"contents\": [{ \"parts\": [{ \"text\": \"Context: ${TEXT_CONTEXT}\n\nQuestion: ${QUESTION}\" }] }] }" ``` ## 7. Advanced: Custom prompt template You can also override the prompt template that formats the context and question: ```bash theme={null} curl -X POST "${MORPHIK_API_URL}/query" \ -H "Content-Type: application/json" \ -d '{ "query": "What are the revenue figures?", "k": 4, "prompt_overrides": { "query": { "system_prompt": "You are a financial analyst. Provide precise numerical answers.", "prompt_template": "Based on the following financial data:\n\n{context}\n\nAnalyze and answer: {question}\n\nProvide specific numbers and percentages where available." } } }' ``` ## Summary Morphik's REST API provides flexible integration options: 1. **Managed completions**: Use Morphik's configured LLM with optional system prompt overrides 2. **Bring your own LLM**: Retrieve curated chunks and forward to any LLM provider 3. **Multimodal support**: Handle both text and visual content seamlessly 4. **Full control**: Override system prompts, prompt templates, and completion parameters This separation of concerns lets you focus on your application logic while Morphik handles high-quality retrieval and chunking. # Complex Metadata Filtering Source: https://morphik.ai/docs/cookbooks/complex-metadata-filtering Advanced document filtering using dates, arrays, decimals, and multiple operators for precise retrieval. This cookbook demonstrates Morphik's advanced metadata filtering capabilities with rich typed metadata fields including dates, decimals, booleans, arrays, and nested objects. > **Prerequisites** > > * Install the Morphik SDK: `pip install morphik` > * Provide credentials via Morphik URI > * Basic understanding of document ingestion ## 1. Ingest Documents with Rich Typed Metadata Morphik supports various metadata types for sophisticated filtering: ```python theme={null} from datetime import date, datetime, timezone from decimal import Decimal from morphik import Morphik client = Morphik("morphik://your-app:token@api.morphik.ai") # Rich metadata with multiple types metadata = { # Strings "region": "andes", "project_code": "hydro-life-2024", # Dates and datetimes "fieldwork_date": date(2024, 9, 18), "monitoring_window_start": datetime(2024, 9, 18, 9, 10, tzinfo=timezone.utc), "monitoring_window_end": datetime(2024, 9, 18, 17, 35, tzinfo=timezone.utc), # Numbers "hazard_score": 41, # Integer "ph_reading": Decimal("6.3"), # Decimal (precise) "water_depth_cm": 12.4, # Float "samples_collected": 18, # Boolean "is_priority_site": True, # Arrays "tags": ["wildlife", "flood-risk", "community"], # Nested objects "sensor_loadout": { "drone": "Skydio X10", "camera": "multispectral", "thermal_gain": 0.43, }, } # Ingest document with metadata doc = client.ingest_text( content="Laguna Amazonas boardwalk inspection for wetlands buffers...", filename="laguna-amazonas-field-brief.md", metadata=metadata, use_colpali=True, ) # Wait for completion doc.wait_for_completion(timeout_seconds=150) print(f"Ingested: {doc.external_id}") ``` ## 2. Build Complex Filters Combine multiple operators to create sophisticated queries: ```python theme={null} from datetime import date # Complex filter with multiple conditions filters = { "$and": [ # Exact match {"project_code": {"$eq": "hydro-life-2024"}}, # Array membership {"region": {"$in": ["andes"]}}, # Date range (>= September 15, 2024) {"fieldwork_date": {"$gte": date(2024, 9, 15).isoformat()}}, # Number range (<= 45) {"hazard_score": {"$lte": 45}}, # Boolean match {"is_priority_site": True}, # Array contains value {"tags": {"$contains": {"value": "wildlife"}}}, # Decimal comparison {"ph_reading": {"$lte": "6.5"}}, ] } ``` ### Filtering by Folder Name Documents ingested with a `folder_name` parameter can be filtered using that value in metadata. This enables cross-folder queries and pattern matching: ```python theme={null} # Filter specific folder filters = {"folder_name": "reports"} # Query multiple folders filters = { "folder_name": {"$in": ["reports", "invoices", "contracts"]} } # Exclude archived folders filters = { "folder_name": {"$nin": ["archived", "drafts", "test"]} } # Pattern matching on folder names filters = { "folder_name": {"$regex": {"pattern": "^project_", "flags": "i"}} } # Combine folder with other metadata filters = { "$and": [ {"folder_name": {"$in": ["legal", "compliance"]}}, {"priority": {"$gte": 70}}, {"status": "active"}, {"year": 2024} ] } ``` ## 3. List Documents with Filters Find documents matching your criteria: ```python theme={null} # Query documents with filters response = client.list_documents( filters=filters, include_total_count=True, completed_only=True ) print(f"\nFound {response.total_count} matching documents:") for doc in response.documents: print(f"- {doc.filename}") print(f" Hazard Score: {doc.metadata.get('hazard_score')}") print(f" Tags: {doc.metadata.get('tags')}") ``` ## 4. Retrieve Chunks with Filters Get document chunks that match your metadata filters: ```python theme={null} # Retrieve filtered chunks chunks = client.retrieve_chunks( query="Summarize wildlife or flood risks that impact the wetlands buffer program", filters=filters, k=4, padding=1, use_colpali=True, ) print(f"\nRetrieved {len(chunks)} filtered chunks:") for chunk in chunks: print(f"\nChunk {chunk.chunk_number} from {chunk.filename} (score={chunk.score:.3f})") print(f"Content preview: {chunk.content[:200]}...") print(f"Metadata: {chunk.metadata}") ``` ## Supported Filter Operators | Operator | Description | Example | | ----------- | ------------------------- | ---------------------------------------------- | | `$eq` | Exact match | `{"status": {"$eq": "active"}}` | | `$in` | Value in array | `{"region": {"$in": ["andes", "altiplano"]}}` | | `$gte` | Greater than or equal | `{"date": {"$gte": "2024-01-01"}}` | | `$lte` | Less than or equal | `{"score": {"$lte": 45}}` | | `$gt` | Greater than | `{"temperature": {"$gt": 0}}` | | `$lt` | Less than | `{"count": {"$lt": 100}}` | | `$contains` | Array contains value | `{"tags": {"$contains": {"value": "urgent"}}}` | | `$and` | All conditions must match | `{"$and": [condition1, condition2]}` | | `$or` | Any condition must match | `{"$or": [condition1, condition2]}` | ## Use Cases Complex metadata filtering is ideal for: * **Document management systems** with multi-dimensional categorization * **Compliance and audit systems** requiring date-based queries * **Scientific data repositories** with measurements and precise numerical filtering * **Multi-tenant applications** with scope-based isolation * **Time-series document collections** with date range queries * **Hierarchical data** with nested metadata structures ## Best Practices ### 1. Use Appropriate Types Use the correct Python types for metadata: ```python theme={null} # ✅ Correct metadata = { "date": date(2024, 9, 15), # Use date objects "price": Decimal("19.99"), # Use Decimal for precision "is_active": True, # Use bool for flags } # ❌ Avoid metadata = { "date": "2024-09-15", # String instead of date "price": 19.99, # Float loses precision "is_active": "true", # String instead of bool } ``` ### 2. Convert Dates for Filtering Always convert date objects to ISO format when building filters: ```python theme={null} # ✅ Correct {"fieldwork_date": {"$gte": date(2024, 9, 15).isoformat()}} # ❌ Wrong {"fieldwork_date": {"$gte": date(2024, 9, 15)}} # Date object won't work ``` ### 3. Combine Operators Strategically * Use `$and` for required conditions that must all match * Use `$in` when a field can have multiple possible values * Use range operators (`$gte`, `$lte`) for numerical and date filtering * Use `$contains` for array membership checks ### 4. Index Important Fields Frequently filtered fields benefit from proper indexing. Consider performance when adding many metadata fields. ## Running the Example ```bash theme={null} # Set your Morphik URI export MORPHIK_URI="morphik://your-app:your-token@api.morphik.ai" # Run your Python script with the code above python your_script.py ``` ## Related Cookbooks * [Generating Completions with Retrieved Chunks](./generating-completions-with-retrieved-chunks) - Send filtered chunks to OpenAI * [Python SDK Basic Operations](./python-basic-operations) - Core Morphik operations # Generating Completions with Retrieved Chunks Source: https://morphik.ai/docs/cookbooks/generating-completions-with-retrieved-chunks Send Morphik document chunks to OpenAI using presigned URLs or base64-encoded images for vision model completions. This cookbook demonstrates how to retrieve document chunks from Morphik and send them to OpenAI for completion generation, using both presigned URLs and base64-encoded images. > **Prerequisites** > > * Install the Morphik SDK: `pip install morphik` > * Install OpenAI SDK: `pip install openai` > * Provide credentials via `MORPHIK_URI` and `OPENAI_API_KEY` > * Documents ingested with multimodal support (`use_colpali=True`) ## 1. Ingest Documents with Multimodal Support First, ingest your documents with multimodal retrieval enabled: ```python theme={null} from datetime import date from morphik import Morphik from openai import OpenAI # Initialize clients morphik_client = Morphik("morphik://your-app:token@api.morphik.ai") openai_client = OpenAI(api_key="your-openai-key") # Ingest PDF with multimodal support doc = morphik_client.ingest_file( file="morphik_platform_brief.pdf", metadata={ "collection": "demo-ai-briefs", "published_date": date(2024, 9, 14), "priority_score": 42, "requires_followup": True, "tags": ["morphik", "roadmap"], }, use_colpali=True, # Enable multimodal retrieval ) # Wait for processing doc.wait_for_completion(timeout_seconds=240) print(f"Ingested: {doc.external_id}") ``` ## 2. Retrieve Chunks as Presigned URLs Get chunks as URLs that can be sent directly to vision models: ```python theme={null} # Retrieve chunks as URLs url_chunks = morphik_client.retrieve_chunks( query="List notable roadmap items and vision experiments", filters={"collection": "demo-ai-briefs"}, k=4, padding=1, use_colpali=True, # Must match ingestion setting output_format="url", # Get presigned URLs ) # Extract URLs from chunks urls = [] for chunk in url_chunks: if isinstance(chunk.content, str) and chunk.content.startswith("http"): urls.append(chunk.content) elif chunk.download_url: urls.append(chunk.download_url) print(f"Found {len(urls)} image URLs") ``` ## 3. Send URLs to OpenAI Send the presigned URLs to OpenAI's vision model: ```python theme={null} QUESTION = "List notable roadmap items, vision experiments, and follow-up actions" # Build content with text and image URLs content = [{"type": "input_text", "text": QUESTION}] for url in urls: content.append({"type": "input_image", "image_url": url}) # Call OpenAI Responses API response = openai_client.responses.create( model="gpt-5.1", input=[{"role": "user", "content": content}], ) print(response.output_text) ``` ## 4. Retrieve Chunks as Base64 Images For cases where you need base64-encoded images: ```python theme={null} import base64 from io import BytesIO from PIL.Image import Image as PILImage # Retrieve chunks as PIL Images (default format) base64_chunks = morphik_client.retrieve_chunks( query=QUESTION, filters={"collection": "demo-ai-briefs"}, k=4, padding=1, use_colpali=True, output_format=None, # Default: returns PIL Images ) # Convert PIL Images to base64 data URIs def encode_chunk_image(chunk) -> str: if not isinstance(chunk.content, PILImage): return None # Always use image/png for PIL Images buffer = BytesIO() chunk.content.save(buffer, format="PNG") encoded = base64.b64encode(buffer.getvalue()).decode("utf-8") return f"data:image/png;base64,{encoded}" data_uris = [encode_chunk_image(chunk) for chunk in base64_chunks] data_uris = [uri for uri in data_uris if uri] # Filter out None print(f"Found {len(data_uris)} base64 images") ``` ## 5. Send Base64 Images to OpenAI ```python theme={null} # Build content with text and base64 images content = [{"type": "input_text", "text": QUESTION}] for data_uri in data_uris: content.append({"type": "input_image", "image_url": data_uri}) # Call OpenAI response = openai_client.responses.create( model="gpt-5.1", input=[{"role": "user", "content": content}], ) print(response.output_text) ``` ## Important Notes ### Multimodal Ingestion and Retrieval When you ingest with `use_colpali=True`, you **must** retrieve with `use_colpali=True`: ```python theme={null} # ✅ Correct doc = client.ingest_file(file="doc.pdf", use_colpali=True) chunks = client.retrieve_chunks(query="...", use_colpali=True) # ❌ Wrong - Mismatch will return 0 results doc = client.ingest_file(file="doc.pdf", use_colpali=True) chunks = client.retrieve_chunks(query="...", use_colpali=False) ``` ### Content Type Handling Chunks from PDFs ingested with multimodal support will have: * `chunk.content_type` = `"application/pdf"` (original document type) * `chunk.content` = PIL Image object (actual content) When encoding to base64, always use `"image/png"` as the MIME type: ```python theme={null} # ✅ Correct: Use image/png for PIL Images data_uri = f"data:image/png;base64,{encoded}" # ❌ Wrong: Don't use chunk.content_type (would be "application/pdf") data_uri = f"data:{chunk.content_type};base64,{encoded}" ``` ### Output Format Comparison | Format | When to Use | Pros | Cons | | ----------------------------- | -------------------------------- | ---------------------------- | --------------------------- | | `output_format="url"` | Production, large images | No encoding overhead, faster | URLs expire after some time | | `output_format=None` (base64) | Small images, offline processing | Always available | Larger payload size | ## Use Cases This pattern is ideal for: * **Document Q\&A** over visual documents (PDFs, scans, diagrams) * **Report generation** from technical documentation with charts and tables * **Visual data analysis** combining text and image understanding * **Multi-document synthesis** aggregating information across documents * **Chart and diagram interpretation** using vision-capable models * **Technical specification review** analyzing mixed text-visual content ## Best Practices ### 1. Choose the Right Output Format Use URLs for production workloads with large images: ```python theme={null} # Production: Use URLs chunks = client.retrieve_chunks(..., output_format="url") ``` Use base64 for small images or offline processing: ```python theme={null} # Small images or offline: Use base64 chunks = client.retrieve_chunks(..., output_format=None) ``` ### 2. Handle Chunk Padding Use `padding` to include adjacent chunks/pages for better context: ```python theme={null} chunks = client.retrieve_chunks( query="...", k=4, padding=1, # Include 1 adjacent chunk on each side use_colpali=True, ) ``` ### 3. Filter with Metadata Combine retrieval with metadata filtering for precise results: ```python theme={null} chunks = client.retrieve_chunks( query="roadmap items", filters={ "$and": [ {"collection": {"$eq": "technical-docs"}}, {"published_date": {"$gte": "2024-01-01"}}, {"priority_score": {"$gte": 40}}, ] }, k=4, use_colpali=True, ) ``` ## Running the Example ```bash theme={null} # Set environment variables export MORPHIK_URI="morphik://your-app:your-token@api.morphik.ai" export OPENAI_API_KEY="sk-..." # Run your Python script with the code above python your_script.py ``` ## Related Cookbooks * [Complex Metadata Filtering](./complex-metadata-filtering) - Advanced document filtering with dates, arrays, and more * [Python SDK Basic Operations](./python-basic-operations) - Core Morphik operations # Python SDK Basic Operations Source: https://morphik.ai/docs/cookbooks/python-basic-operations End-to-end walkthrough of ingestion, retrieval, and LLM integration using the Morphik Python SDK. This cookbook walks through the core Morphik workflow using the Python SDK—multimodal ingestion for high-accuracy retrieval, text ingestion for OCR-driven chunks, and integrating with your own LLM. > **Prerequisites** > > * Install the Morphik SDK: `pip install morphik` > * Provide credentials via Morphik URI > * For the optional OpenAI example, set `OPENAI_API_KEY` > **Note on Async Support** > This guide uses the synchronous `Morphik` client. An async version `AsyncMorphik` is also available for async workflows with the same API. > **Note on Streaming** > Response streaming is not currently available in the Python SDK as of version 0.2.12. This feature will be added in a future release. ## 1. Initialize the Morphik client ```python theme={null} from morphik import Morphik # Initialize with Morphik URI client = Morphik( uri="morphik://your-name:your-token@api.morphik.ai" ) QUESTION = "What are the key takeaways from the uploaded document?" ``` ## 2. Multimodal ingestion (ColPali) This path indexes the original file contents directly, yielding higher accuracy for scanned pages, tables, and images. ```python theme={null} # Ingest with ColPali (multimodal) multimodal_doc = client.ingest_file( file="path/to/document.pdf", metadata={"demo_variant": "multimodal"}, use_colpali=True ) print(f"Ingested document: {multimodal_doc.external_id}") print(f"Status: {multimodal_doc.status}") ``` ### Wait for processing to complete ```python theme={null} # Wait for processing using built-in method doc = client.wait_for_document_completion( multimodal_doc.external_id, timeout_seconds=120, check_interval_seconds=2 ) print(f"Document {doc.external_id} processing completed!") ``` ## 3. Text ingestion (OCR + chunking) This path OCRs the document before chunking it, making the text immediately available for retrieval. ```python theme={null} # Ingest without ColPali (standard text extraction) text_doc = client.ingest_file( file="path/to/document.pdf", metadata={"demo_variant": "standard"}, use_colpali=False ) # Wait for processing client.wait_for_document_completion(text_doc.external_id, timeout_seconds=120) ``` ## 4. Query with Morphik completion Generate a completion directly using Morphik's configured LLM. ### Multimodal query (ColPali) ```python theme={null} # Query with multimodal chunks multimodal_response = client.query( query=QUESTION, use_colpali=True, k=4, filters={"demo_variant": "multimodal"} ) print(f"Multimodal answer: {multimodal_response.completion}") print(f"Token usage: {multimodal_response.usage}") print(f"Sources: {len(multimodal_response.sources)} chunks used") ``` ### Text query (standard) ```python theme={null} # Query with text chunks text_response = client.query( query=QUESTION, use_colpali=False, k=4, filters={"demo_variant": "standard"} ) print(f"Text answer: {text_response.completion}") ``` ## 5. Query with system prompt override Override the default system prompt to customize the LLM's behavior and response style. ```python theme={null} # Pirate assistant example pirate_response = client.query( query="What is this document about?", k=3, prompt_overrides={ "query": { "system_prompt": ( "You are a pirate assistant. Always respond in pirate speak " "with arrr and matey! Be entertaining while staying accurate " "to the document content." ) } } ) print(f"Pirate response: {pirate_response.completion}") # Output: "Arrr, me hearty! This here document be talkin' about..." ``` ### Legal expert example ```python theme={null} # Legal expert persona legal_response = client.query( query="What are the key terms?", k=4, prompt_overrides={ "query": { "system_prompt": ( "You are a legal expert. Analyze the document and provide " "insights in formal legal language. Always cite specific " "sections when making claims." ) } } ) print(f"Legal analysis: {legal_response.completion}") ``` ### Custom prompt template You can also override the prompt template that formats the context and question: ```python theme={null} # Financial analyst with custom template financial_response = client.query( query="What are the revenue figures?", k=4, prompt_overrides={ "query": { "system_prompt": "You are a financial analyst. Provide precise numerical answers.", "prompt_template": ( "Based on the following financial data:\n\n" "{context}\n\n" "Analyze and answer: {question}\n\n" "Provide specific numbers and percentages where available." ) } } ) ``` ## 6. Morphik On-the-Fly document query Analyse a document inline with Morphik On-the-Fly before committing it to long-term storage. You can enforce structured output and decide whether to enqueue ingestion afterwards. ```python theme={null} document_query = client.query_document( file="path/to/document.pdf", prompt="Extract the parties, effective date, and governing jurisdiction.", schema={ "parties": {"type": "ARRAY"}, "effective_date": "date", "jurisdiction": "string", }, ingestion_options={ "ingest": True, "metadata": {"source": "contracts"}, "use_colpali": True, } ) print("Structured output:", document_query.structured_output) print("Ingestion queued:", document_query.ingestion_enqueued) ``` Folder and user scoping work exactly like other SDK calls: ```python theme={null} folder_query = client.get_folder_by_name("contracts").query_document( file="path/to/nda.pdf", prompt="Summarize the non-disclosure obligations.", ingestion_options={"ingest": False} ) user_query = client.signin("end-user-42").query_document( file="path/to/statement.pdf", prompt="Capture the account balance and due date.", ingestion_options={"metadata": {"channel": "upload"}} ) ``` ## 7. Retrieve chunks for your own LLM For production workloads, retrieve Morphik's curated chunks and forward them to your preferred LLM. This gives you full control over prompts, orchestration, and rate limits. ### Step 1: Retrieve relevant chunks ```python theme={null} # Retrieve multimodal chunks chunks = client.retrieve_chunks( query=QUESTION, use_colpali=True, k=4, padding=1, # Get 1 additional chunk before/after each match filters={"demo_variant": "multimodal"} ) print(f"Retrieved {len(chunks)} chunks") # Inspect chunk details for i, chunk in enumerate(chunks): print(f"\nChunk {i + 1}:") print(f" Document: {chunk.document_id}") print(f" Score: {chunk.score:.2f}") print(f" Content preview: {chunk.content[:100]}...") if chunk.download_url: print(f" Image URL: {chunk.download_url}") ``` ### Step 2: Forward to your LLM (OpenAI example) #### Text-only chunks with OpenAI ```python theme={null} from openai import OpenAI openai_client = OpenAI() # Retrieve text chunks text_chunks = client.retrieve_chunks( query=QUESTION, use_colpali=False, k=4, filters={"demo_variant": "standard"} ) # Build context from chunks context = "\n\n".join([ f"Source #{i + 1}:\n{chunk.content}" for i, chunk in enumerate(text_chunks) ]) # Query OpenAI openai_response = openai_client.chat.completions.create( model="gpt-4o-mini", messages=[ { "role": "system", "content": "You are a helpful assistant. Use only the provided context to answer questions." }, { "role": "user", "content": f"Context:\n\n{context}\n\nQuestion: {QUESTION}" } ] ) print(openai_response.choices[0].message.content) ``` #### Multimodal chunks with OpenAI (vision) ```python theme={null} # Retrieve multimodal chunks with images multimodal_chunks = client.retrieve_chunks( query=QUESTION, use_colpali=True, k=4, filters={"demo_variant": "multimodal"} ) # Filter chunks that have image URLs image_chunks = [ chunk for chunk in multimodal_chunks if chunk.download_url and chunk.content_type and chunk.content_type.startswith("image/") ] # Build multimodal content content = [ { "type": "text", "text": f"Answer using these images.\n\nQuestion: {QUESTION}" } ] # Add images for chunk in image_chunks: content.append({ "type": "image_url", "image_url": {"url": chunk.download_url} }) # Query OpenAI with vision vision_response = openai_client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": content}] ) print(vision_response.choices[0].message.content) ``` ### Step 3: Using other LLM providers The same pattern works with any LLM provider. Morphik handles retrieval and chunking; you control the completion. #### Anthropic Claude ```python theme={null} from anthropic import Anthropic anthropic_client = Anthropic() # Build context context = "\n\n".join([f"Source #{i + 1}:\n{c.content}" for i, c in enumerate(text_chunks)]) # Query Claude claude_response = anthropic_client.messages.create( model="claude-3-5-sonnet-20241022", max_tokens=1024, messages=[{ "role": "user", "content": f"Context:\n\n{context}\n\nQuestion: {QUESTION}" }] ) print(claude_response.content[0].text) ``` #### Google Gemini ```python theme={null} import google.generativeai as genai genai.configure(api_key="your-google-api-key") model = genai.GenerativeModel("gemini-pro") # Build context and query prompt = f"Context:\n\n{context}\n\nQuestion: {QUESTION}" # Query Gemini gemini_response = model.generate_content(prompt) print(gemini_response.text) ``` ## 8. Working with folders and user scopes Morphik supports organizing documents into folders and scoping queries by end users. ```python theme={null} # Query within a specific folder folder_response = client.get_folder_by_name("my-folder").query( query=QUESTION, k=4 ) # Scope to a specific end user user_response = client.signin("user-123").query( query=QUESTION, k=4 ) # Combine folder and user scoping scoped_response = client.get_folder_by_name("my-folder").signin("user-123").query( query=QUESTION, k=4 ) ``` ## 9. Additional features ### Chat history Maintain conversation context across queries: ```python theme={null} chat_response = client.query( query="What is the main topic?", chat_id="conversation-123", k=4 ) # Follow-up question with context followup_response = client.query( query="Can you elaborate on that?", chat_id="conversation-123", k=4 ) ``` ### Structured output Extract structured data using Pydantic models: ```python theme={null} from pydantic import BaseModel from typing import List class DocumentSummary(BaseModel): title: str key_points: List[str] category: str structured_response = client.query( query="Summarize this document", k=4, schema=DocumentSummary ) # Response is now a dictionary matching the schema print(structured_response.completion) # Output: {"title": "...", "key_points": [...], "category": "..."} ``` ### Custom LLM configuration Use a different LLM for specific queries: ```python theme={null} custom_llm_response = client.query( query=QUESTION, k=4, llm_config={ "model": "gpt-4o", "api_key": "your-openai-key" } ) ``` ## Summary The Morphik Python SDK provides flexible integration options: 1. **Managed completions**: Use Morphik's configured LLM with optional system prompt overrides 2. **Bring your own LLM**: Retrieve curated chunks and forward to any LLM provider 3. **Multimodal support**: Handle both text and visual content seamlessly 4. **Morphik On-the-Fly**: Run inline analyses with structured output and optional ingestion 5. **Full control**: Override system prompts, prompt templates, and completion parameters 6. **Organization**: Folder and user-based scoping for multi-tenant applications This separation of concerns lets you focus on your application logic while Morphik handles high-quality retrieval and chunking. # TypeScript Basic Operations Source: https://morphik.ai/docs/cookbooks/typescript-basic-operations End-to-end walkthrough of ingestion, retrieval, and LLM handoff with the Morphik TypeScript SDK. This cookbook walks through the core Morphik workflow in TypeScript—multimodal ingestion for high-accuracy retrieval, text ingestion for OCR-driven chunks, and piping the results into the LLM you control. > **Prerequisites** > > * Install the Morphik SDK: `npm install morphik` > * Provide credentials via `MORPHIK_API_KEY` > * For the optional OpenAI example, set `OPENAI_API_KEY` and (optionally) `OPENAI_MODEL` ## 1. Initialize the Morphik client ```ts theme={null} import fs from 'fs'; import Morphik from 'morphik'; const client = new Morphik({ apiKey: process.env.MORPHIK_API_KEY!, baseURL: 'https://api.morphik.ai', }); const filePath = 'path/to/document.pdf'; const QUESTION = 'What are the key takeaways from the uploaded document?'; ``` ## 2. Helper utilities ```ts theme={null} function getDocumentId(document: any): string { if (document.external_id) return document.external_id; const systemMetadata = document.system_metadata as { document_id?: unknown } | undefined; if (systemMetadata?.document_id && typeof systemMetadata.document_id === 'string') { return systemMetadata.document_id; } throw new Error('Document response did not include an external_id or document_id.'); } async function waitForProcessing(documentId: string) { const timeoutMs = 120_000; const intervalMs = 2_000; const started = Date.now(); while (true) { const status = await client.documents.getStatus(documentId); const state = (status as any).status ?? (status as any).document_status; if (state === 'completed') return; if (state === 'failed') throw new Error(`Document ${documentId} failed to process.`); if (Date.now() - started > timeoutMs) throw new Error('Processing timed out.'); await new Promise((resolve) => setTimeout(resolve, intervalMs)); } } async function logSources(sources: any[] | undefined, label: string) { if (!sources?.length) { console.log(`${label}: no sources returned.`); return; } const chunkDetails = await client.batch.retrieveChunks({ body: { sources: sources.map((source) => ({ document_id: source.document_id, chunk_number: source.chunk_number, })), }, }); chunkDetails.forEach((chunk: any, index: number) => { const preview = chunk.content.slice(0, 120).replace(/\s+/g, ' '); console.log(`${label} #${index + 1} doc ${chunk.document_id} chunk ${chunk.chunk_number}: ${preview}`); }); } ``` ## 3. Multimodal workflow (direct file indexing) This path indexes the original file contents directly, yielding higher accuracy for scanned pages, tables, and images. ```ts theme={null} const multimodalDocument = await client.ingest.ingestFile({ file: fs.createReadStream(filePath), metadata: JSON.stringify({ demo_variant: 'multimodal' }), use_colpali: true, }); await waitForProcessing(getDocumentId(multimodalDocument)); ``` ### Retrieve multimodal chunks first ```ts theme={null} const multimodalChunks = await client.retrieve.chunks.create({ query: QUESTION, use_colpali: true, k: 4, padding: 1, filters: { demo_variant: 'multimodal' }, }); ``` At this point you can either: * ask Morphik to draft the answer for you, or * forward the curated chunks to your own LLM (see [Use your own LLM](#5-use-your-own-llm-openai-example)). ### Option A: generate a Morphik completion (multimodal) ```ts theme={null} const multimodalAnswer = await client.query.generateCompletion({ query: QUESTION, use_colpali: true, k: 4, filters: { demo_variant: 'multimodal' }, }); console.log('Multimodal answer:', multimodalAnswer.completion); await logSources(multimodalAnswer.sources, 'Multimodal source'); ``` ## 4. Text workflow (OCR + chunking) This path OCRs the document before chunking it, making the text immediately available for retrieval. ```ts theme={null} const standardDocument = await client.ingest.ingestFile({ file: fs.createReadStream(filePath), metadata: JSON.stringify({ demo_variant: 'standard' }), }); ``` ### Retrieve text chunks first ```ts theme={null} const textChunks = await client.retrieve.chunks.create({ query: QUESTION, use_colpali: false, k: 4, padding: 1, filters: { demo_variant: 'standard' }, }); ``` Choose the same fork as above: * stick with Morphik completions for a managed response, or * jump to [Use your own LLM](#5-use-your-own-llm-openai-example) to prompt your custom model with these chunks. ### Option A: generate a Morphik completion (text) ```ts theme={null} const textAnswer = await client.query.generateCompletion({ query: QUESTION, use_colpali: false, k: 4, filters: { demo_variant: 'standard' }, }); console.log('Text answer:', textAnswer.completion); await logSources(textAnswer.sources, 'Text source'); ``` ## 5. Use your own LLM (OpenAI example) For production workloads, forward Morphik's curated context to your preferred LLM. This gives you full control over system prompts, orchestration, and rate limits, while Morphik focuses on delivering retrieval tools your agent can trust. ```ts theme={null} import OpenAI from 'openai'; const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY }); // Text-only prompt const textContext = textChunks .map((chunk: any, idx: number) => `Source #${idx + 1}:\n${chunk.content}`) .join('\n\n'); const textResponse = await openai.responses.create({ model: process.env.OPENAI_MODEL ?? 'gpt-4o-mini', input: [{ role: 'user', content: [{ type: 'text', text: `Use the context below to answer.\n\n${textContext}\n\nQuestion: ${QUESTION}` }], }], }); // Multimodal prompt (only includes chunks with image URLs) const imageChunks = multimodalChunks.filter( (chunk: any) => chunk.download_url && chunk.content_type?.startsWith('image/'), ); const multimodalResponse = await openai.responses.create({ model: process.env.OPENAI_MODEL ?? 'gpt-4o-mini', input: [{ role: 'user', content: [ { type: 'text', text: `Answer using only these images.\nQuestion: ${QUESTION}` }, ...imageChunks.map((chunk: any) => ({ type: 'image_url', image_url: { url: chunk.download_url }, })), ], }], }); ``` Morphik stays focused on high-quality retrieval and chunking. Pair it with the LLM of your choice to enforce your policies, prompts, and cost controls end to end. # Batch Get Chunks Source: https://morphik.ai/docs/core-functions/batch-get-chunks Retrieve specific chunks by document ID and chunk number Retrieve specific chunks by their document ID and chunk number in a single batch operation. Useful for fetching exact chunks after retrieval or for building custom pipelines. ```python theme={null} from morphik import Morphik db = Morphik("your-uri") chunks = db.batch_get_chunks( sources=[ {"document_id": "doc_abc123", "chunk_number": 0}, {"document_id": "doc_abc123", "chunk_number": 1}, {"document_id": "doc_xyz789", "chunk_number": 5} ], folder_name="/reports", use_colpali=True, output_format="url" ) for chunk in chunks: print(f"Doc {chunk.document_id}, Chunk {chunk.chunk_number}") print(f"Content: {chunk.content[:200]}...") ``` ```typescript theme={null} import Morphik from 'morphik'; // For Teams/Enterprise, use your dedicated host: https://companyname-api.morphik.ai const client = new Morphik({ apiKey: process.env.MORPHIK_API_KEY, baseURL: 'https://api.morphik.ai' }); const chunks = await client.batch.retrieveChunks({ sources: [ { document_id: 'doc_abc123', chunk_number: 0 }, { document_id: 'doc_abc123', chunk_number: 1 }, { document_id: 'doc_xyz789', chunk_number: 5 } ], folder_name: '/reports', use_colpali: true, output_format: 'url' }); chunks.forEach(chunk => { console.log(`Doc ${chunk.document_id}, Chunk ${chunk.chunk_number}`); console.log(`Content: ${chunk.content.slice(0, 200)}...`); }); ``` ```bash theme={null} curl -X POST "https://api.morphik.ai/batch/chunks" \ -H "Authorization: Bearer $MORPHIK_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "sources": [ {"document_id": "doc_abc123", "chunk_number": 0}, {"document_id": "doc_abc123", "chunk_number": 1}, {"document_id": "doc_xyz789", "chunk_number": 5} ], "folder_name": "/reports", "use_colpali": true, "output_format": "url" }' ``` ## Parameters | Parameter | Type | Default | Description | | --------------- | ------- | ---------- | ------------------------------------------------ | | `sources` | array | required | List of `{document_id, chunk_number}` objects | | `use_colpali` | boolean | `true` | Use Morphik multimodal embeddings when available | | `output_format` | string | `"base64"` | Image format: `base64`, `url`, or `text` | | `folder_name` | string | `null` | Optional folder scope | ## Response ```json theme={null} [ { "document_id": "doc_abc123", "chunk_number": 0, "content": "Introduction to the quarterly report...", "content_type": "text/plain", "score": 1.0, "metadata": { "department": "sales" } }, { "document_id": "doc_abc123", "chunk_number": 1, "content": "Revenue highlights for Q4...", "content_type": "text/plain", "score": 1.0, "metadata": { "department": "sales" } } ] ``` This is useful when you already know which chunks you need (e.g., from a previous retrieval result) and want to fetch their full content efficiently. # Delete Document Source: https://morphik.ai/docs/core-functions/delete-document Remove a document and all its associated data Delete a document and all its associated data including metadata, stored content, and vector embeddings. ```python theme={null} from morphik import Morphik db = Morphik("your-uri") # Delete by document ID result = db.delete_document("doc_abc123") print(result) # {"status": "success", "message": "Document deleted"} # Delete by filename result = db.delete_document_by_filename("report.pdf") ``` ```typescript theme={null} import Morphik from 'morphik'; // For Teams/Enterprise, use your dedicated host: https://companyname-api.morphik.ai const client = new Morphik({ apiKey: process.env.MORPHIK_API_KEY, baseURL: 'https://api.morphik.ai' }); // Delete by document ID const result = await client.documents.delete('doc_abc123'); console.log(result); // { status: 'success', message: 'Document deleted' } ``` ```bash theme={null} curl -X DELETE "https://api.morphik.ai/documents/doc_abc123" \ -H "Authorization: Bearer $MORPHIK_API_KEY" ``` ## Parameters | Parameter | Type | Description | | ------------- | ------ | ----------------------------------------------- | | `document_id` | string | The unique identifier of the document to delete | ## Response ```json theme={null} { "status": "success", "message": "Document doc_abc123 and all associated data deleted successfully" } ``` This action is irreversible. All document data including chunks and embeddings will be permanently removed. # Ingest File Source: https://morphik.ai/docs/core-functions/ingest-file Upload and process a file into Morphik Upload a file to Morphik for processing. The file is stored and a background worker handles parsing and chunking. ```python theme={null} from morphik import Morphik db = Morphik("your-uri") doc = db.ingest_file( file="report.pdf", filename="Q4_Report.pdf", metadata={"department": "sales", "year": 2025, "quarter": "Q4"}, use_colpali=True ) doc.wait_for_completion() print(f"Document ID: {doc.external_id}") ``` ```typescript theme={null} import Morphik from 'morphik'; import fs from 'fs'; // For Teams/Enterprise, use your dedicated host: https://companyname-api.morphik.ai const client = new Morphik({ apiKey: process.env.MORPHIK_API_KEY, baseURL: 'https://api.morphik.ai' }); const doc = await client.ingest.ingestFile({ file: fs.createReadStream('report.pdf'), metadata: JSON.stringify({ department: 'sales', year: 2025, quarter: 'Q4' }), use_colpali: true, folder_name: '/reports/quarterly' }); console.log(`Document ID: ${doc.external_id}`); ``` ```bash theme={null} curl -X POST "https://api.morphik.ai/ingest/file" \ -H "Authorization: Bearer $MORPHIK_API_KEY" \ -F "file=@report.pdf" \ -F 'metadata={"department": "sales", "year": 2025, "quarter": "Q4"}' \ -F "use_colpali=true" \ -F "folder_name=/reports/quarterly" ``` ## Parameters | Parameter | Type | Default | Description | | ------------- | ------- | -------- | ------------------------------------------------------------------ | | `file` | file | required | The file to upload | | `filename` | string | `null` | Override filename (optional) | | `metadata` | object | `{}` | Custom metadata to attach to the document | | `use_colpali` | boolean | `true` | Use Morphik multimodal embeddings for better image/table retrieval | | `folder_name` | string | `null` | Target folder path for organization | ## Supported File Types | Category | Extensions | | -------------- | -------------------------------------------------------- | | **Documents** | `.pdf` | | **Word** | `.docx`, `.doc` | | **PowerPoint** | `.pptx`, `.ppt`, `.ppsx` | | **Excel** | `.xlsx`, `.xls`, `.xlsm` | | **Images** | `.jpg`, `.png`, `.gif`, `.webp`, `.tiff`, `.bmp`, `.svg` | | **Video** | `.mp4`, `.mpeg`, `.mov`, `.avi`, `.webm`, `.mkv`, `.3gp` | | **Text** | `.txt`, `.md`, `.rst`, `.log` | | **Data** | `.json`, `.csv`, `.tsv`, `.yaml`, `.xml` | | **Web** | `.html`, `.htm` | ## Response ```json theme={null} { "external_id": "doc_abc123", "filename": "report.pdf", "content_type": "application/pdf", "metadata": { "department": "sales", "year": 2025 }, "system_metadata": { "status": "processing" } } ``` ## Waiting for Processing Documents are processed asynchronously. Use these methods to wait for completion: ```python theme={null} doc.wait_for_completion() ``` ```typescript theme={null} const status = await client.documents.getStatus(doc.external_id); // status.status will be "processing", "completed", or "failed" ``` ```bash theme={null} curl -X GET "https://api.morphik.ai/documents/{document_id}/status" \ -H "Authorization: Bearer $MORPHIK_API_KEY" ``` # List Documents Source: https://morphik.ai/docs/core-functions/list-documents List and filter documents in your Morphik database Retrieve a list of documents with optional filtering, sorting, and pagination. ```python theme={null} from morphik import Morphik db = Morphik("your-uri") response = db.list_documents( skip=0, limit=10, filters={"department": {"$eq": "sales"}}, sort_by="created_at", sort_direction="desc", include_total_count=True, include_status_counts=True, completed_only=False ) for doc in response.documents: print(f"{doc.filename}: {doc.external_id}") print(f"Total: {response.total_count}") if response.has_more: next_page = db.list_documents(skip=response.next_skip, limit=10) ``` ```typescript theme={null} import Morphik from 'morphik'; // For Teams/Enterprise, use your dedicated host: https://companyname-api.morphik.ai const client = new Morphik({ apiKey: process.env.MORPHIK_API_KEY, baseURL: 'https://api.morphik.ai' }); const response = await client.documents.listDocs({ skip: 0, limit: 10, document_filters: { department: { $eq: 'sales' } }, sort_by: 'created_at', sort_direction: 'desc', include_total_count: true, include_status_counts: true, completed_only: false }); response.documents?.forEach(doc => { console.log(`${doc.filename}: ${doc.external_id}`); }); console.log(`Total: ${response.total_count}`); if (response.has_more) { const nextPage = await client.documents.listDocs({ skip: response.next_skip, limit: 10 }); } ``` ```bash theme={null} curl -X POST "https://api.morphik.ai/documents/list_docs" \ -H "Authorization: Bearer $MORPHIK_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "skip": 0, "limit": 10, "document_filters": {"department": {"$eq": "sales"}}, "sort_by": "created_at", "sort_direction": "desc", "include_total_count": true, "include_status_counts": true, "completed_only": false }' ``` For advanced filtering with operators like `$and`, `$or`, `$gte`, `$in`, see [Metadata Filtering](/concepts/metadata-filtering). ## Parameters | Parameter | Type | Default | Description | | --------------------- | ------- | -------------- | ------------------------------------------------------------------------- | | `skip` | int | `0` | Number of documents to skip (pagination) | | `limit` | int | `100` | Maximum documents to return | | `filters` | object | `null` | Metadata filters (see [Metadata Filtering](/concepts/metadata-filtering)) | | `sort_by` | string | `"updated_at"` | Sort field: `created_at`, `updated_at`, `filename`, `external_id` | | `sort_direction` | string | `"desc"` | Sort direction: `asc` or `desc` | | `completed_only` | boolean | `false` | Only return fully processed documents | | `include_total_count` | boolean | `false` | Include total count in response | ## Response ```json theme={null} { "documents": [ { "external_id": "doc_abc123", "filename": "report.pdf", "content_type": "application/pdf", "metadata": { "department": "sales" } } ], "returned_count": 10, "total_count": 42, "has_more": true, "next_skip": 10 } ``` # Retrieve Chunks Source: https://morphik.ai/docs/core-functions/retrieve-chunks Search and retrieve relevant chunks from your documents Search your documents and retrieve the most relevant chunks based on a natural language query. ```python theme={null} from morphik import Morphik db = Morphik("your-uri") chunks = db.retrieve_chunks( query="What are the quarterly results?", filters={"department": {"$eq": "sales"}}, k=5, min_score=0.0, use_colpali=True, folder_name="/reports", padding=1, output_format="url" ) for chunk in chunks: print(f"Score: {chunk.score:.2f}") print(f"Content: {chunk.content[:200]}...") ``` ```typescript theme={null} import Morphik from 'morphik'; // For Teams/Enterprise, use your dedicated host: https://companyname-api.morphik.ai const client = new Morphik({ apiKey: process.env.MORPHIK_API_KEY, baseURL: 'https://api.morphik.ai' }); const chunks = await client.retrieve.chunks.create({ query: 'What are the quarterly results?', filters: { department: { $eq: 'sales' } }, k: 5, min_score: 0.0, use_colpali: true, folder_name: '/reports', padding: 1, output_format: 'url' }); chunks.forEach(chunk => { console.log(`Score: ${chunk.score.toFixed(2)}`); console.log(`Content: ${chunk.content.slice(0, 200)}...`); }); ``` ```bash theme={null} curl -X POST "https://api.morphik.ai/retrieve/chunks" \ -H "Authorization: Bearer $MORPHIK_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "query": "What are the quarterly results?", "filters": {"department": {"$eq": "sales"}}, "k": 5, "min_score": 0.0, "use_colpali": true, "folder_name": "/reports", "padding": 1, "output_format": "url" }' ``` For advanced filtering with operators like `$and`, `$or`, `$gte`, `$in`, see [Metadata Filtering](/concepts/metadata-filtering). ## Parameters | Parameter | Type | Default | Description | | --------------- | ------- | ---------- | ------------------------------------------------------------------------- | | `query` | string | required | Natural language search query | | `k` | int | `4` | Maximum number of chunks to return | | `min_score` | float | `0.0` | Minimum similarity score threshold | | `use_colpali` | boolean | `true` | Use Morphik multimodal embeddings (better for images/tables) | | `filters` | object | `null` | Metadata filters (see [Metadata Filtering](/concepts/metadata-filtering)) | | `padding` | int | `0` | Extra pages to include before/after matches (multimodal only) | | `output_format` | string | `"base64"` | Image format: `base64`, `url`, or `text` | ## Response ```json theme={null} [ { "document_id": "doc_abc123", "chunk_number": 3, "content": "Q4 revenue increased by 15% year-over-year...", "score": 0.89, "metadata": { "department": "sales" }, "filename": "quarterly_report.pdf" } ] ``` Use `use_colpali=true` for documents with images, charts, or tables. Morphik multimodal embeddings provide significantly better retrieval accuracy for visual content. # Getting Started Source: https://morphik.ai/docs/getting-started Get up and running with Morphik On a new tab, navigate to the [Morphik website](https://morphik.ai) and click the "Get Started" button on the top right. Sign up page Once you have signed up, you will be redirected to the Morphik Cloud dashboard. Click the "Create Application" button to create a new application. Enter your app name and click "Create Application". Create application dialog You should now see your application in the dashboard with "Copy URI" and "Copy Token" buttons: * **For Python SDK**: Click "Copy URI" * **For TypeScript/API**: Click "Copy Token" Created application with copy buttons That's it! You're ready to use Morphik now :) ## Using Morphik While most users integrate Morphik into their applications via our SDKs and APIs, we also provide a comprehensive web interface that serves as both a playground and management console. ### Morphik Web Interface The Morphik web interface provides a complete view of your application: Morphik Application Interface In the web interface, you can: * 📄 **Browse and manage** all your documents and folders * 💬 **Use the Chat and Agent** for interactive queries and complex tasks * 🔗 **Visualize Knowledge Graphs** to understand document relationships * 📊 **Monitor logs** and track API usage * 💳 **View billing and usage details** for your account * ⚙️ **Configure settings** and manage your application This interface is perfect for testing queries, debugging, and getting familiar with Morphik's capabilities before integrating it into your code. ### Using Morphik via Code For production use, you'll want to integrate Morphik using our SDKs or API: ```bash theme={null} python3.12 -m venv .venv ``` ```bash theme={null} source .venv/bin/activate ``` ```bash theme={null} pip install morphik ``` ```python theme={null} from morphik import Morphik # Initialize the Morphik client morphik = Morphik(uri="your-morphik-uri") # Ingest a file doc = morphik.ingest_file(file_path="super/complex/file.pdf") doc.wait_for_completion() # Query the file response = morphik.query(query="What percentage of Morphik users are building something cool?") print(response) # Responds with 100% :) ``` You can find our entire SDK documentation [here](/python-sdk/morphik). ```bash theme={null} npm install morphik # or yarn add morphik ``` ```typescript theme={null} import Morphik from 'morphik'; import * as fs from 'fs'; // Initialize the Morphik client // Copy token from dashboard const morphik = new Morphik({ apiKey: 'your-morphik-token' }); // Ingest a file const file = fs.createReadStream('super/complex/file.pdf'); const doc = await morphik.ingest.ingestFile({ file }); // Wait for processing to complete await new Promise(resolve => setTimeout(resolve, 5000)); // Query the file const response = await morphik.query.generateCompletion({ query: 'What percentage of Morphik users are building something cool?' }); console.log(response.completion); // Responds with 100% :) ``` You can find our API reference with SDK examples [here](../api-reference/getting-started). The bearer token can be extracted directly from your Morphik URI. The URI has the following format: `morphik://:@` The middle part between the colon and the @ symbol is your bearer token. When making requests to the Morphik API, include your bearer token in the `Authorization` header: ```bash theme={null} curl -X 'POST' 'https://api.morphik.ai/documents?skip=0&limit=10000' \ -H "Authorization: Bearer " -H 'accept: application/json' \ -H 'Content-Type: application/json' \ -d '{}' ``` You can find our complete API reference [here](/api-reference/getting-started). You can find more information about MCP [here](/using-morphik/mcp). ## Community Support We have an open community with lots of discussion where you can get help, report bugs, and share your experiences with Morphik. If you need assistance or want to contribute, please join our community! Get help, report bugs, and connect with other Morphik users. ## Next Steps Now that you have the server running, you can explore the different ways to interact with the server. Configure Morphik using the `morphik.toml` file. Use the API to interact with the server. Use the Python SDK to interact with the server. Use the TypeScript/JavaScript SDK to interact with the server. # What is Morphik? Source: https://morphik.ai/docs/introduction An overview of Morphik Morphik is a database that makes it easy to create fast, versatile, and production-ready AI apps and agents. Key features include: * **First class support for Unstructured Data**: Unlike traditional databases, Morphik allows users to directly ingest unstructured data of all forms - including (but not limited to) videos and PDFs. We've built research-driven custom algorithms to ensure state-of-the-art retrieval accuracy. * **Out of the box MCP support**: Morphik has [built in support](/using-morphik/mcp) for [Model Context Protocol](https://modelcontextprotocol.io/introduction) - so you can integrate your knowledge with any MCP client in a single click. * **User and Folder Scoping**: Organize and isolate your data with multi-user and folder-based access controls. Create logical boundaries for different projects or user groups while maintaining a unified database. * **Completely source-available**: You can check out Morphik core code [here!](https://github.com/morphik-org/morphik-core/). * **Flexible Model Registry**: Easily register and use hundreds of different AI models across your application with our new registered models approach. Mix and match models by task (e.g., use smaller models for simpler tasks, powerful models for complex reasoning) through a simple configuration. * **Creating Apps**: Use [Creating Apps](/api-reference/creating-apps) to provision isolated apps. Each app keeps data separated and has its own Morphik URI, which is useful when you want per-customer or per-project isolation. Check out our [GitHub repository](https://github.com/morphik-org/morphik-core/) and please give us a star ⭐ if you like what we're building! Get started with Morphik with [our guide](/getting-started), or jump straight to the [Core Functions](/core-functions/ingest-file) to learn the basic building blocks: ingesting files, retrieving chunks, listing documents, and more. Want to wire Morphik tools into your own loop? See the [Build a Retrieval Agent with Morphik Tools](/cookbooks/agent-workflows#build-a-retrieval-agent-with-morphik-tools) walkthrough. Start using Morphik now! Customize Morphik to your liking using the `morphik.toml` file. Get help, report bugs, and connect with other Morphik users in our open community. An explanation of the underlying ideas behind RAG # How do I set up RAG? Source: https://morphik.ai/docs/knowledge-base/how-do-i-set-up-rag Overview of building a retrieval augmented generation workflow Retrieval Augmented Generation (RAG) combines search with language models to provide contextually rich responses. A typical setup involves: 1. Ingesting your documents and generating embeddings. 2. Storing those embeddings in a vector database. 3. Retrieving relevant chunks based on user queries. 4. Passing the retrieved context to your language model for generation. Morphik streamlines these steps by providing ingestion utilities, vector storage, and easy retrieval APIs so you can focus on building your application. ```python theme={null} from morphik import Morphik # Initialize the client db = Morphik("morphik://owner_id:token@api.morphik.ai") # 1. Ingest a document doc = db.ingest_file("document.pdf") # 2. Retrieve and generate with context response = db.query("What topics are covered?", k=4) print(response.text) ``` ### Related questions * **Q:** What are the steps to set up a RAG pipeline with Morphik? **A:** The key steps are: (1) Initialize the Morphik client, (2) Ingest your documents using `ingest_file()`, and (3) Query using `query()` with your question and desired number of results. * **Q:** How can I quickly build a retrieval-augmented generation workflow?\ **A:** Use Morphik's built-in RAG capabilities by following the code example above. The `query()` method handles both retrieval and generation in one step when you provide a question and set `k` for the number of relevant chunks to retrieve. * **Q:** What is the easiest way to implement RAG in my application?\ **A:** The simplest approach is to use Morphik's unified API which handles document processing, embedding, and querying. Just ingest your documents and call `query()` with natural language questions to get AI-generated answers with source citations. # How do I handle multi-modal inputs? Source: https://morphik.ai/docs/knowledge-base/how-to-handle-multi-modal-inputs Combining text, images, and other data types in a search system Modern applications often need to search across different data types like PDFs, images, and transcribed audio. Managing those sources separately can be cumbersome. Morphik accepts multiple modalities out of the box, storing them in a unified index so you can search across all your data with one query. Simply ingest each item with `ingest_file` or `ingest_text`: ```python theme={null} from morphik import Morphik db = Morphik() db.ingest_file("diagram.png") db.ingest_text("README.md") results = db.retrieve_docs(query="project overview") ``` ### Related questions * **Q:** How do I combine text and images in a single search query?\ **A:** Simply include both text and image files in your ingestion process. Morphik will automatically process and index each file according to its type. You can then search across all modalities with a single query, and the system will return the most relevant results regardless of format. * **Q:** What file formats does Morphik support for multi-modal ingestion?\ **A:** Morphik supports a wide range of formats including PDFs, images (PNG, JPG, etc.), and text files. The system automatically detects and processes each file type appropriately, extracting text from documents and generating embeddings for all content. * **Q:** Can I search audio transcripts together with documents?\ **A:** Yes, once audio files are transcribed to text, you can search them alongside other documents. Use the same `ingest_file` method for both audio transcripts and other document types, and they'll be included in unified search results. # How can I improve retrieval accuracy? Source: https://morphik.ai/docs/knowledge-base/how-to-improve-retrieval-accuracy Best practices for getting the most relevant results from your data Quality retrieval depends on thoughtful preprocessing and query handling. Consider: * Chunking documents into meaningful sections. * Choosing the right embedding model for your domain. * Applying metadata filters or reranking to refine results. Morphik supports custom chunk sizes, multiple embedding models, and powerful filtering so you can tune retrieval to your needs. For example: ```python theme={null} from morphik import Morphik db = Morphik() # Query with filters and reranking docs = db.query( "renewable energy projects", filters={"category": "energy"}, k=8, use_reranking=True, ) ``` ### Related questions * **Q:** What embedding model should I choose for technical documentation?\ **A:** For technical documentation, choose an embedding model trained on technical or scientific text. The best choice depends on your specific domain, but models like `text-embedding-3-large` or domain-specific variants often perform well for technical content. * **Q:** How can metadata filtering improve search precision?\ **A:** Metadata filtering allows you to narrow down search results by document attributes like creation date, author, or category. This is particularly useful when you know certain metadata about the documents you're looking for, as it helps eliminate irrelevant results before semantic matching. * **Q:** When should I enable reranking for better results?\ **A:** Enable reranking when you need higher precision in your top results. Reranking is especially valuable when the initial vector search returns many similar results, as it uses more sophisticated algorithms to reorder them based on relevance to your query. # How should I chunk my documents? Source: https://morphik.ai/docs/knowledge-base/how-to-manage-document-chunking Guidance on splitting data for optimal retrieval Choosing an effective chunking strategy balances context length with retrieval precision. Overly large chunks may dilute relevance, while tiny chunks can miss important context. Morphik lets you control chunk sizes in your `morphik.toml` configuration. Tweak `parser.chunk_size` and `parser.chunk_overlap` to balance context and recall: ```toml theme={null} [parser] chunk_size = 1000 chunk_overlap = 200 ``` Re-ingest documents after updating these settings to apply the new configuration. ### Related questions * **Q:** What chunk size works best for large PDFs?\ **A:** For large PDFs, a chunk size of 1000-2000 characters often works well, as it provides enough context while maintaining retrieval precision. However, the ideal size depends on your content and use case - technical documents might benefit from larger chunks, while conversational text might work better with smaller ones. * **Q:** How does chunk overlap affect answer quality?\ **A:** Chunk overlap (typically 10-20% of chunk size) helps maintain context between chunks and prevents important information from being split across chunk boundaries. This is particularly important for questions that might span multiple sections of a document. * **Q:** Do I need to re-ingest data after changing chunk settings?\ **A:** Yes, any changes to chunk size or overlap require re-ingesting your documents, as these settings determine how the text is processed and indexed. The system needs to recreate the document chunks with your new settings to ensure proper retrieval. # How do I perform search over documents? Source: https://morphik.ai/docs/knowledge-base/how-to-perform-search-over-documents Techniques for searching document collections efficiently Effective document search relies on representing your data in a way that captures meaning. Common approaches include keyword search and vector similarity search. With Morphik, you can ingest text, images, and other modalities. Use the `retrieve_docs` function for a simple vector similarity search or `query` to combine retrieval with language model generation: ```python theme={null} from morphik import Morphik db = Morphik() # Retrieve top matching documents docs = db.retrieve_docs(query="latest sales figures", k=3) # Or generate an answer from the documents answer = db.query("summarize the trends", k=3) print(answer.text) ``` ### Related questions * **Q:** What is the difference between keyword and vector search?\ **A:** Keyword search matches exact terms via an inverted index, while vector search compares dense embeddings to capture semantic similarity even when different words are used. * **Q:** How can I limit search to a specific document category?\ **A:** Pass a `filters` dictionary when calling `retrieve_docs` or `query`, e.g. `filters={"category": "finance"}`, to restrict results to documents with matching metadata. * **Q:** When should I use `query` instead of `retrieve_docs`?\ **A:** Use `query` when you need the language model to read the retrieved docs and generate a synthesized answer; use `retrieve_docs` when you only need the raw documents. # Local Inference Source: https://morphik.ai/docs/local-inference Run Morphik completely offline with local embedding and completion models Morphik comes with built-in support for running **both embeddings and completions** locally, ensuring your data never leaves your machine. Choose between two powerful local inference engines: * **Lemonade** - Windows-only, optimized for AMD GPUs and NPUs * **Ollama** - Cross-platform (Windows, macOS, Linux), supports various hardware Both are pre-configured in Morphik and can be selected through the UI or configuration file. ## Why Local Inference? Running models locally provides several key advantages: * **Complete Privacy**: Your data never leaves your machine * **No API Costs**: Eliminate ongoing API expenses * **Low Latency**: No network round-trips for inference * **Offline Capability**: Work without internet connectivity * **Hardware Acceleration**: Leverage your local GPU, NPU, or specialized AI processors
AMD 🍋

Lemonade

Run embeddings & completions locally with AMD GPU/NPU acceleration

Lemonade SDK provides high-performance local inference on Windows, with optimizations for AMD hardware. It exposes an OpenAI-compatible API and is **already configured in Morphik**. **Built-in Support**: Lemonade models are pre-configured in `morphik.toml` for both embeddings and completions. Simply install Lemonade Server and select the models in the UI. ### System Requirements * **Windows 10/11 only** (x86/x64) * **8GB+ RAM** (16GB recommended) * **Python 3.10+** * **Optional but recommended**: * AMD Ryzen AI 300 series (NPU acceleration) * AMD Radeon 7000/9000 series (GPU acceleration) ### Quick Start Download and install Lemonade from the official site: [lemonade-server.ai](https://lemonade-server.ai/). Start the Lemonade server following their documentation. Make sure it is running and note the port. The API is OpenAI-compatible (e.g., `/api/v1/models`). ### Option 1: Using the UI (Recommended) 1. Open the Morphik UI and go to Settings → API Keys 2. Select "Lemonade" (🍋). No API key is required 3. Enter the host and port where Lemonade is running Lemonade provider settings with host and port 4. Open Chat and use the model selector pill (top left) to pick a Lemonade model Chat model selector showing Lemonade models Running inside Docker? Use `host.docker.internal` instead of `localhost` for the host field. If you are not using a vision-capable model, turn off ColPali in chat settings (settings → ColPali) to avoid vision-dependent paths. ### Option 2: Edit morphik.toml You can also set Lemonade models directly in `morphik.toml` so they're used by default. Ensure the `api_base` points to your Lemonade server: ```toml theme={null} lemonade_qwen = { model_name = "openai/Qwen2.5-VL-7B-Instruct-GGUF", api_base = "http://localhost:8020/api/v1", vision = true } lemonade_embedding = { model_name = "openai/nomic-embed-text-v1-GGUF", api_base = "http://localhost:8020/api/v1" } [completion] model = "lemonade_qwen" [embedding] model = "lemonade_embedding" ``` If your system has under 16GB RAM, prefer models under \~4B parameters or smaller quantizations (e.g., Q4/Q5). Larger models may fail to load or will be very slow on low-memory systems. ### Performance Tips * **Model Quantization**: Use GGUF quantized models for better performance * **Low-memory systems**: Under 16GB RAM, prefer models under 4B parameters * **Hardware Acceleration**: Automatically detects and uses AMD GPUs/NPUs when available * **Memory Management**: Models are cached after first download ### Troubleshooting * Verify server health: `curl http://localhost:8020/health` * List models: `curl http://localhost:8020/api/v1/models` * For Docker: Use `host.docker.internal` instead of `localhost` * Check firewall settings for port 8020 * Ensure sufficient disk space (5-15GB per model) * Try smaller quantized versions (Q4, Q5) * Check model compatibility with `lemonade list` * Use GGUF quantized models for better performance * Monitor GPU/NPU usage with system tools * Adjust batch size and context length in model config
Ollama

Ollama - All Platforms

Run embeddings & completions locally on Windows, macOS, or Linux

Ollama provides cross-platform local inference for both embeddings and completions. It's **already configured in Morphik** and supports various hardware accelerators. **Built-in Support**: Ollama models are pre-configured in `morphik.toml` for both embeddings and completions. Simply install Ollama and select the models in the UI. ### System Requirements * **macOS**: Apple Silicon (M1/M2/M3) or Intel Mac with 8GB+ RAM * **Linux**: x86\_64 or ARM64, 8GB+ RAM, optional NVIDIA GPU * **Windows**: Windows 10/11, 8GB+ RAM, optional NVIDIA GPU ### Quick Start ```bash theme={null} brew install ollama # Or: curl -fsSL https://ollama.com/install.sh | sh ``` ```bash theme={null} curl -fsSL https://ollama.com/install.sh | sh ``` Download installer from [ollama.com/download](https://ollama.com/download/windows) ```bash theme={null} # Start Ollama service ollama serve ``` Or use Docker Compose with Morphik: ```bash theme={null} docker compose --profile ollama -f docker-compose.run.yml up -d ``` ### Option 1: Using the UI (Recommended) 1. Open Morphik UI and navigate to Settings 2. Select Ollama models from the dropdown for: * **Completion Model**: `ollama_qwen_vision` or `ollama_llama_vision` * **Embedding Model**: `ollama_embedding` (nomic-embed-text) ### Option 2: Edit morphik.toml Morphik comes with pre-configured Ollama models: ```toml theme={null} # Already configured in morphik.toml ollama_qwen_vision = { model_name = "ollama_chat/qwen2.5vl:latest", api_base = "http://localhost:11434", vision = true } ollama_embedding = { model_name = "ollama/nomic-embed-text", api_base = "http://localhost:11434" } # To use Ollama as default: [completion] model = "ollama_qwen_vision" [embedding] model = "ollama_embedding" ``` When running Morphik in Docker, change `localhost` to `ollama:11434` if using the Ollama profile, or `host.docker.internal:11434` if running Ollama separately. Pull the pre-configured models: ```bash theme={null} # For embeddings (required for RAG) ollama pull nomic-embed-text # For completions (choose one) ollama pull qwen2.5vl:latest # Vision-capable, 7B ollama pull llama3.2-vision # Vision-capable, 11B ollama pull qwen2:1.5b # Text-only, fast ``` Then select them in the UI chat interface! ### Hardware Acceleration **Apple Silicon (M1/M2/M3)** * Ollama automatically uses Metal for GPU acceleration * No additional configuration needed * Excellent performance on unified memory architecture **NVIDIA GPUs** * Install CUDA drivers (11.8+ recommended) * Ollama auto-detects and uses available GPUs * Monitor usage: `nvidia-smi` **AMD GPUs (Linux)** * ROCm support is experimental * Set environment variable: `HSA_OVERRIDE_GFX_VERSION=10.3.0` ### Performance Tuning **Memory Management** ```bash theme={null} # Set GPU memory limit (NVIDIA) OLLAMA_MAX_VRAM=8GB ollama serve # Adjust number of parallel requests OLLAMA_NUM_PARALLEL=4 ollama serve # Keep models loaded in memory OLLAMA_KEEP_ALIVE=30m ollama serve ``` **Model Quantization** Ollama supports various quantization levels: * `q4_0` - 4-bit quantization (smallest, fastest) * `q5_1` - 5-bit quantization (balanced) * `q8_0` - 8-bit quantization (best quality) ```bash theme={null} # Pull specific quantization ollama pull llama3.2:3b-q4_0 # Smaller, faster ollama pull llama3.2:3b-q8_0 # Better quality ``` ### Monitoring & Management **Check Status** ```bash theme={null} # List loaded models ollama list # View running models ollama ps # Check API health curl http://localhost:11434/api/tags ``` **Resource Usage** ```bash theme={null} # Monitor in real-time watch -n 1 ollama ps # Check model details ollama show llama3.2 --modelfile ``` ### Creating Custom Models Create specialized models for your use case: ```dockerfile theme={null} # Modelfile FROM llama3.2:3b # Set parameters PARAMETER temperature 0.1 PARAMETER num_ctx 4096 # Add system prompt SYSTEM """You are a helpful assistant specialized in document analysis and information retrieval. Always provide accurate, concise responses based on the provided context.""" ``` Build and use: ```bash theme={null} ollama create morphik-assistant -f Modelfile ollama run morphik-assistant ``` # add_document_to_folder Source: https://morphik.ai/docs/python-sdk/add_document_to_folder Add an existing document to a folder ```python theme={null} def add_document_to_folder( folder_id_or_name: str, document_id: str, ) -> Dict[str, str] ``` ```python theme={null} async def add_document_to_folder( folder_id_or_name: str, document_id: str, ) -> Dict[str, str] ``` ## Parameters * `folder_id_or_name` (str): Folder identifier. Accepts the folder's UUID, name, or canonical path (e.g., `/projects/alpha/specs`; leading slash optional). * `document_id` (str): Identifier of the document to move into the folder. ## Returns * `Dict[str, str]`: Dictionary with `status` and `message` describing the result. ## Examples ```python theme={null} from morphik import Morphik db = Morphik() folder = db.get_folder("marketing_docs") db.add_document_to_folder(folder.id, "doc_123") db.add_document_to_folder("/projects/alpha/specs", "doc_456") ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: folder = await db.get_folder("marketing_docs") await db.add_document_to_folder(folder.id, "doc_123") await db.add_document_to_folder("/projects/alpha/specs", "doc_456") ``` # batch_get_chunks Source: https://morphik.ai/docs/python-sdk/batch_get_chunks Retrieve specific chunks by their document ID and chunk number ```python theme={null} def batch_get_chunks( sources: List[Union[ChunkSource, Dict[str, Any]]], folder_name: Optional[Union[str, List[str]]] = None, use_colpali: bool = True, output_format: Optional[str] = None, ) -> List[FinalChunkResult] ``` ```python theme={null} async def batch_get_chunks( sources: List[Union[ChunkSource, Dict[str, Any]]], folder_name: Optional[Union[str, List[str]]] = None, use_colpali: bool = True, output_format: Optional[str] = None, ) -> List[FinalChunkResult] ``` ## Parameters * `sources` (List\[Union\[ChunkSource, Dict\[str, Any]]]): List of ChunkSource objects or dictionaries with document\_id and chunk\_number * `folder_name` (str | List\[str], optional): Optional folder scope. Accepts canonical paths or a list of paths/names. * `use_colpali` (bool, optional): Whether to request multimodal chunks when available. Defaults to True. * `output_format` (str, optional): Controls how image chunks are returned. Set to `"url"` to receive presigned URLs; omit or set to `"base64"` (default) to receive base64 content. ## Returns * `List[FinalChunkResult]`: List of chunk results ## Examples ```python theme={null} from morphik import Morphik from morphik.models import ChunkSource db = Morphik() # Using dictionaries sources = [ {"document_id": "doc_123", "chunk_number": 0}, {"document_id": "doc_456", "chunk_number": 2} ] # Or using ChunkSource objects sources = [ ChunkSource(document_id="doc_123", chunk_number=0), ChunkSource(document_id="doc_456", chunk_number=2) ] chunks = db.batch_get_chunks(sources) for chunk in chunks: print(f"Chunk from {chunk.document_id}, number {chunk.chunk_number}: {chunk.content[:50]}...") ``` ```python theme={null} from morphik import AsyncMorphik from morphik.models import ChunkSource async with AsyncMorphik() as db: # Using dictionaries sources = [ {"document_id": "doc_123", "chunk_number": 0}, {"document_id": "doc_456", "chunk_number": 2} ] # Or using ChunkSource objects sources = [ ChunkSource(document_id="doc_123", chunk_number=0), ChunkSource(document_id="doc_456", chunk_number=2) ] chunks = await db.batch_get_chunks(sources) for chunk in chunks: print(f"Chunk from {chunk.document_id}, number {chunk.chunk_number}: {chunk.content[:50]}...") ``` ## FinalChunkResult Properties Each `FinalChunkResult` object in the returned list has the following properties: * `content` (str | PILImage): Chunk content (text or image) * `score` (float): Relevance score * `document_id` (str): Parent document ID * `chunk_number` (int): Chunk sequence number * `metadata` (Dict\[str, Any]): Document metadata * `content_type` (str): Content type * `filename` (Optional\[str]): Original filename * `download_url` (Optional\[str]): URL to download full document # batch_get_documents Source: https://morphik.ai/docs/python-sdk/batch_get_documents Retrieve multiple documents by their IDs in a single batch operation ```python theme={null} def batch_get_documents( document_ids: List[str], folder_name: Optional[Union[str, List[str]]] = None, ) -> List[Document] ``` ```python theme={null} async def batch_get_documents( document_ids: List[str], folder_name: Optional[Union[str, List[str]]] = None, ) -> List[Document] ``` ## Parameters * `document_ids` (List\[str]): List of document IDs to retrieve * `folder_name` (str | List\[str], optional): Optional folder scope. Accepts canonical paths or a list of paths/names. ## Returns * `List[Document]`: List of document metadata for found documents ## Examples ```python theme={null} from morphik import Morphik db = Morphik() docs = db.batch_get_documents(["doc_123", "doc_456", "doc_789"]) for doc in docs: print(f"Document {doc.external_id}: {doc.metadata.get('title')}") ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: docs = await db.batch_get_documents(["doc_123", "doc_456", "doc_789"]) for doc in docs: print(f"Document {doc.external_id}: {doc.metadata.get('title')}") ``` ## Document Properties Each `Document` object in the returned list has the following properties: * `external_id` (str): Unique document identifier * `content_type` (str): Content type of the document * `filename` (Optional\[str]): Original filename if available * `metadata` (Dict\[str, Any]): User-defined metadata * `storage_info` (Dict\[str, str]): Storage-related information * `system_metadata` (Dict\[str, Any]): System-managed metadata * `chunk_ids` (List\[str]): IDs of document chunks * `folder_path` (Optional\[str]): Canonical folder path (includes nested parents when scoped) # close Source: https://morphik.ai/docs/python-sdk/close Close the HTTP session or client ```python theme={null} def close() -> None ``` ```python theme={null} async def close() -> None ``` ## Parameters None ## Returns None ## Examples ```python theme={null} from morphik import Morphik db = Morphik() # Perform operations doc = db.ingest_text("Sample content") # Close the session when done db.close() ``` ```python theme={null} import asyncio from morphik import AsyncMorphik async def main(): db = AsyncMorphik() # Perform operations doc = await db.ingest_text("Sample content") # Close the client when done await db.close() asyncio.run(main()) ``` ## Context Manager Alternative Instead of manually calling `close()`, you can use the Morphik client as a context manager: ```python theme={null} from morphik import Morphik with Morphik() as db: doc = db.ingest_text("Sample content") # Session is automatically closed when exiting the with block ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: doc = await db.ingest_text("Sample content") # Client is automatically closed when exiting the with block ``` # create_app Source: https://morphik.ai/docs/python-sdk/create_app Create a cloud app and return its authenticated URI ```python theme={null} def create_app( name: str, ) -> Dict[str, str] ``` ```python theme={null} async def create_app( name: str, ) -> Dict[str, str] ``` ## Parameters * `name` (str): App display name ## Returns * `Dict[str, str]`: Response containing the authenticated URI and app metadata ## Examples ```python theme={null} from morphik import Morphik db = Morphik() resp = db.create_app(name="demo") print(resp) ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: resp = await db.create_app(name="demo") print(resp) ``` ## Notes * `generate_cloud_uri` is a deprecated alias for this method. * The SDK only accepts `name` for app creation. # create_folder Source: https://morphik.ai/docs/python-sdk/create_folder Create a new folder for organizing documents ```python theme={null} def create_folder( name: str, description: Optional[str] = None, full_path: Optional[str] = None, parent_id: Optional[str] = None, ) -> Folder ``` ```python theme={null} async def create_folder( name: str, description: Optional[str] = None, full_path: Optional[str] = None, parent_id: Optional[str] = None, ) -> Folder ``` ## Parameters * `name` (str): Folder name (leaf segment when using nested paths). If `full_path` is omitted, this becomes the canonical path. * `description` (str, optional): Optional description of the folder. * `full_path` (str, optional): Canonical folder path (e.g., `"/projects/alpha/specs"`). Leading slash is optional; parents are created automatically. * `parent_id` (str, optional): Explicit parent folder ID. Usually not needed—`full_path` handles hierarchy creation. ## Returns * `Folder`: Newly created folder object. ## Examples ```python theme={null} from morphik import Morphik db = Morphik() folder = db.create_folder("marketing_docs", description="All marketing collateral") # Create a nested folder (parents auto-created) nested = db.create_folder( name="specs", full_path="/projects/alpha/specs", description="All project specs", ) print(nested.full_path) # "/projects/alpha/specs" ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: folder = await db.create_folder("marketing_docs", description="All marketing collateral") nested = await db.create_folder( name="specs", full_path="/projects/alpha/specs", description="All project specs", ) print(nested.full_path) # "/projects/alpha/specs" ``` # delete_app Source: https://morphik.ai/docs/python-sdk/delete_app Delete a cloud app by name ```python theme={null} def delete_app( app_name: str, ) -> Dict[str, Any] ``` ```python theme={null} async def delete_app( app_name: str, ) -> Dict[str, Any] ``` ## Parameters * `app_name` (str): Name of the app to delete ## Returns * `Dict[str, Any]`: API response with delete status ## Examples ```python theme={null} db.delete_app("staging-app") ``` # delete_document Source: https://morphik.ai/docs/python-sdk/delete_document Delete a document and all its associated data ## Usage ```python theme={null} from morphik import Morphik db = Morphik() # Delete a document by its ID result = db.delete_document("doc_123456") print(result["message"]) # "Document doc_123456 deleted successfully" ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: # Delete a document by its ID result = await db.delete_document("doc_123456") print(result["message"]) # "Document doc_123456 deleted successfully" ``` ## Parameters * `document_id` (str): ID of the document to delete ## Returns A dictionary containing information about the deletion operation: * `message` (str): A success message indicating the document was deleted * `document_id` (str): The ID of the deleted document * Additional fields may be present with more details about the operation ## Description This method deletes a document and all its associated data, including: * Document metadata in the database * Document content in storage * Document chunks and embeddings in the vector store This operation is permanent and cannot be undone. ## Finding Document IDs If you don't know the document ID, you can use other methods to find it: ```python theme={null} # List documents to find IDs docs = db.list_documents(limit=10) for doc in docs: print(f"ID: {doc.external_id}, Filename: {doc.filename}") # Or get document by filename doc = db.get_document_by_filename("report.pdf") document_id = doc.external_id # Then delete it result = db.delete_document(document_id) ``` ```python theme={null} # List documents to find IDs docs = await db.list_documents(limit=10) for doc in docs: print(f"ID: {doc.external_id}, Filename: {doc.filename}") # Or get document by filename doc = await db.get_document_by_filename("report.pdf") document_id = doc.external_id # Then delete it result = await db.delete_document(document_id) ``` ## Notes * For convenience, you can also use the [delete\_document\_by\_filename](/python-sdk/delete_document_by_filename) method if you know the filename but not the ID. * This operation requires appropriate permissions for the document. # delete_document_by_filename Source: https://morphik.ai/docs/python-sdk/delete_document_by_filename Delete a document by its filename ## Usage ```python theme={null} from morphik import Morphik db = Morphik() # Delete a document by its filename result = db.delete_document_by_filename("report.pdf") print(result["message"]) # "Document doc_123456 deleted successfully" ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: # Delete a document by its filename result = await db.delete_document_by_filename("report.pdf") print(result["message"]) # "Document doc_123456 deleted successfully" ``` ## Parameters * `filename` (str): Filename of the document to delete ## Returns A dictionary containing information about the deletion operation: * `message` (str): A success message indicating the document was deleted * `document_id` (str): The ID of the deleted document * Additional fields may be present with more details about the operation ## Description This method is a convenience wrapper that: 1. Retrieves the document ID by filename using [get\_document\_by\_filename](/python-sdk/get_document_by_filename) 2. Deletes the document using [delete\_document](/python-sdk/delete_document) This operation is permanent and cannot be undone. ## Multiple Documents with Same Filename If multiple documents have the same filename, this method will delete the most recently updated one. To delete a specific document when duplicates exist, use [delete\_document](/python-sdk/delete_document) with the exact document ID instead. ```python theme={null} # To handle multiple documents with the same filename docs = db.list_documents() # Find all documents with a specific filename matching_docs = [doc for doc in docs if doc.filename == "report.pdf"] # Review them for doc in matching_docs: print(f"ID: {doc.external_id}, Created: {doc.system_metadata.get('created_at')}") # Delete a specific one by ID result = db.delete_document(matching_docs[0].external_id) ``` ```python theme={null} # To handle multiple documents with the same filename docs = await db.list_documents() # Find all documents with a specific filename matching_docs = [doc for doc in docs if doc.filename == "report.pdf"] # Review them for doc in matching_docs: print(f"ID: {doc.external_id}, Created: {doc.system_metadata.get('created_at')}") # Delete a specific one by ID result = await db.delete_document(matching_docs[0].external_id) ``` ## Notes * This operation requires appropriate permissions for the document. * If no document exists with the specified filename, a `ValueError` will be raised. # delete_folder Source: https://morphik.ai/docs/python-sdk/delete_folder Delete a folder and its documents ```python theme={null} def delete_folder( folder_id_or_name: str, ) -> Dict[str, Any] ``` ```python theme={null} async def delete_folder( folder_id_or_name: str, ) -> Dict[str, Any] ``` ## Parameters * `folder_id_or_name` (str): Folder identifier. Accepts the folder's UUID, name, or canonical path (e.g., `/projects/alpha/specs`; leading slash optional). ## Returns * `Dict[str, Any]`: Dictionary containing `status` and `message` describing the deletion. ## Examples ```python theme={null} from morphik import Morphik db = Morphik() db.delete_folder("marketing_docs") db.delete_folder("/projects/alpha/specs") db.delete_folder("bfd74128-8539-4050-8938-542d6ee68be0") ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: await db.delete_folder("marketing_docs") await db.delete_folder("/projects/alpha/specs") await db.delete_folder("bfd74128-8539-4050-8938-542d6ee68be0") ``` # extract_document_pages Source: https://morphik.ai/docs/python-sdk/extract_document_pages Extract specific pages from a document ```python theme={null} def extract_document_pages( document_id: str, start_page: int, end_page: int, ) -> DocumentPagesResponse ``` ```python theme={null} async def extract_document_pages( document_id: str, start_page: int, end_page: int, ) -> DocumentPagesResponse ``` ## Parameters * `document_id` (str): ID of the document to extract pages from * `start_page` (int): Starting page number (1-indexed) * `end_page` (int): Ending page number (1-indexed) ## Returns * `DocumentPagesResponse`: Object containing extracted pages with metadata ## Examples ```python theme={null} from morphik import Morphik db = Morphik() # Extract pages 1-3 from a document response = db.extract_document_pages( document_id="doc_123abc", start_page=1, end_page=3, ) print(f"Document ID: {response.document_id}") print(f"Extracted pages {response.start_page}-{response.end_page}") print(f"Total pages in document: {response.total_pages}") print(f"Number of pages extracted: {len(response.pages)}") # Pages are base64 encoded for i, page_content in enumerate(response.pages): print(f"Page {response.start_page + i}: {len(page_content)} chars") ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: # Extract pages 1-3 from a document response = await db.extract_document_pages( document_id="doc_123abc", start_page=1, end_page=3, ) print(f"Document ID: {response.document_id}") print(f"Extracted pages {response.start_page}-{response.end_page}") print(f"Total pages in document: {response.total_pages}") print(f"Number of pages extracted: {len(response.pages)}") # Pages are base64 encoded for i, page_content in enumerate(response.pages): print(f"Page {response.start_page + i}: {len(page_content)} chars") ``` ## DocumentPagesResponse Properties The `DocumentPagesResponse` object has the following properties: * `document_id` (str): ID of the document * `pages` (List\[str]): List of page contents as base64 encoded strings * `start_page` (int): Start page number (1-indexed) * `end_page` (int): End page number (1-indexed) * `total_pages` (int): Total number of pages in the document ## Notes * Page numbers are 1-indexed (first page is 1, not 0). * The `pages` list contains base64 encoded representations of each page. * Useful for extracting specific sections of large documents. # Folder Management Source: https://morphik.ai/docs/python-sdk/folders Organize and isolate data into logical folder groups in Morphik ## Overview Folders in Morphik provide a way to organize documents into logical groups. This is particularly useful for multi-project environments where you want to maintain separation between different contexts. Documents within a folder are isolated from those in other folders, allowing for clean organization and data separation. > ℹ️ All folder APIs accept **folder UUIDs, names, or canonical paths** (e.g., `"/projects/alpha/specs"`). Folder objects expose `full_path`, `parent_id`, `depth`, and `child_count`; documents expose `folder_path` to mirror server responses. ## Creating and Accessing Folders ```python theme={null} from morphik import Morphik db = Morphik() # Create a new folder folder = db.create_folder("marketing_docs") # Create a nested folder (parents created automatically) nested = db.create_folder(full_path="/projects/alpha/specs") # Access an existing folder by name/path or UUID folder = db.get_folder("/projects/alpha") folder_by_id = db.get_folder(folder.id) ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: # Create a new folder folder = await db.create_folder("marketing_docs") nested = await db.create_folder(full_path="/projects/alpha/specs") # Access an existing folder by name/path or UUID folder = await db.get_folder("/projects/alpha") folder_by_id = await db.get_folder(folder.id) ``` ## Operations Within a Folder Once you have a folder object, all operations performed on it are scoped to that folder. Documents created, retrieved, or manipulated will be contained within this folder's scope. ```python theme={null} # Get a folder folder = db.get_folder("marketing_docs") # Ingest a document into this folder doc = folder.ingest_text("New marketing strategy for Q3", filename="strategy_q3.txt", metadata={"department": "marketing", "quarter": "Q3"}) # Retrieve documents only from this folder marketing_docs = folder.list_documents() # Search documents only within this folder results = folder.retrieve_chunks("marketing strategy") # Query within the folder context response = folder.query("What is our Q3 marketing strategy?") ``` ```python theme={null} # Get a folder folder = db.get_folder("marketing_docs") # Ingest a document into this folder doc = await folder.ingest_text("New marketing strategy for Q3", filename="strategy_q3.txt", metadata={"department": "marketing", "quarter": "Q3"}) # Retrieve documents only from this folder marketing_docs = await folder.list_documents() # Search documents only within this folder results = await folder.retrieve_chunks("marketing strategy") # Query within the folder context response = await folder.query("What is our Q3 marketing strategy?") ``` ## Nested Folders and Scope Depth Folders can be nested arbitrarily. Use canonical paths (leading slash optional) to address them, and include descendant folders in retrieval/listing by setting `folder_depth`: ```python theme={null} folder = db.create_folder(full_path="/projects/alpha/specs") # Query across /projects/alpha and all children chunks = db.retrieve_chunks( query="design notes", folder_name="/projects/alpha", folder_depth=-1, # -1: all descendants, 0/None: exact only, n>0: include up to n levels ) ``` Folder-scoped helpers inherit the path automatically, so `folder.retrieve_chunks(..., folder_depth=-1)` will include its children. ## Expanding Scope with Additional Folders Folder-scoped retrieval/list/query helpers accept `additional_folders` to include extra folders in the same request: ```python theme={null} folder = db.get_folder("/projects/alpha") # Search across /projects/alpha plus shared archives results = folder.retrieve_chunks( "design notes", additional_folders=["/shared", "/archive"], ) ``` ## Folder Methods All the core document operations available on the main Morphik client are also available on folder objects, but they are automatically scoped to the specific folder: * `ingest_text` - Ingest text content into this folder * `ingest_file` - Ingest a file into this folder * `ingest_files` - Ingest multiple files into this folder * `ingest_directory` - Ingest all files from a directory into this folder * `retrieve_chunks` - Retrieve chunks matching a query from this folder (supports [reverse image search](/python-sdk/retrieve_chunks#reverse-image-search)) * `retrieve_docs` - Retrieve documents matching a query from this folder * `query` - Generate a completion using context from this folder (supports `llm_config` parameter for custom LLM configuration) * `list_documents` - List all documents in this folder * `batch_get_documents` - Get multiple documents by their IDs from this folder * `batch_get_chunks` - Get specific chunks by source from this folder * `get_info` - Fetch the latest folder metadata from the API * `get_summary` - Fetch the latest folder summary * `upsert_summary` - Create or update the folder summary * `delete_document_by_filename` - Delete a document by filename from this folder ## Managing Existing Documents and Folders You can move previously ingested documents into a folder, remove them, or delete the entire folder. The SDK methods accept a folder UUID, name, or canonical path. ```python theme={null} from morphik import Morphik db = Morphik() folder = db.get_folder("marketing_docs") document_id = "doc_123" # Add an existing document to the folder (name/path or UUID works) db.add_document_to_folder(folder.id, document_id) db.add_document_to_folder("/projects/alpha/specs", document_id) # Remove the document from the folder db.remove_document_from_folder("marketing_docs", document_id) # Delete the folder (also removes its documents) db.delete_folder(folder.id) ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: folder = await db.get_folder("marketing_docs") document_id = "doc_123" await db.add_document_to_folder(folder.id, document_id) await db.add_document_to_folder("/projects/alpha/specs", document_id) await db.remove_document_from_folder("marketing_docs", document_id) await db.delete_folder(folder.id) ``` ## Using Custom LLM Configuration with Folders You can pass a custom LLM configuration when querying within a folder: ```python theme={null} from morphik import Morphik db = Morphik() folder = db.get_folder("research") # Use a specific model for this query response = folder.query( "Summarize the latest findings", llm_config={ "model": "claude-3-opus-20240229", "temperature": 0.5 } ) print(response.completion) ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: folder = db.get_folder("research") # Use a specific model for this query response = await folder.query( "Summarize the latest findings", llm_config={ "model": "claude-3-opus-20240229", "temperature": 0.5 } ) print(response.completion) ``` ## Example: Project Document Management A common use case for folders is separating different projects. Here's an example of how to organize project documentation: ```python theme={null} from morphik import Morphik from pathlib import Path db = Morphik() # Create folders for different projects project_a = db.create_folder("project_a") project_b = db.create_folder("project_b") # Ingest project documentation into respective folders project_a.ingest_directory(Path("/path/to/project_a_docs"), recursive=True, metadata={"project": "Project A"}) project_b.ingest_directory(Path("/path/to/project_b_docs"), recursive=True, metadata={"project": "Project B"}) # Query is scoped to just Project A documents project_a_response = project_a.query("What are the key milestones for this project?") # Query is scoped to just Project B documents project_b_response = project_b.query("What are the technical requirements?") ``` ```python theme={null} from morphik import AsyncMorphik from pathlib import Path async with AsyncMorphik() as db: # Create folders for different projects project_a = db.create_folder("project_a") project_b = db.create_folder("project_b") # Ingest project documentation into respective folders await project_a.ingest_directory(Path("/path/to/project_a_docs"), recursive=True, metadata={"project": "Project A"}) await project_b.ingest_directory(Path("/path/to/project_b_docs"), recursive=True, metadata={"project": "Project B"}) # Query is scoped to just Project A documents project_a_response = await project_a.query("What are the key milestones for this project?") # Query is scoped to just Project B documents project_b_response = await project_b.query("What are the technical requirements?") ``` ## Accessing a Folder's User Scope You can further scope operations within a folder to a specific user by using the `signin` method on the folder object: ```python theme={null} # Get a folder project_folder = db.get_folder("project_x") # Create a user scope within this folder user_in_project = project_folder.signin("user_123") # This document is accessible only to user_123 within project_x doc = user_in_project.ingest_text("User-specific project notes") ``` ```python theme={null} # Get a folder project_folder = db.get_folder("project_x") # Create a user scope within this folder user_in_project = project_folder.signin("user_123") # This document is accessible only to user_123 within project_x doc = await user_in_project.ingest_text("User-specific project notes") ``` See [User Management](/python-sdk/users) for more details on working with user scopes. # generate_cloud_uri Source: https://morphik.ai/docs/python-sdk/generate_cloud_uri Deprecated alias for create_app `generate_cloud_uri` is a deprecated alias for [`create_app`](./create_app). Use `create_app` for all new integrations. See `create_app` for security notes about which fields are honored based on the token type. ```python theme={null} def generate_cloud_uri( name: str, ) -> Dict[str, str] ``` ```python theme={null} async def generate_cloud_uri( name: str, ) -> Dict[str, str] ``` # get_app_storage_usage Source: https://morphik.ai/docs/python-sdk/get_app_storage_usage Return storage usage metrics for the authenticated app ```python theme={null} def get_app_storage_usage() -> AppStorageUsageResponse ``` ```python theme={null} async def get_app_storage_usage() -> AppStorageUsageResponse ``` ## Returns * `AppStorageUsageResponse`: Storage usage details (chunk, raw, multivector, total, document count) ## Examples ```python theme={null} usage = db.get_app_storage_usage() print(usage.total_mb, usage.document_count) ``` # get_chat_history Source: https://morphik.ai/docs/python-sdk/get_chat_history Fetch the full message history for a specific chat conversation ```python theme={null} def get_chat_history(chat_id: str) -> List[Dict[str, Any]] ``` ```python theme={null} async def get_chat_history(chat_id: str) -> List[Dict[str, Any]] ``` ## Parameters * `chat_id` (str): Identifier of the conversation to retrieve. ## Returns * `List[Dict[str, Any]]`: A list of message dictionaries in chronological order. Each message dictionary contains: * `role` – either `"user"` or `"assistant"` * `content` – the original text of the message * `timestamp` – ISO-8601 timestamp string ## Example ```python theme={null} db = Morphik() history = db.get_chat_history("chat_123") for msg in history: print(f"[{msg['role']}] {msg['content']}") ``` ```python theme={null} async with AsyncMorphik() as db: history = await db.get_chat_history("chat_123") print(history[-1]) ``` # get_document Source: https://morphik.ai/docs/python-sdk/get_document Get document metadata by ID ```python theme={null} def get_document(document_id: str) -> Document ``` ```python theme={null} async def get_document(document_id: str) -> Document ``` ## Parameters * `document_id` (str): ID of the document ## Returns * `Document`: Document metadata ## Examples ```python theme={null} from morphik import Morphik db = Morphik() doc = db.get_document("doc_123") print(f"Title: {doc.metadata.get('title')}") print(f"Content Type: {doc.content_type}") print(f"Filename: {doc.filename}") ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: doc = await db.get_document("doc_123") print(f"Title: {doc.metadata.get('title')}") print(f"Content Type: {doc.content_type}") print(f"Filename: {doc.filename}") ``` ## Document Properties The `Document` object returned by this method has the following properties: * `external_id` (str): Unique document identifier * `content_type` (str): Content type of the document * `filename` (Optional\[str]): Original filename if available * `metadata` (Dict\[str, Any]): User-defined metadata * `storage_info` (Dict\[str, str]): Storage-related information * `system_metadata` (Dict\[str, Any]): System-managed metadata * `chunk_ids` (List\[str]): IDs of document chunks ## Document Methods The `Document` object also provides the following methods: * `update_with_text()`: Update the document with new text content * `update_with_file()`: Update the document with content from a file * `update_metadata()`: Update the document's metadata only * `update_with_text()`: Update the document with new text content (use with `await`) * `update_with_file()`: Update the document with content from a file (use with `await`) * `update_metadata()`: Update the document's metadata only (use with `await`) # get_document_by_filename Source: https://morphik.ai/docs/python-sdk/get_document_by_filename Get document metadata by filename ```python theme={null} def get_document_by_filename(filename: str) -> Document ``` ```python theme={null} async def get_document_by_filename(filename: str) -> Document ``` ## Parameters * `filename` (str): Filename of the document to retrieve ## Returns * `Document`: Document metadata ## Examples ```python theme={null} from morphik import Morphik db = Morphik() doc = db.get_document_by_filename("report.pdf") print(f"Document ID: {doc.external_id}") print(f"Content Type: {doc.content_type}") print(f"Metadata: {doc.metadata}") ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: doc = await db.get_document_by_filename("report.pdf") print(f"Document ID: {doc.external_id}") print(f"Content Type: {doc.content_type}") print(f"Metadata: {doc.metadata}") ``` ## Document Properties The `Document` object returned by this method has the following properties: * `external_id` (str): Unique document identifier * `content_type` (str): Content type of the document * `filename` (Optional\[str]): Original filename if available * `metadata` (Dict\[str, Any]): User-defined metadata * `storage_info` (Dict\[str, str]): Storage-related information * `system_metadata` (Dict\[str, Any]): System-managed metadata * `chunk_ids` (List\[str]): IDs of document chunks ## Document Methods The `Document` object also provides the following methods: * `update_with_text()`: Update the document with new text content * `update_with_file()`: Update the document with content from a file * `update_metadata()`: Update the document's metadata only * `update_with_text()`: Update the document with new text content (use with `await`) * `update_with_file()`: Update the document with content from a file (use with `await`) * `update_metadata()`: Update the document's metadata only (use with `await`) # get_document_download_url Source: https://morphik.ai/docs/python-sdk/get_document_download_url Generate a presigned URL to download a document's raw file content ```python theme={null} def get_document_download_url(document_id: str, expires_in: int = 3600) -> Dict[str, Any] ``` ```python theme={null} async def get_document_download_url(document_id: str, expires_in: int = 3600) -> Dict[str, Any] ``` ## Parameters * `document_id` (str): External ID of the document. * `expires_in` (int, optional): URL expiration time in seconds. Default is 3600 (1 hour). ## Returns * `Dict[str, Any]` containing: * `download_url` – presigned URL usable in a browser or `requests.get`. * `filename`, `content_type`, `expires_in`, `document_id` (echoed back). ## Example ```python theme={null} info = db.get_document_download_url(doc.external_id) print(info["download_url"]) ``` # get_document_file Source: https://morphik.ai/docs/python-sdk/get_document_file Download the raw file content of a document ```python theme={null} def get_document_file( document_id: str, ) -> bytes ``` ```python theme={null} async def get_document_file( document_id: str, ) -> bytes ``` ## Parameters * `document_id` (str): ID of the document to download ## Returns * `bytes`: Raw file content as bytes ## Examples ```python theme={null} from morphik import Morphik db = Morphik() # Download a document's raw file doc_id = "doc_123abc" file_content = db.get_document_file(doc_id) # Save to local file with open("downloaded_file.pdf", "wb") as f: f.write(file_content) print(f"Downloaded {len(file_content)} bytes") ``` ```python theme={null} from morphik import AsyncMorphik import aiofiles async with AsyncMorphik() as db: # Download a document's raw file doc_id = "doc_123abc" file_content = await db.get_document_file(doc_id) # Save to local file async with aiofiles.open("downloaded_file.pdf", "wb") as f: await f.write(file_content) print(f"Downloaded {len(file_content)} bytes") ``` ## Notes * This method returns the raw file bytes, which you can save to disk or process in memory. * For getting a downloadable URL instead of raw bytes, use [`get_document_download_url`](./get_document_download_url). * The returned bytes match the original file that was uploaded/ingested. # get_document_status Source: https://morphik.ai/docs/python-sdk/get_document_status Get the current processing status of a document ```python theme={null} def get_document_status( document_id: str, ) -> Dict[str, Any] ``` ```python theme={null} async def get_document_status( document_id: str, ) -> Dict[str, Any] ``` ## Parameters * `document_id` (str): ID of the document to check ## Returns * `Dict[str, Any]`: Status information including current status, potential errors, and other metadata ## Examples ```python theme={null} from morphik import Morphik db = Morphik() # Check document processing status status = db.get_document_status("doc_123abc") print(f"Status: {status.get('status')}") if status.get('error'): print(f"Error: {status.get('error')}") # Use in a polling loop import time while True: status = db.get_document_status("doc_123abc") if status.get('status') == 'completed': print("Document processing complete!") break elif status.get('status') == 'failed': print(f"Document processing failed: {status.get('error')}") break time.sleep(2) ``` ```python theme={null} from morphik import AsyncMorphik import asyncio async with AsyncMorphik() as db: # Check document processing status status = await db.get_document_status("doc_123abc") print(f"Status: {status.get('status')}") if status.get('error'): print(f"Error: {status.get('error')}") # Use in a polling loop while True: status = await db.get_document_status("doc_123abc") if status.get('status') == 'completed': print("Document processing complete!") break elif status.get('status') == 'failed': print(f"Document processing failed: {status.get('error')}") break await asyncio.sleep(2) ``` ## Notes * Common status values include: `"processing"`, `"completed"`, `"failed"` * This is a lightweight endpoint useful for checking progress without fetching the full document. * The SDK also provides a helper method for polling: see the document ingestion methods which can wait for completion automatically. # get_document_summary Source: https://morphik.ai/docs/python-sdk/get_document_summary Fetch the stored summary for a document ```python theme={null} def get_document_summary( document_id: str, ) -> Summary ``` ```python theme={null} async def get_document_summary( document_id: str, ) -> Summary ``` ## Parameters * `document_id` (str): ID of the document ## Returns * `Summary`: Stored summary payload for the document ## Examples ```python theme={null} from morphik import Morphik db = Morphik() summary = db.get_document_summary("doc_123") print(summary.content) ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: summary = await db.get_document_summary("doc_123") print(summary.content) ``` # get_folder Source: https://morphik.ai/docs/python-sdk/get_folder Retrieve a folder by name, canonical path, or UUID ```python theme={null} def get_folder( folder_id_or_name: str, ) -> Folder ``` ```python theme={null} async def get_folder( folder_id_or_name: str, ) -> Folder ``` ## Parameters * `folder_id_or_name` (str): Folder identifier. Accepts the folder's UUID, name, or canonical path (e.g., `/projects/alpha/specs`; leading slash optional). ## Returns * `Folder`: Folder object that can be used to scope operations (ingest, query, etc.). Folder objects include hierarchy metadata such as `full_path`, `parent_id`, `depth`, and `child_count`, mirroring the server response. ## Examples ```python theme={null} from morphik import Morphik db = Morphik() folder = db.get_folder("marketing_docs") same_folder = db.get_folder(folder.id) assert folder.id == same_folder.id ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: folder = await db.get_folder("marketing_docs") same_folder = await db.get_folder(folder.id) assert folder.id == same_folder.id ``` # get_folder_by_name Source: https://morphik.ai/docs/python-sdk/get_folder_by_name Create a folder scope from a name or path ```python theme={null} def get_folder_by_name( name: str, ) -> Folder ``` ```python theme={null} async def get_folder_by_name( name: str, ) -> Folder ``` ## Parameters * `name` (str): Folder name or canonical path ## Returns * `Folder`: Folder scope object for subsequent operations ## Examples ```python theme={null} from morphik import Morphik db = Morphik() folder = db.get_folder_by_name("/projects/alpha") docs = folder.list_documents() ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: folder = await db.get_folder_by_name("/projects/alpha") docs = await folder.list_documents() ``` # get_folder_summary Source: https://morphik.ai/docs/python-sdk/get_folder_summary Fetch the stored summary for a folder ```python theme={null} def get_folder_summary( folder_id_or_path: str, ) -> Summary ``` ```python theme={null} async def get_folder_summary( folder_id_or_path: str, ) -> Summary ``` ## Parameters * `folder_id_or_path` (str): Folder identifier (UUID, name, or canonical path) ## Returns * `Summary`: Stored summary payload for the folder ## Examples ```python theme={null} from morphik import Morphik db = Morphik() summary = db.get_folder_summary("/projects/alpha") print(summary.content) ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: summary = await db.get_folder_summary("/projects/alpha") print(summary.content) ``` # get_folders_details Source: https://morphik.ai/docs/python-sdk/get_folders_details Get detailed information about folders with optional document statistics ```python theme={null} def get_folders_details( identifiers: Optional[List[str]] = None, include_document_count: bool = True, include_status_counts: bool = False, include_documents: bool = False, document_filters: Optional[Dict[str, Any]] = None, document_skip: int = 0, document_limit: int = 25, document_fields: Optional[List[str]] = None, sort_by: Optional[str] = None, sort_direction: Optional[str] = None, ) -> FolderDetailsResponse ``` ```python theme={null} async def get_folders_details( identifiers: Optional[List[str]] = None, include_document_count: bool = True, include_status_counts: bool = False, include_documents: bool = False, document_filters: Optional[Dict[str, Any]] = None, document_skip: int = 0, document_limit: int = 25, document_fields: Optional[List[str]] = None, sort_by: Optional[str] = None, sort_direction: Optional[str] = None, ) -> FolderDetailsResponse ``` ## Parameters * `identifiers` (List\[str], optional): List of folder IDs, names, or canonical paths (e.g., `/projects/alpha/specs`). If None, returns all accessible folders. * `include_document_count` (bool, optional): Include total document count. Defaults to True. * `include_status_counts` (bool, optional): Include document counts grouped by processing status. Defaults to False. * `include_documents` (bool, optional): Include paginated document list. Defaults to False. * `document_filters` (Dict\[str, Any], optional): Optional metadata filters for documents. * `document_skip` (int, optional): Number of documents to skip for pagination. Defaults to 0. * `document_limit` (int, optional): Maximum documents per folder. Defaults to 25. * `document_fields` (List\[str], optional): Optional list of fields to project for documents (dot notation supported). * `sort_by` (str, optional): Field to sort documents by. Options: "created\_at", "updated\_at", "filename", "external\_id". * `sort_direction` (str, optional): Sort direction. Options: "asc", "desc". ## Returns * `FolderDetailsResponse`: Response containing detailed folder information ## Metadata Filters Filters follow the same JSON syntax across the API. See the [Metadata Filtering guide](/concepts/metadata-filtering) for supported operators and typed comparisons. ## Examples ```python theme={null} from morphik import Morphik db = Morphik() # Get details for all folders with document counts response = db.get_folders_details() for folder_detail in response.folders: folder = folder_detail.folder doc_info = folder_detail.document_info print(f"Folder: {folder.name}") if doc_info and doc_info.total_count is not None: print(f" Total documents: {doc_info.total_count}") # Get specific folders with status counts response = db.get_folders_details( identifiers=["/projects/reports", "invoices"], include_status_counts=True, ) for folder_detail in response.folders: doc_info = folder_detail.document_info if doc_info and doc_info.status_counts: print(f"{folder_detail.folder.name} status breakdown:") for status, count in doc_info.status_counts.items(): print(f" {status}: {count}") # Get folders with document list response = db.get_folders_details( identifiers=["marketing"], include_documents=True, document_limit=10, sort_by="updated_at", sort_direction="desc", ) for folder_detail in response.folders: doc_info = folder_detail.document_info if doc_info and doc_info.documents: print(f"Recent documents in {folder_detail.folder.name}:") for doc in doc_info.documents: print(f" - {doc.filename}") # With document filtering and field projection response = db.get_folders_details( include_documents=True, document_filters={"department": "sales"}, document_fields=["external_id", "filename", "metadata.department"], document_limit=50, ) ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: # Get details for all folders with document counts response = await db.get_folders_details() for folder_detail in response.folders: folder = folder_detail.folder doc_info = folder_detail.document_info print(f"Folder: {folder.name}") if doc_info and doc_info.total_count is not None: print(f" Total documents: {doc_info.total_count}") # Get specific folders with status counts response = await db.get_folders_details( identifiers=["reports", "invoices"], include_status_counts=True, ) for folder_detail in response.folders: doc_info = folder_detail.document_info if doc_info and doc_info.status_counts: print(f"{folder_detail.folder.name} status breakdown:") for status, count in doc_info.status_counts.items(): print(f" {status}: {count}") # Get folders with document list response = await db.get_folders_details( identifiers=["marketing"], include_documents=True, document_limit=10, sort_by="updated_at", sort_direction="desc", ) for folder_detail in response.folders: doc_info = folder_detail.document_info if doc_info and doc_info.documents: print(f"Recent documents in {folder_detail.folder.name}:") for doc in doc_info.documents: print(f" - {doc.filename}") # With document filtering and field projection response = await db.get_folders_details( include_documents=True, document_filters={"department": "sales"}, document_fields=["external_id", "filename", "metadata.department"], document_limit=50, ) ``` ## Response Structure ### FolderDetailsResponse * `folders` (List\[FolderDetails]): List of folder details ### FolderDetails * `folder` (FolderInfo): Folder information * `document_info` (FolderDocumentInfo | None): Document statistics and list `FolderInfo` includes hierarchy fields: `full_path`, `parent_id`, `depth`, and `child_count`, plus description/name metadata. ### FolderDocumentInfo * `total_count` (int | None): Total document count (when `include_document_count=True`) * `status_counts` (Dict\[str, int] | None): Document counts by status (when `include_status_counts=True`) * `documents` (List\[Document] | None): Paginated document list (when `include_documents=True`) ## Notes * For a lightweight summary, use [`get_folders_summary`](./get_folders_summary) instead. * The `identifiers` parameter accepts both folder IDs and folder names. * Document pagination uses `document_skip` and `document_limit` to control the document list. * Use `document_fields` to reduce response size by projecting only needed fields. # get_folders_summary Source: https://morphik.ai/docs/python-sdk/get_folders_summary Get summary information for all accessible folders ```python theme={null} def get_folders_summary() -> List[FolderSummary] ``` ```python theme={null} async def get_folders_summary() -> List[FolderSummary] ``` ## Parameters This method takes no parameters. ## Returns * `List[FolderSummary]`: List of folder summaries with document counts ## Examples ```python theme={null} from morphik import Morphik db = Morphik() # Get summary of all folders summaries = db.get_folders_summary() for folder in summaries: print(f"Folder: {folder.name}") print(f" ID: {folder.id}") print(f" Description: {folder.description}") print(f" Document count: {folder.doc_count}") print(f" Last updated: {folder.updated_at}") print("---") # Find folders with most documents sorted_folders = sorted(summaries, key=lambda f: f.doc_count, reverse=True) print(f"Largest folder: {sorted_folders[0].name} ({sorted_folders[0].doc_count} docs)") ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: # Get summary of all folders summaries = await db.get_folders_summary() for folder in summaries: print(f"Folder: {folder.name}") print(f" ID: {folder.id}") print(f" Description: {folder.description}") print(f" Document count: {folder.doc_count}") print(f" Last updated: {folder.updated_at}") print("---") # Find folders with most documents sorted_folders = sorted(summaries, key=lambda f: f.doc_count, reverse=True) print(f"Largest folder: {sorted_folders[0].name} ({sorted_folders[0].doc_count} docs)") ``` ## FolderSummary Properties The `FolderSummary` objects have the following properties: * `id` (str): Unique folder identifier * `name` (str): Folder name * `full_path` (str | None): Canonical folder path (e.g., `/projects/alpha/specs`) * `parent_id` (str | None): Parent folder ID * `depth` (int | None): Depth in the hierarchy (root = 1) * `description` (str | None): Folder description * `doc_count` (int): Number of documents in the folder * `updated_at` (str | None): Last update timestamp ## Notes * This is a lightweight method for getting an overview of all folders. * For more detailed information including document lists and status counts, use [`get_folders_details`](./get_folders_details). * Returns only folders accessible to the authenticated user. # get_health Source: https://morphik.ai/docs/python-sdk/get_health Return detailed health status for the API ```python theme={null} def get_health() -> DetailedHealthCheckResponse ``` ```python theme={null} async def get_health() -> DetailedHealthCheckResponse ``` ## Returns * `DetailedHealthCheckResponse`: Overall health plus per-service status details ## Examples ```python theme={null} health = db.get_health() print(health.status) for svc in health.services: print(svc.name, svc.status) ``` # get_info Source: https://morphik.ai/docs/python-sdk/get_info Fetch the latest folder metadata from the API This method is available on `Folder` objects. ```python theme={null} def get_info() -> FolderInfo ``` ```python theme={null} async def get_info() -> FolderInfo ``` ## Returns * `FolderInfo`: Folder metadata including `full_path`, `parent_id`, `depth`, and `child_count` ## Examples ```python theme={null} from morphik import Morphik db = Morphik() folder = db.get_folder("/projects/alpha") info = folder.get_info() print(info.full_path, info.child_count) ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: folder = await db.get_folder("/projects/alpha") info = await folder.get_info() print(info.full_path, info.child_count) ``` # get_logs Source: https://morphik.ai/docs/python-sdk/get_logs Fetch recent API log events for the authenticated app ```python theme={null} def get_logs( limit: int = 100, hours: float = 4.0, op_type: Optional[str] = None, status: Optional[str] = None, ) -> List[LogResponse] ``` ```python theme={null} async def get_logs( limit: int = 100, hours: float = 4.0, op_type: Optional[str] = None, status: Optional[str] = None, ) -> List[LogResponse] ``` ## Parameters * `limit` (int, optional): Maximum number of log entries. Defaults to 100. * `hours` (float, optional): Lookback window in hours. Defaults to 4.0. * `op_type` (str, optional): Filter by operation type (for example `query`, `ingest`) * `status` (str, optional): Filter by status (for example `ok`, `error`) ## Returns * `List[LogResponse]`: Recent log entries ## Examples ```python theme={null} logs = db.get_logs(limit=20, hours=24) for item in logs: print(item.operation_type, item.status, item.timestamp) ``` # get_summary Source: https://morphik.ai/docs/python-sdk/get_summary Fetch the latest summary for a folder scope This method is available on `Folder` objects. ```python theme={null} def get_summary() -> Summary ``` ```python theme={null} async def get_summary() -> Summary ``` ## Returns * `Summary`: Stored summary payload for the folder ## Examples ```python theme={null} from morphik import Morphik db = Morphik() folder = db.get_folder("/projects/alpha") summary = folder.get_summary() print(summary.content) ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: folder = await db.get_folder("/projects/alpha") summary = await folder.get_summary() print(summary.content) ``` # ingest_directory Source: https://morphik.ai/docs/python-sdk/ingest_directory Ingest all files in a directory ```python theme={null} def ingest_directory( directory: Union[str, Path], recursive: bool = False, pattern: str = "*", metadata: Optional[Dict[str, Any]] = None, use_colpali: bool = True, parallel: bool = True, ) -> List[Document] ``` ```python theme={null} async def ingest_directory( directory: Union[str, Path], recursive: bool = False, pattern: str = "*", metadata: Optional[Dict[str, Any]] = None, use_colpali: bool = True, parallel: bool = True, ) -> List[Document] ``` ## Parameters * `directory` (str | Path): Directory containing files to ingest * `recursive` (bool, optional): Whether to recurse into subdirectories. Defaults to False. * `pattern` (str, optional): Glob pattern to select files (for example `"*.pdf"`). Defaults to `"*"`. * `metadata` (Dict\[str, Any], optional): Metadata applied to each ingested file * `use_colpali` (bool, optional): Whether to use ColPali-style embedding. Defaults to True. * `parallel` (bool, optional): Whether to process files in parallel. Defaults to True. ## Returns * `List[Document]`: List of ingested document metadata ## Examples ```python theme={null} from pathlib import Path from morphik import Morphik db = Morphik() docs = db.ingest_directory( Path("/data/contracts"), recursive=True, pattern="*.pdf", metadata={"category": "contracts"}, ) print(f"Ingested {len(docs)} documents") ``` ```python theme={null} from pathlib import Path from morphik import AsyncMorphik async with AsyncMorphik() as db: docs = await db.ingest_directory( Path("/data/contracts"), recursive=True, pattern="*.pdf", metadata={"category": "contracts"}, ) print(f"Ingested {len(docs)} documents") ``` # ingest_file Source: https://morphik.ai/docs/python-sdk/ingest_file Ingest a file document into Morphik ```python theme={null} def ingest_file( file: Union[str, bytes, BinaryIO, Path], filename: Optional[str] = None, metadata: Optional[Dict[str, Any]] = None, use_colpali: bool = True, ) -> Document ``` ```python theme={null} async def ingest_file( file: Union[str, bytes, BinaryIO, Path], filename: Optional[str] = None, metadata: Optional[Dict[str, Any]] = None, use_colpali: bool = True, ) -> Document ``` ## Parameters * `file` (Union\[str, bytes, BinaryIO, Path]): File to ingest (path string, bytes, file object, or Path) * `filename` (str, optional): Name of the file * `metadata` (Dict\[str, Any], optional): Optional metadata dictionary * `use_colpali` (bool, optional): Whether to use ColPali-style embedding model to ingest the file (slower, but significantly better retrieval accuracy for images). Defaults to True. ### Typed Metadata Use Python-native values (e.g., `datetime`, `date`, `Decimal`) in the `metadata` dict. The SDK serializes them and adds the corresponding `metadata_types`, so you can run the advanced filters documented in [Metadata Filtering](/concepts/metadata-filtering). ## Returns * `Document`: Metadata of the ingested document ## Examples ```python theme={null} from morphik import Morphik db = Morphik() doc = db.ingest_file( "document.pdf", filename="document.pdf", metadata={"category": "research", "owner": "alice"}, use_colpali=True, ) ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: doc = await db.ingest_file( "document.pdf", filename="document.pdf", metadata={"category": "research", "owner": "alice"}, use_colpali=True, ) ``` ## Async Processing Document ingestion is processed asynchronously. The returned `Document` object contains an `external_id` that you can use to track the document's processing status: * Use [`get_document_status`](./get_document_status) to check if processing is complete * The document won't appear in retrieval results until processing completes ```python theme={null} # Check processing status doc = db.ingest_file("document.pdf") status = db.get_document_status(doc.external_id) print(f"Status: {status.get('status')}") # "processing", "completed", or "failed" ``` # ingest_files Source: https://morphik.ai/docs/python-sdk/ingest_files Batch ingest multiple files into Morphik ## Usage ```python theme={null} from morphik import Morphik db = Morphik() # Batch ingest files with shared metadata result = db.ingest_files( files=["document1.pdf", "document2.docx", "image.png"], metadata={"category": "reports"}, use_colpali=True, parallel=True ) # Process the results for doc in result["documents"]: print(f"Successfully ingested: {doc.filename} (ID: {doc.external_id})") # Check for errors for error in result["errors"]: print(f"Error ingesting {error.get('filename')}: {error.get('error')}") ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: # Batch ingest files with shared metadata result = await db.ingest_files( files=["document1.pdf", "document2.docx", "image.png"], metadata={"category": "reports"}, use_colpali=True, parallel=True ) # Process the results for doc in result["documents"]: print(f"Successfully ingested: {doc.filename} (ID: {doc.external_id})") # Check for errors for error in result["errors"]: print(f"Error ingesting {error.get('filename')}: {error.get('error')}") ``` ## Parameters * `files` (List\[Union\[str, bytes, BinaryIO, Path]]): List of files to ingest (path strings, bytes, file objects, or Path objects) * `metadata` (Dict\[str, Any] | List\[Dict\[str, Any]], optional): Metadata to apply to the files. Can be either: * A single dict to apply to all files * A list of dicts, one per file (must match the length of `files`) * `use_colpali` (bool, optional): Whether to use ColPali-style embedding model. Defaults to True. * `parallel` (bool, optional): Whether to process files in parallel. Defaults to True. ### Typed Metadata When specifying `metadata`, you can include Python `datetime`, `date`, `Decimal`, or numeric types. The SDK normalizes them, sends the accompanying `metadata_types`, and unlocks the advanced queries outlined in [Metadata Filtering](/concepts/metadata-filtering). ## Returns An object containing: * `documents`: List of successfully ingested [Document](/python-sdk/document) objects * `errors`: List of errors encountered during ingestion (each error is a dict with 'filename' and 'error' keys) ## Advanced Examples ### Per-File Metadata ```python theme={null} # Ingest files with different metadata for each file files = ["report.pdf", "data.csv", "presentation.pptx"] # Metadata must match the length of files list metadata_list = [ {"category": "reports", "author": "Alice"}, {"category": "data", "source": "database"}, {"category": "presentations", "department": "marketing"} ] result = db.ingest_files( files=files, metadata=metadata_list ) ``` ```python theme={null} # Ingest files with different metadata for each file files = ["report.pdf", "data.csv", "presentation.pptx"] # Metadata must match the length of files list metadata_list = [ {"category": "reports", "author": "Alice"}, {"category": "data", "source": "database"}, {"category": "presentations", "department": "marketing"} ] result = await db.ingest_files( files=files, metadata=metadata_list ) ``` ### Using Different File Input Types ```python theme={null} import io from pathlib import Path # Mixing different file input types file1 = "document.pdf" # Path string file2 = Path("image.png") # Path object file3 = open("data.csv", "rb") # File object file4 = b"Hello, world!" # Bytes (requires filename) file5 = io.BytesIO(b"Some in-memory data") # BytesIO (requires filename) result = db.ingest_files( files=[file1, file2, file3, file4, file5], metadata=[ {"type": "document"}, {"type": "image"}, {"type": "data"}, {"type": "text", "filename": "hello.txt"}, {"type": "text", "filename": "memory-data.txt"} ] ) # Don't forget to close file objects file3.close() ``` ```python theme={null} import io from pathlib import Path # Mixing different file input types file1 = "document.pdf" # Path string file2 = Path("image.png") # Path object file3 = open("data.csv", "rb") # File object file4 = b"Hello, world!" # Bytes (requires filename) file5 = io.BytesIO(b"Some in-memory data") # BytesIO (requires filename) result = await db.ingest_files( files=[file1, file2, file3, file4, file5], metadata=[ {"type": "document"}, {"type": "image"}, {"type": "data"}, {"type": "text", "filename": "hello.txt"}, {"type": "text", "filename": "memory-data.txt"} ] ) # Don't forget to close file objects file3.close() ``` # ingest_text Source: https://morphik.ai/docs/python-sdk/ingest_text Ingest a text document into Morphik ```python theme={null} def ingest_text( content: str, filename: Optional[str] = None, metadata: Optional[Dict[str, Any]] = None, use_colpali: bool = True, ) -> Document ``` ```python theme={null} async def ingest_text( content: str, filename: Optional[str] = None, metadata: Optional[Dict[str, Any]] = None, use_colpali: bool = True, ) -> Document ``` ## Parameters * `content` (str): Text content to ingest * `filename` (str, optional): Optional filename for the document * `metadata` (Dict\[str, Any], optional): Optional metadata dictionary * `use_colpali` (bool, optional): Whether to use ColPali-style embedding model to ingest the text (slower, but significantly better retrieval accuracy for text and images). Defaults to True. ### Typed Metadata Pass native Python types for metadata (e.g., `datetime`, `date`, `Decimal`, `bool`). The SDK normalizes them, forwards the appropriate `metadata_types`, and unlocks range queries described in the [Metadata Filtering guide](/concepts/metadata-filtering). Example: ```python theme={null} from datetime import datetime, date from decimal import Decimal doc = db.ingest_text( "SOW details …", metadata={ "priority": 42, "start_date": datetime.utcnow(), "end_date": date(2024, 12, 31), "cost": Decimal("1234.56") } ) ``` ## Returns * `Document`: Metadata of the ingested document ## Examples ```python theme={null} from morphik import Morphik db = Morphik() doc = db.ingest_text( "Machine learning is fascinating...", metadata={"category": "tech"}, use_colpali=True, ) ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: doc = await db.ingest_text( "Machine learning is fascinating...", metadata={"category": "tech"}, use_colpali=True, ) ``` # list_apps Source: https://morphik.ai/docs/python-sdk/list_apps List cloud apps accessible to the current credentials ```python theme={null} def list_apps( org_id: Optional[str] = None, user_id: Optional[str] = None, app_id_filter: Optional[Union[str, Dict[str, Any], List[Any]]] = None, app_name_filter: Optional[Union[str, Dict[str, Any], List[Any]]] = None, limit: int = 100, offset: int = 0, ) -> Dict[str, Any] ``` ```python theme={null} async def list_apps( org_id: Optional[str] = None, user_id: Optional[str] = None, app_id_filter: Optional[Union[str, Dict[str, Any], List[Any]]] = None, app_name_filter: Optional[Union[str, Dict[str, Any], List[Any]]] = None, limit: int = 100, offset: int = 0, ) -> Dict[str, Any] ``` ## Parameters * `org_id` (str, optional): Filter by organization ID * `user_id` (str, optional): Filter by user ID * `app_id_filter` (str | dict | list, optional): JSON filter for app IDs (dict/list will be serialized) * `app_name_filter` (str | dict | list, optional): JSON filter for app names (dict/list will be serialized) * `limit` (int, optional): Max results per page. Defaults to 100 (clamped to 500). * `offset` (int, optional): Pagination offset. Defaults to 0. ## Returns * `Dict[str, Any]`: API response containing apps and pagination metadata ## Examples ```python theme={null} from morphik import Morphik db = Morphik() apps = db.list_apps(limit=20) print(apps) ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: apps = await db.list_apps(app_name_filter={"$eq": "demo"}) print(apps) ``` # list_chat_conversations Source: https://morphik.ai/docs/python-sdk/list_chat_conversations List recent chat conversations for the authenticated user ```python theme={null} def list_chat_conversations(limit: int = 100) -> List[Dict[str, Any]] ``` ```python theme={null} async def list_chat_conversations(limit: int = 100) -> List[Dict[str, Any]] ``` ## Parameters * `limit` (int, optional): Maximum number of conversations to return. Valid range is 1-500. Default is 100. ## Returns * `List[Dict[str, Any]]`: Each dictionary contains: * `chat_id` – conversation identifier * `updated_at` – last activity timestamp * `message_preview` – snippet of the last user/assistant message ## Example ```python theme={null} db = Morphik() convos = db.list_chat_conversations(limit=20) for convo in convos: print(convo["chat_id"], convo["updated_at"]) ``` ```python theme={null} async with AsyncMorphik() as db: convos = await db.list_chat_conversations() print(f"You have {len(convos)} conversations") ``` # list_documents Source: https://morphik.ai/docs/python-sdk/list_documents List accessible documents in Morphik This method returns a `ListDocsResponse` object. Access documents via `response.documents`. ```python theme={null} def list_documents( skip: int = 0, limit: int = 100, filters: Optional[Dict[str, Any]] = None, folder_name: Optional[Union[str, List[str]]] = None, folder_depth: Optional[int] = None, include_total_count: bool = False, include_status_counts: bool = False, include_folder_counts: bool = False, completed_only: bool = False, sort_by: Optional[str] = "updated_at", sort_direction: str = "desc" ) -> ListDocsResponse ``` ```python theme={null} async def list_documents( skip: int = 0, limit: int = 100, filters: Optional[Dict[str, Any]] = None, folder_name: Optional[Union[str, List[str]]] = None, folder_depth: Optional[int] = None, include_total_count: bool = False, include_status_counts: bool = False, include_folder_counts: bool = False, completed_only: bool = False, sort_by: Optional[str] = "updated_at", sort_direction: str = "desc" ) -> ListDocsResponse ``` ## Parameters * `skip` (int, optional): Number of documents to skip for pagination. Defaults to 0. * `limit` (int, optional): Maximum number of documents to return. Defaults to 100. * `filters` (Dict\[str, Any], optional): Metadata filters to apply * `folder_name` (str | List\[str], optional): Optional folder scope. Accepts a canonical path (e.g., `/projects/alpha`) or a list of paths/names. * `folder_depth` (int, optional): Folder scope depth. `None`/`0` = exact match, `-1` = include all descendants, `n > 0` = include descendants up to `n` levels deep. * `include_total_count` (bool, optional): Include total count of matching documents. Defaults to False. * `include_status_counts` (bool, optional): Include counts grouped by processing status. Defaults to False. * `include_folder_counts` (bool, optional): Include counts grouped by folder. Defaults to False. * `completed_only` (bool, optional): Only return documents with completed status. Defaults to False. * `sort_by` (str, optional): Field to sort by (`created_at`, `updated_at`, `filename`, `external_id`). Defaults to "updated\_at". * `sort_direction` (str, optional): Sort direction (`asc` or `desc`). Defaults to "desc". ### Metadata Filters Pass any JSON filter described in the [Metadata Filtering guide](/concepts/metadata-filtering) via the `filters` argument. Example: ```python theme={null} filters = { "$and": [ {"department": {"$eq": "research"}}, {"priority": {"$gte": 40}}, {"start_date": {"$lte": "2024-06-01"}} ] } response = db.list_documents(filters=filters, include_total_count=True) ``` ## Returns `ListDocsResponse` object with the following properties: * `documents` (List\[Document]): The list of documents * `skip` (int): Pagination offset used * `limit` (int): Limit used * `returned_count` (int): Number of documents in this response * `total_count` (Optional\[int]): Total matching documents (if `include_total_count=True`) * `has_more` (bool): Whether more documents exist beyond this page * `next_skip` (Optional\[int]): Skip value to use for the next page * `status_counts` (Optional\[Dict\[str, int]]): Document counts by status (if requested) * `folder_counts` (Optional\[List]): Document counts by folder (if requested) ## Examples ### Basic Usage ```python theme={null} from morphik import Morphik db = Morphik() # Basic listing response = db.list_documents(limit=10) print(f"Returned {response.returned_count} documents") for doc in response.documents: print(f"Document: {doc.filename}") ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: response = await db.list_documents(limit=10) for doc in response.documents: print(f"Document: {doc.filename}") ``` ### Pagination with Total Count ```python theme={null} # Get total count and paginate response = db.list_documents(limit=10, include_total_count=True) print(f"Showing {response.returned_count} of {response.total_count} documents") # Check if more pages exist if response.has_more: next_page = db.list_documents(skip=response.next_skip, limit=10) ``` ```python theme={null} response = await db.list_documents(limit=10, include_total_count=True) if response.has_more: next_page = await db.list_documents(skip=response.next_skip, limit=10) ``` ### Sorting and Filtering ```python theme={null} # Sort by creation date, newest first response = db.list_documents( sort_by="created_at", sort_direction="desc", filters={"department": "research"} ) # Only completed documents response = db.list_documents(completed_only=True) ``` ```python theme={null} response = await db.list_documents( sort_by="created_at", sort_direction="desc", filters={"department": "research"} ) ``` ### Nested Folder Queries ```python theme={null} # Include all documents under /projects/alpha and its children response = db.list_documents( folder_name="/projects/alpha", folder_depth=-1, # descend through nested folders include_folder_counts=True, ) ``` ### Aggregates and Counts ```python theme={null} # Get status breakdown response = db.list_documents(include_status_counts=True) print(response.status_counts) # {"completed": 42, "processing": 3, "failed": 1} # Get folder distribution response = db.list_documents(include_folder_counts=True) for folder in response.folder_counts: print(f"{folder.folder}: {folder.count} documents") ``` ```python theme={null} response = await db.list_documents(include_status_counts=True) print(response.status_counts) ``` ## Document Properties The `Document` objects returned by this method have the following properties: * `external_id` (str): Unique document identifier * `content_type` (str): Content type of the document * `filename` (Optional\[str]): Original filename if available * `metadata` (Dict\[str, Any]): User-defined metadata * `storage_info` (Dict\[str, str]): Storage-related information * `system_metadata` (Dict\[str, Any]): System-managed metadata * `chunk_ids` (List\[str]): IDs of document chunks * `folder_name` (Optional\[str]): Folder leaf name * `folder_path` (Optional\[str]): Canonical folder path (includes nested parents) # list_folders Source: https://morphik.ai/docs/python-sdk/list_folders List all folders available to the client ```python theme={null} def list_folders() -> List[Folder] ``` ```python theme={null} async def list_folders() -> List[Folder] ``` ## Returns * `List[Folder]`: Collection of folders the current auth context can access. Each `Folder` now surfaces hierarchy details: * `full_path`: Canonical path (e.g., `/projects/alpha/specs`) * `parent_id`: Parent folder ID (if any) * `depth`: Depth in the tree (root = 1) * `child_count`: Number of direct children when provided ## Examples ```python theme={null} from morphik import Morphik db = Morphik() for folder in db.list_folders(): print(folder.name, folder.id) ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: folders = await db.list_folders() for folder in folders: print(folder.name, folder.id) ``` # Morphik Source: https://morphik.ai/docs/python-sdk/morphik Main client for document operations in Morphik ## Usage ```python theme={null} from morphik import Morphik # Without authentication db = Morphik() # With authentication db = Morphik("morphik://owner_id:token@api.morphik.ai") ``` ```python theme={null} from morphik import AsyncMorphik # Without authentication async with AsyncMorphik() as db: doc = await db.ingest_text("Sample content") # With authentication async with AsyncMorphik("morphik://owner_id:token@api.morphik.ai") as db: doc = await db.ingest_text("Sample content") ``` ## User and Folder Scoping Morphik supports organizing and isolating data by user and folder. This provides a way to build multi-tenant applications and organize documents across projects. ### Quick Overview ```python theme={null} # Folder scoping - organize by project or category folder = db.create_folder("project_x") doc = folder.ingest_text("This document belongs to Project X") # User scoping - isolate data by end user user_scope = db.signin("user123") doc = user_scope.ingest_text("This belongs to user123 only") # Combined scoping - organize by both user and folder user_folder_scope = folder.signin("user123") doc = user_folder_scope.ingest_text("This belongs to user123 in project_x") ``` ```python theme={null} # Folder scoping - organize by project or category folder = db.create_folder("project_x") doc = await folder.ingest_text("This document belongs to Project X") # User scoping - isolate data by end user user_scope = db.signin("user123") doc = await user_scope.ingest_text("This belongs to user123 only") # Combined scoping - organize by both user and folder user_folder_scope = folder.signin("user123") doc = await user_folder_scope.ingest_text("This belongs to user123 in project_x") ``` Nested folders are supported across the SDK. Use canonical paths (e.g., `"/projects/alpha/specs"`) when creating or scoping folders, and pass `folder_depth` on retrieval/list helpers to include descendant folders. For detailed documentation and examples: * [Folder Management](/python-sdk/folders) - Organizing documents by logical groups * [User Management](/python-sdk/users) - Multi-tenant isolation and user-level data management ## Constructor Both clients share the same constructor parameters: ```python theme={null} Morphik( uri: Optional[str] = None, timeout: int = 30, is_local: bool = False, http2: Optional[bool] = None, http2_fallback: bool = True, ) ``` ```python theme={null} AsyncMorphik( uri: Optional[str] = None, timeout: int = 30, is_local: bool = False, http2: Optional[bool] = None, http2_fallback: bool = True, ) ``` ### Parameters * `uri` (str, optional): Morphik URI in format "morphik://\:\@\". If not provided, connects to [http://localhost:8000](http://localhost:8000) without authentication. * `timeout` (int, optional): Request timeout in seconds. Defaults to 30. * `is_local` (bool, optional): Whether connecting to local development server. Defaults to False. * `http2` (bool, optional): Enable HTTP/2 when possible. Defaults to None (auto-disabled for local). * `http2_fallback` (bool, optional): Fall back to HTTP/1.1 if HTTP/2 fails. Defaults to True. ## Methods Morphik provides the following methods. Each method page includes both synchronous and asynchronous versions. ### Document Ingestion * [ingest\_text](/python-sdk/ingest_text) * [ingest\_file](/python-sdk/ingest_file) * [ingest\_files](/python-sdk/ingest_files) * [ingest\_directory](/python-sdk/ingest_directory) * [query\_document](/python-sdk/query_document) ### Document Retrieval * [retrieve\_chunks](/python-sdk/retrieve_chunks) * [retrieve\_chunks\_grouped](/python-sdk/retrieve_chunks_grouped) * [retrieve\_docs](/python-sdk/retrieve_docs) * [query](/python-sdk/query) * [list\_documents](/python-sdk/list_documents) * [search\_documents](/python-sdk/search_documents) * [get\_document](/python-sdk/get_document) * [get\_document\_by\_filename](/python-sdk/get_document_by_filename) ### Document Updates & Summaries * [update\_document\_with\_text](/python-sdk/update_document_with_text) * [update\_document\_with\_file](/python-sdk/update_document_with_file) * [update\_document\_metadata](/python-sdk/update_document_metadata) * [update\_document\_by\_filename\_with\_text](/python-sdk/update_document_by_filename_with_text) * [update\_document\_by\_filename\_with\_file](/python-sdk/update_document_by_filename_with_file) * [update\_document\_by\_filename\_metadata](/python-sdk/update_document_by_filename_metadata) * [get\_document\_summary](/python-sdk/get_document_summary) * [upsert\_document\_summary](/python-sdk/upsert_document_summary) * [get\_document\_status](/python-sdk/get_document_status) * [wait\_for\_document\_completion](/python-sdk/wait_for_document_completion) ### Document Management * [get\_document\_file](/python-sdk/get_document_file) * [extract\_document\_pages](/python-sdk/extract_document_pages) * [get\_document\_download\_url](/python-sdk/get_document_download_url) * [delete\_document](/python-sdk/delete_document) * [delete\_document\_by\_filename](/python-sdk/delete_document_by_filename) * [batch\_get\_documents](/python-sdk/batch_get_documents) * [batch\_get\_chunks](/python-sdk/batch_get_chunks) ### Folder & User Scoping * [folders](/python-sdk/folders) * [create\_folder](/python-sdk/create_folder) * [list\_folders](/python-sdk/list_folders) * [get\_folder](/python-sdk/get_folder) * [get\_folder\_by\_name](/python-sdk/get_folder_by_name) * [get\_folder\_summary](/python-sdk/get_folder_summary) * [upsert\_folder\_summary](/python-sdk/upsert_folder_summary) * [get\_folders\_summary](/python-sdk/get_folders_summary) * [get\_folders\_details](/python-sdk/get_folders_details) * [add\_document\_to\_folder](/python-sdk/add_document_to_folder) * [remove\_document\_from\_folder](/python-sdk/remove_document_from_folder) * [delete\_folder](/python-sdk/delete_folder) * [users](/python-sdk/users) ### Apps & Ops * [list\_apps](/python-sdk/list_apps) * [create\_app](/python-sdk/create_app) * [generate\_cloud\_uri](/python-sdk/generate_cloud_uri) * [rename\_app](/python-sdk/rename_app) * [rotate\_app\_token](/python-sdk/rotate_app_token) * [delete\_app](/python-sdk/delete_app) * [requeue\_ingestion\_jobs](/python-sdk/requeue_ingestion_jobs) * [get\_logs](/python-sdk/get_logs) * [get\_health](/python-sdk/get_health) * [get\_app\_storage\_usage](/python-sdk/get_app_storage_usage) * [ping](/python-sdk/ping) ### Chat & Conversation * [list\_chat\_conversations](/python-sdk/list_chat_conversations) * [get\_chat\_history](/python-sdk/get_chat_history) ### Client Management * [close](/python-sdk/close) ## Context Manager Using the Morphik client as a context manager ensures that resources are properly closed when the context exits. ```python theme={null} with Morphik() as db: doc = db.ingest_text("Sample content") ``` ```python theme={null} async with AsyncMorphik() as db: doc = await db.ingest_text("Sample content") ``` # ping Source: https://morphik.ai/docs/python-sdk/ping Health-check endpoint – verify that your Morphik server is reachable ```python theme={null} def ping() -> Dict[str, Any] ``` ```python theme={null} async def ping() -> Dict[str, Any] ``` ## Returns * `Dict[str, Any]`: A JSON object with two keys: * `status` – always `"ok"` when the server is running * `message` – human-readable confirmation string ## Example ```python theme={null} from morphik import Morphik db = Morphik() resp = db.ping() assert resp["status"] == "ok" ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: resp = await db.ping() print(resp) ``` # query Source: https://morphik.ai/docs/python-sdk/query Generate completion using relevant chunks as context # query Generate completion using relevant chunks as context. ```python theme={null} def query( query: str, filters: Optional[Dict[str, Any]] = None, k: int = 4, min_score: float = 0.0, max_tokens: Optional[int] = None, temperature: Optional[float] = None, use_colpali: bool = True, use_reranking: Optional[bool] = None, prompt_overrides: Optional[Union[QueryPromptOverrides, Dict[str, Any]]] = None, folder_name: Optional[Union[str, List[str]]] = None, folder_depth: Optional[int] = None, chat_id: Optional[str] = None, schema: Optional[Union[Type[BaseModel], Dict[str, Any]]] = None, llm_config: Optional[Dict[str, Any]] = None, padding: int = 0, ) -> CompletionResponse ``` ```python theme={null} async def query( query: str, filters: Optional[Dict[str, Any]] = None, k: int = 4, min_score: float = 0.0, max_tokens: Optional[int] = None, temperature: Optional[float] = None, use_colpali: bool = True, use_reranking: Optional[bool] = None, prompt_overrides: Optional[Union[QueryPromptOverrides, Dict[str, Any]]] = None, folder_name: Optional[Union[str, List[str]]] = None, folder_depth: Optional[int] = None, chat_id: Optional[str] = None, schema: Optional[Union[Type[BaseModel], Dict[str, Any]]] = None, llm_config: Optional[Dict[str, Any]] = None, padding: int = 0, ) -> CompletionResponse ``` ## Parameters * `query` (str): Query text * `filters` (Dict\[str, Any], optional): Optional metadata filters * `k` (int, optional): Number of chunks to use as context. Defaults to 4. * `min_score` (float, optional): Minimum similarity threshold. Defaults to 0.0. * `max_tokens` (int, optional): Maximum tokens in completion * `temperature` (float, optional): Model temperature * `use_colpali` (bool, optional): Whether to use ColPali-style embedding model to generate the completion (only works for documents ingested with `use_colpali=True`). Defaults to True. * `use_reranking` (bool, optional): Override workspace reranking configuration for this request. * `prompt_overrides` (QueryPromptOverrides | Dict\[str, Any], optional): Optional customizations for entity extraction, resolution, and query prompts * `folder_name` (str | List\[str], optional): Optional folder scope. Accepts canonical paths (e.g., `/projects/alpha/specs`) or a list of paths/names. * `folder_depth` (int, optional): Folder scope depth. `None`/`0` = exact match, `-1` = include all descendants, `n > 0` = include descendants up to `n` levels deep. * `chat_id` (str, optional): Optional chat session ID for persisting conversation history. * `schema` (Type\[BaseModel] | Dict\[str, Any], optional): Optional schema for structured output, can be a Pydantic model or a JSON schema dict * `llm_config` (Dict\[str, Any], optional): Optional LiteLLM-compatible model configuration (e.g., model name, API key, base URL). Allows overriding the default LLM configuration on a per-query basis. Defaults to None. * `padding` (int, optional): Number of additional chunks/pages to retrieve before and after matched chunks (ColPali only). Defaults to 0. ## Metadata Filters Use the same JSON filters described in the [Metadata Filtering concept](/concepts/metadata-filtering) to restrict which documents feed into the completion. Example: ```python theme={null} filters = { "$and": [ {"project": {"$eq": "delta"}}, {"start_date": {"$lte": "2024-06-01T00:00:00Z"}}, {"end_date": {"$gte": "2024-06-01T00:00:00Z"}} ] } response = db.query( "Summarize current scope changes", filters=filters, k=6, temperature=0.3 ) ``` You can also filter by folder name and use expressive operators like `$in`, `$regex`, and `$nin`: ```python theme={null} # Query across multiple folders filters = { "$and": [ {"folder_name": {"$in": ["reports", "invoices"]}}, {"year": 2024}, {"priority": {"$gte": 50}} ] } response = db.query("What are the key financial highlights?", filters=filters) ``` For more advanced filtering patterns, see the [Complex Metadata Filtering cookbook](/cookbooks/complex-metadata-filtering). ## Returns * `CompletionResponse`: Response containing the completion, source information, and potentially structured output. ## Examples ### Standard Query ```python theme={null} from morphik import Morphik db = Morphik() response = db.query( "What are the key findings about customer satisfaction?", filters={"department": "research"}, temperature=0.7 ) nested = db.query( "List open design questions", folder_name="/projects/alpha", folder_depth=-1, k=6, ) print(response.completion) # Print the sources used for the completion for source in response.sources: print(f"Document ID: {source.document_id}, Chunk: {source.chunk_number}, Score: {source.score}") ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: response = await db.query( "What are the key findings about customer satisfaction?", filters={"department": "research"}, temperature=0.7 ) nested = await db.query( "List open design questions", folder_name="/projects/alpha", folder_depth=-1, k=6, ) print(response.completion) # Print the sources used for the completion for source in response.sources: print(f"Document ID: {source.document_id}, Chunk: {source.chunk_number}, Score: {source.score}") ``` ### With Custom Prompt Overrides ```python theme={null} from morphik import Morphik from morphik.models import QueryPromptOverride, QueryPromptOverrides db = Morphik() # Using the QueryPromptOverrides object response = db.query( "What are the key findings?", filters={"category": "research"}, prompt_overrides=QueryPromptOverrides( query=QueryPromptOverride( prompt_template="Answer the question in a formal, academic tone: {question}\n\nContext:\n{context}\n\nAnswer:" ) ) ) # Alternatively, using a dictionary response = db.query( "What are the key findings?", filters={"category": "research"}, prompt_overrides={ "query": { "prompt_template": "Answer the question in a formal, academic tone: {question}\n\nContext:\n{context}\n\nAnswer:" } } ) print(response.completion) ``` ```python theme={null} from morphik import AsyncMorphik from morphik.models import QueryPromptOverride, EntityExtractionPromptOverride, QueryPromptOverrides async with AsyncMorphik() as db: # Example with both query and entity extraction customization response = await db.query( "How does the medication affect diabetes?", prompt_overrides=QueryPromptOverrides( # Customize how responses are generated query=QueryPromptOverride( prompt_template="Provide a concise, medically accurate answer: {question}\n\nContext:\n{context}\n\nAnswer:" ), # Customize entity extraction entity_extraction=EntityExtractionPromptOverride( examples=[ {"label": "Insulin", "type": "MEDICATION"}, {"label": "Diabetes", "type": "CONDITION"} ] ) ) ) print(response.completion) ``` ## CompletionResponse Properties The `CompletionResponse` object returned by this method has the following properties: * `completion` (str | Dict\[str, Any] | None): The generated completion text or the structured output dictionary. * `usage` (Dict\[str, int]): Token usage information * `sources` (List\[ChunkSource]): Sources of chunks used in the completion * `metadata` (Dict\[str, Any], optional): Additional metadata about the completion (if provided by the server). * `finish_reason` (Optional\[str]): Reason the generation finished (e.g., 'stop', 'length') ### ChunkSource Properties Each `ChunkSource` object in the `sources` list has the following properties: * `document_id` (str): ID of the source document * `chunk_number` (int): Chunk number within the document * `score` (Optional\[float]): Relevance score (if available) ### Using Custom LLM Configuration The `llm_config` parameter is available in SDK version 0.2.5 and later. Use the `llm_config` parameter to override the default LLM configuration on a per-query basis. This allows you to use different models, API keys, or other LiteLLM-compatible settings for specific queries. ```python theme={null} from morphik import Morphik db = Morphik() # Use GPT-4 for a specific query response = db.query( "What are the key findings?", llm_config={ "model": "gpt-4", "api_key": "sk-...", "base_url": "https://api.openai.com/v1", "temperature": 0.7, "max_tokens": 2000 } ) print(response.completion) # Use Claude for another query response = db.query( "Summarize the research findings", llm_config={ "model": "claude-3-opus-20240229", "api_key": "your-anthropic-key" } ) print(response.completion) ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: # Use GPT-4 for a specific query response = await db.query( "What are the key findings?", llm_config={ "model": "gpt-4", "api_key": "sk-...", "base_url": "https://api.openai.com/v1", "temperature": 0.7, "max_tokens": 2000 } ) print(response.completion) # Use Claude for another query response = await db.query( "Summarize the research findings", llm_config={ "model": "claude-3-opus-20240229", "api_key": "your-anthropic-key" } ) print(response.completion) ``` ### Using Structured Output Use the `schema` parameter to get the completion response in a structured format according to a Pydantic model or a JSON schema dictionary. ```python theme={null} from morphik import Morphik from pydantic import BaseModel from typing import List # Define the desired output structure class ResearchFindings(BaseModel): main_finding: str supporting_evidence: List[str] limitations: List[str] db = Morphik() response = db.query( "Summarize the key research findings from these documents", filters={"department": "research"}, schema=ResearchFindings ) # Check if the completion is a dictionary (structured output) if isinstance(response.completion, dict): try: # Parse the dictionary into the Pydantic model findings = ResearchFindings(**response.completion) print(f"Main finding: {findings.main_finding}") print("Supporting evidence:") for evidence in findings.supporting_evidence: print(f"- {evidence}") print("Limitations:") for limitation in findings.limitations: print(f"- {limitation}") except Exception as e: print(f"Error parsing structured output: {e}") # Fallback: print the raw dictionary print(response.completion) elif isinstance(response.completion, str): # Fallback to text completion print(response.completion) ``` ```python theme={null} from morphik import AsyncMorphik from pydantic import BaseModel from typing import List # Define the desired output structure class ResearchFindings(BaseModel): main_finding: str supporting_evidence: List[str] limitations: List[str] async with AsyncMorphik() as db: response = await db.query( "Summarize the key research findings from these documents", filters={"department": "research"}, schema=ResearchFindings ) # Check if the completion is a dictionary (structured output) if isinstance(response.completion, dict): try: # Parse the dictionary into the Pydantic model findings = ResearchFindings(**response.completion) print(f"Main finding: {findings.main_finding}") print("Supporting evidence:") for evidence in findings.supporting_evidence: print(f"- {evidence}") print("Limitations:") for limitation in findings.limitations: print(f"- {limitation}") except Exception as e: print(f"Error parsing structured output: {e}") # Fallback: print the raw dictionary print(response.completion) elif isinstance(response.completion, str): # Fallback to text completion print(response.completion) ``` # query_document Source: https://morphik.ai/docs/python-sdk/query_document Run a one-off Morphik On-the-Fly analysis with optional ingestion follow-up ```python theme={null} def query_document( file: Union[str, bytes, BinaryIO, Path], prompt: str, schema: Optional[Union[Dict[str, Any], Type[BaseModel], BaseModel, str]] = None, ingestion_options: Optional[Dict[str, Any]] = None, filename: Optional[str] = None, folder_name: Optional[Union[str, List[str]]] = None, end_user_id: Optional[str] = None, ) -> DocumentQueryResponse ``` ```python theme={null} async def query_document( file: Union[str, bytes, BinaryIO, Path], prompt: str, schema: Optional[Union[Dict[str, Any], Type[BaseModel], BaseModel, str]] = None, ingestion_options: Optional[Dict[str, Any]] = None, filename: Optional[str] = None, folder_name: Optional[Union[str, List[str]]] = None, end_user_id: Optional[str] = None, ) -> DocumentQueryResponse ``` ## Parameters * `file` (Union\[str, bytes, BinaryIO, Path]): Document to analyse inline. Accepts a file path, bytes buffer, or file-like object. * `prompt` (str): Instruction Morphik On-the-Fly should execute against the document. * `schema` (dict | BaseModel | Type\[BaseModel] | str, optional): Schema that enforces structured output. Accepts a plain dict, a Pydantic model or class, or a pre-serialized JSON string. * `ingestion_options` (Dict\[str, Any], optional): Controls follow-up ingestion. Supported keys: * `ingest` (bool): Queue the file for ingestion after analysis. * `metadata` (dict): Metadata supplied with the request. When `schema` yields a JSON object, those fields are merged into this metadata before ingestion. * `use_colpali` (bool): Override the embedding strategy used during ingestion. * `folder_name` (str | list\[str]): Folder scope for the queued ingestion (canonical path or list of paths/names; nested parents are created automatically). * `end_user_id` (str): End-user scope for the queued ingestion. Unsupported keys are ignored. * `filename` (str, optional): Filename override when uploading bytes or file-like objects. * `folder_name` (str | list\[str], optional): Folder scope applied to the inline request (canonical path or list of paths/names). Automatically set when calling from folder helpers; merged into `ingestion_options` if not already present. * `end_user_id` (str, optional): End-user scope for the inline request. Automatically set when using user scope helpers; merged into `ingestion_options` if not already present. ### Metadata Filters Some `ingestion_options` workflows or follow-up ingestion steps require metadata filters. Use the JSON operators documented in [Metadata Filtering](/concepts/metadata-filtering) to keep behavior consistent with other endpoints. ## Returns * `DocumentQueryResponse`: Contains `structured_output`, `text_output`, `input_metadata`, `combined_metadata`, and ingestion status. When ingestion is requested and the schema produces a JSON object, `combined_metadata` reflects the union of the supplied metadata and the extracted fields used for ingestion. ## Behaviour * **Structured extraction:** When `schema` is provided, Morphik validates the response against the schema. If the structured output is a dict, it is returned in `structured_output` and copied to `extracted_metadata`. * **Metadata merge:** `combined_metadata` is always derived from the original `metadata` supplied in `ingestion_options`. When structured extraction returns a dict, those fields are merged into the metadata before any ingestion takes place. * **Ingestion queuing:** Setting `ingest=True` enqueues the document for ingestion (requires `write` permission). The response includes `ingestion_enqueued` and, when available, an `ingestion_document` stub you can monitor. ## Examples ### Extract structured data and ingest ```python theme={null} from typing import Optional from pydantic import BaseModel from morphik import Morphik class ContractSummary(BaseModel): parties: list[str] effective_date: str auto_renew: Optional[bool] db = Morphik() result = db.query_document( file="contracts/acme_supply.pdf", prompt="Extract the parties, effective date, and whether the agreement auto-renews.", schema=ContractSummary, ingestion_options={ "ingest": True, "metadata": {"source": "contracts", "region": "NA"}, "folder_name": "contracts", }, ) print(result.structured_output) print(result.combined_metadata) # original metadata merged with schema fields ``` ```python theme={null} from typing import Optional import asyncio from pydantic import BaseModel from morphik import AsyncMorphik class ContractSummary(BaseModel): parties: list[str] effective_date: str auto_renew: Optional[bool] async def run(): async with AsyncMorphik() as db: result = await db.query_document( file="contracts/acme_supply.pdf", prompt="Extract the parties, effective date, and whether the agreement auto-renews.", schema=ContractSummary, ingestion_options={ "ingest": True, "metadata": {"source": "contracts", "region": "NA"}, "use_colpali": False, }, ) print(result.structured_output) print(result.ingestion_enqueued) asyncio.run(run()) ``` ### Quick inline analysis without ingestion ```python theme={null} from morphik import Morphik db = Morphik() summary = db.query_document( file="notes.pdf", prompt="Summarize the key takeaways in two sentences.", ) print(summary.text_output) ``` ```python theme={null} from morphik import AsyncMorphik async def main(): async with AsyncMorphik() as db: summary = await db.query_document( file="notes.pdf", prompt="Summarize the key takeaways in two sentences.", ) print(summary.text_output) asyncio.run(main()) ``` # remove_document_from_folder Source: https://morphik.ai/docs/python-sdk/remove_document_from_folder Remove a document from a folder ```python theme={null} def remove_document_from_folder( folder_id_or_name: str, document_id: str, ) -> Dict[str, str] ``` ```python theme={null} async def remove_document_from_folder( folder_id_or_name: str, document_id: str, ) -> Dict[str, str] ``` ## Parameters * `folder_id_or_name` (str): Folder identifier. Accepts the folder's UUID, name, or canonical path (e.g., `/projects/alpha/specs`; leading slash optional). * `document_id` (str): Identifier of the document to remove. ## Returns * `Dict[str, str]`: Dictionary with `status` and `message` describing the outcome. ## Examples ```python theme={null} from morphik import Morphik db = Morphik() folder = db.get_folder("marketing_docs") db.remove_document_from_folder(folder.id, "doc_123") db.remove_document_from_folder("/projects/alpha/specs", "doc_456") ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: folder = await db.get_folder("marketing_docs") await db.remove_document_from_folder(folder.id, "doc_123") await db.remove_document_from_folder("/projects/alpha/specs", "doc_456") ``` # rename_app Source: https://morphik.ai/docs/python-sdk/rename_app Rename a cloud app by ID or current name ```python theme={null} def rename_app( new_name: str, app_id: Optional[str] = None, app_name: Optional[str] = None, ) -> Dict[str, Any] ``` ```python theme={null} async def rename_app( new_name: str, app_id: Optional[str] = None, app_name: Optional[str] = None, ) -> Dict[str, Any] ``` ## Parameters * `new_name` (str): New app name * `app_id` (str, optional): App ID to rename * `app_name` (str, optional): Current app name to rename ## Returns * `Dict[str, Any]`: API response with rename status ## Examples ```python theme={null} db.rename_app(new_name="prod-app", app_name="staging-app") ``` # requeue_ingestion_jobs Source: https://morphik.ai/docs/python-sdk/requeue_ingestion_jobs Requeue ingestion jobs for failed or stuck documents ```python theme={null} def requeue_ingestion_jobs( *, jobs: Optional[List[Union[RequeueIngestionJob, Dict[str, Any]]]] = None, include_all: bool = False, statuses: Optional[List[str]] = None, limit: Optional[int] = None, ) -> RequeueIngestionResponse ``` ```python theme={null} async def requeue_ingestion_jobs( *, jobs: Optional[List[Union[RequeueIngestionJob, Dict[str, Any]]]] = None, include_all: bool = False, statuses: Optional[List[str]] = None, limit: Optional[int] = None, ) -> RequeueIngestionResponse ``` ## Parameters * `jobs` (List\[RequeueIngestionJob | Dict\[str, Any]], optional): Specific jobs to requeue * `include_all` (bool, optional): Requeue all matching jobs. Defaults to False. * `statuses` (List\[str], optional): Limit to specific statuses (for example `["failed"]`) * `limit` (int, optional): Limit the number of jobs to requeue ## Returns * `RequeueIngestionResponse`: Result details for each requeued job ## Notes * You must provide either `jobs` or `include_all=True`. ## Examples ```python theme={null} from morphik import Morphik from morphik.models import RequeueIngestionJob db = Morphik() resp = db.requeue_ingestion_jobs( jobs=[RequeueIngestionJob(external_id="doc_123")], ) print(resp.results) ``` # retrieve_chunks Source: https://morphik.ai/docs/python-sdk/retrieve_chunks Retrieve relevant chunks from Morphik ```python theme={null} def retrieve_chunks( query: Optional[str] = None, filters: Optional[Dict[str, Any]] = None, k: int = 4, min_score: float = 0.0, use_colpali: bool = True, folder_name: Optional[Union[str, List[str]]] = None, folder_depth: Optional[int] = None, padding: int = 0, output_format: Optional[str] = None, query_image: Optional[str] = None, ) -> List[FinalChunkResult] ``` ```python theme={null} async def retrieve_chunks( query: Optional[str] = None, filters: Optional[Dict[str, Any]] = None, k: int = 4, min_score: float = 0.0, use_colpali: bool = True, folder_name: Optional[Union[str, List[str]]] = None, folder_depth: Optional[int] = None, padding: int = 0, output_format: Optional[str] = None, query_image: Optional[str] = None, ) -> List[FinalChunkResult] ``` ## Parameters * `query` (str, optional): Search query text. Mutually exclusive with `query_image`. * `filters` (Dict\[str, Any], optional): Optional metadata filters * `k` (int, optional): Number of results. Defaults to 4. * `min_score` (float, optional): Minimum similarity threshold. Defaults to 0.0. * `use_colpali` (bool, optional): Whether to use ColPali-style embedding model to retrieve the chunks (only works for documents ingested with `use_colpali=True`). Defaults to True. * `folder_name` (str | List\[str], optional): Optional folder scope. Accepts canonical paths (e.g., `/projects/alpha/specs`) or a list of paths/names. * `folder_depth` (int, optional): Folder scope depth. `None`/`0` = exact match, `-1` = include all descendants, `n > 0` = include descendants up to `n` levels deep. * `padding` (int, optional): Number of additional chunks/pages to retrieve before and after matched chunks (ColPali only). Defaults to 0. * `output_format` (str, optional): Controls how image chunks are returned: * `"base64"` (default): Returns base64-encoded image data * `"url"`: Returns presigned HTTPS URLs * `"text"`: Converts images to markdown text via OCR * `query_image` (str, optional): Base64-encoded image for reverse image search. Mutually exclusive with `query`. Requires `use_colpali=True`. ## Metadata Filters Filters follow the same JSON syntax across the API. See the [Metadata Filtering guide](/concepts/metadata-filtering) for supported operators and typed comparisons. Example: ```python theme={null} filters = { "$and": [ {"department": {"$eq": "research"}}, {"priority": {"$gte": 40}}, {"start_date": {"$lte": "2024-06-01T00:00:00Z"}} ] } chunks = db.retrieve_chunks("delta status", filters=filters, k=6) ``` ## Returns * `List[FinalChunkResult]`: List of chunk results ## Examples ```python theme={null} from morphik import Morphik db = Morphik() chunks = db.retrieve_chunks( "What are the key findings?", filters={"department": "research"}, k=5, min_score=0.5, padding=1, output_format="url", # Return image chunks as presigned URLs ) nested_chunks = db.retrieve_chunks( "design decisions", folder_name="/projects/alpha", folder_depth=-1, # include nested child folders ) for chunk in chunks: print(f"Score: {chunk.score}") # For image chunks with output_format="url", content will be a URL string print(f"Content: {chunk.content}") print(f"Document ID: {chunk.document_id}") print(f"Chunk Number: {chunk.chunk_number}") print(f"Metadata: {chunk.metadata}") print("---") ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: chunks = await db.retrieve_chunks( "What are the key findings?", filters={"department": "research"}, k=5, min_score=0.5, padding=1, output_format="url", # Return image chunks as presigned URLs ) nested_chunks = await db.retrieve_chunks( "design decisions", folder_name="/projects/alpha", folder_depth=-1, ) for chunk in chunks: print(f"Score: {chunk.score}") # For image chunks with output_format="url", content will be a URL string print(f"Content: {chunk.content}") print(f"Document ID: {chunk.document_id}") print(f"Chunk Number: {chunk.chunk_number}") print(f"Metadata: {chunk.metadata}") print("---") ``` ## FinalChunkResult Properties The `FinalChunkResult` objects returned by this method have the following properties: * `content` (str | PILImage): Chunk content (text or image) * `score` (float): Relevance score * `document_id` (str): Parent document ID * `chunk_number` (int): Chunk sequence number * `metadata` (Dict\[str, Any]): Document metadata * `content_type` (str): Content type * `filename` (Optional\[str]): Original filename * `download_url` (Optional\[str]): URL to download full document ## Output Format Options * **`"base64"` (default)**: Image chunks are returned as base64 data (the SDK attempts to decode these into a `PIL.Image` for `FinalChunkResult.content`). * **`"url"`**: Image chunks are returned as presigned HTTPS URLs in `content`. This is convenient for UIs and LLMs that accept remote image URLs (e.g., via `image_url`). * **`"text"`**: Image chunks are converted to markdown text via OCR. Use this when you need faster inference or when documents are mostly text-based. * Text chunks are unaffected by `output_format` and are always returned as strings. * The `download_url` field may be populated for image chunks. When using `output_format="url"`, it will typically match `content` for those chunks. ### When to Use Each Format | Format | Best For | | -------- | --------------------------------------------------------------- | | `base64` | Direct image processing, local applications | | `url` | Web UIs, LLMs with vision capabilities (lighter on network) | | `text` | Faster inference, text-heavy documents, context length concerns | **base64 vs url**: Both formats pass images to LLMs for visual understanding and produce similar results. However, `url` is lighter on network transfer since only the URL is sent to your application (the LLM fetches the image directly). This can result in faster response times, especially with multiple images. **When to use text**: Passing images to LLMs for inference can be slow and consume significant context tokens. Use `output_format="text"` when you need faster inference speeds or when your documents are primarily text-based. If you're hitting context limits with images, it may be because they aren't being passed correctly to the model. See [Generating Completions with Retrieved Chunks](/cookbooks/generating-completions-with-retrieved-chunks) for examples of properly passing images (both base64 and URLs) to vision-capable models like GPT-4o. Tip: To download the original raw file for a document, use [`get_document_download_url`](./get_document_download_url). ## Reverse Image Search You can search using an image instead of text by providing `query_image` with a base64-encoded image. This enables finding visually similar content in your documents. ```python theme={null} import base64 from morphik import Morphik db = Morphik() # Load and encode your query image with open("query_image.png", "rb") as f: image_b64 = base64.b64encode(f.read()).decode("utf-8") # Search using the image chunks = db.retrieve_chunks( query_image=image_b64, use_colpali=True, # Required for image queries k=5, ) for chunk in chunks: print(f"Score: {chunk.score}") print(f"Document ID: {chunk.document_id}") print("---") ``` ```python theme={null} import base64 from morphik import AsyncMorphik async with AsyncMorphik() as db: # Load and encode your query image with open("query_image.png", "rb") as f: image_b64 = base64.b64encode(f.read()).decode("utf-8") # Search using the image chunks = await db.retrieve_chunks( query_image=image_b64, use_colpali=True, # Required for image queries k=5, ) for chunk in chunks: print(f"Score: {chunk.score}") print(f"Document ID: {chunk.document_id}") print("---") ``` Reverse image search requires documents to be ingested with `use_colpali=True`. You must provide either `query` or `query_image`, but not both. # retrieve_chunks_grouped Source: https://morphik.ai/docs/python-sdk/retrieve_chunks_grouped Retrieve relevant chunks with grouping for UI display ```python theme={null} def retrieve_chunks_grouped( query: Optional[str] = None, filters: Optional[Dict[str, Any]] = None, k: int = 4, min_score: float = 0.0, use_colpali: bool = True, use_reranking: Optional[bool] = None, folder_name: Optional[Union[str, List[str]]] = None, folder_depth: Optional[int] = None, end_user_id: Optional[str] = None, padding: int = 0, output_format: Optional[str] = None, query_image: Optional[str] = None, ) -> GroupedChunkResponse ``` ```python theme={null} async def retrieve_chunks_grouped( query: Optional[str] = None, filters: Optional[Dict[str, Any]] = None, k: int = 4, min_score: float = 0.0, use_colpali: bool = True, use_reranking: Optional[bool] = None, folder_name: Optional[Union[str, List[str]]] = None, folder_depth: Optional[int] = None, end_user_id: Optional[str] = None, padding: int = 0, output_format: Optional[str] = None, query_image: Optional[str] = None, ) -> GroupedChunkResponse ``` ## Parameters * `query` (str, optional): Search query text. Mutually exclusive with `query_image`. * `filters` (Dict\[str, Any], optional): Optional metadata filters * `k` (int, optional): Number of results. Defaults to 4. * `min_score` (float, optional): Minimum similarity threshold. Defaults to 0.0. * `use_colpali` (bool, optional): Whether to use ColPali-style embedding model. Defaults to True. * `use_reranking` (bool, optional): Override workspace reranking configuration for this request. * `folder_name` (str | List\[str], optional): Optional folder scope. Accepts canonical paths (e.g., `/projects/alpha/specs`) or a list of paths/names. * `folder_depth` (int, optional): Folder scope depth. `None`/`0` = exact match, `-1` = include all descendants, `n > 0` = include descendants up to `n` levels deep. * `end_user_id` (str, optional): Optional end-user scope * `padding` (int, optional): Number of additional chunks/pages to retrieve before and after matched chunks. Defaults to 0. * `output_format` (str, optional): Controls how image chunks are returned: * `"base64"` (default): Returns base64-encoded image data * `"url"`: Returns presigned HTTPS URLs * `"text"`: Converts images to markdown text via OCR (faster inference, best for text-heavy documents) * `query_image` (str, optional): Base64-encoded image for reverse image search. Mutually exclusive with `query`. Requires `use_colpali=True`. ## Returns * `GroupedChunkResponse`: Response containing both flat chunks and grouped chunks for UI display ## Metadata Filters Filters follow the same JSON syntax across the API. See the [Metadata Filtering guide](/concepts/metadata-filtering) for supported operators and typed comparisons. ## Examples ```python theme={null} from morphik import Morphik db = Morphik() # Basic grouped retrieval response = db.retrieve_chunks_grouped( query="What are the key findings?", k=5, ) print(f"Total results: {response.total_results}") print(f"Has padding: {response.has_padding}") # Access flat list of chunks (backward compatible) for chunk in response.chunks: print(f"Score: {chunk.score}, Content: {chunk.content[:100]}...") # Access grouped chunks for UI display for group in response.groups: print(f"\n--- Group with {group.total_chunks} chunks ---") print(f"Main chunk: {group.main_chunk.content[:100]}...") for padding_chunk in group.padding_chunks: print(f" Padding: {padding_chunk.content[:50]}...") # With padding for context response = db.retrieve_chunks_grouped( query="quarterly results", k=3, padding=2, # Get 2 chunks before/after each match folder_name="/projects/reports", folder_depth=-1, ) ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: # Basic grouped retrieval response = await db.retrieve_chunks_grouped( query="What are the key findings?", k=5, ) print(f"Total results: {response.total_results}") print(f"Has padding: {response.has_padding}") # Access flat list of chunks (backward compatible) for chunk in response.chunks: print(f"Score: {chunk.score}, Content: {chunk.content[:100]}...") # Access grouped chunks for UI display for group in response.groups: print(f"\n--- Group with {group.total_chunks} chunks ---") print(f"Main chunk: {group.main_chunk.content[:100]}...") for padding_chunk in group.padding_chunks: print(f" Padding: {padding_chunk.content[:50]}...") # With padding for context response = await db.retrieve_chunks_grouped( query="quarterly results", k=3, padding=2, # Get 2 chunks before/after each match folder_name="/projects/reports", folder_depth=-1, ) ``` ## GroupedChunkResponse Properties The `GroupedChunkResponse` object has the following properties: * `chunks` (List\[ChunkResult]): Flat list of all chunks (for backward compatibility) * `groups` (List\[ChunkGroup]): Grouped chunks for UI display * `total_results` (int): Total number of unique chunks * `has_padding` (bool): Whether padding was applied to any results ## ChunkGroup Properties Each `ChunkGroup` in `groups` has: * `main_chunk` (ChunkResult): The primary matched chunk * `padding_chunks` (List\[ChunkResult]): Surrounding context chunks * `total_chunks` (int): Total number of chunks in this group ## Notes * This method is similar to [`retrieve_chunks`](./retrieve_chunks) but provides additional grouping for UI display. * The `chunks` list provides backward compatibility with flat chunk lists. * The `groups` list organizes results with their padding context, ideal for building search result UIs. * When `padding` is specified, surrounding chunks are included in `padding_chunks` for each group. ## Reverse Image Search You can search using an image instead of text by providing `query_image` with a base64-encoded image: ```python theme={null} import base64 from morphik import Morphik db = Morphik() # Load and encode your query image with open("query_image.png", "rb") as f: image_b64 = base64.b64encode(f.read()).decode("utf-8") # Search using the image with grouped results response = db.retrieve_chunks_grouped( query_image=image_b64, use_colpali=True, # Required for image queries k=5, padding=1, ) for group in response.groups: print(f"Main chunk score: {group.main_chunk.score}") print(f"Document: {group.main_chunk.document_id}") print("---") ``` ```python theme={null} import base64 from morphik import AsyncMorphik async with AsyncMorphik() as db: # Load and encode your query image with open("query_image.png", "rb") as f: image_b64 = base64.b64encode(f.read()).decode("utf-8") # Search using the image with grouped results response = await db.retrieve_chunks_grouped( query_image=image_b64, use_colpali=True, # Required for image queries k=5, padding=1, ) for group in response.groups: print(f"Main chunk score: {group.main_chunk.score}") print(f"Document: {group.main_chunk.document_id}") print("---") ``` Reverse image search requires documents to be ingested with `use_colpali=True`. You must provide either `query` or `query_image`, but not both. # retrieve_docs Source: https://morphik.ai/docs/python-sdk/retrieve_docs Retrieve relevant documents from Morphik ```python theme={null} def retrieve_docs( query: str, filters: Optional[Dict[str, Any]] = None, k: int = 4, min_score: float = 0.0, use_colpali: bool = True, use_reranking: Optional[bool] = None, folder_name: Optional[Union[str, List[str]]] = None, folder_depth: Optional[int] = None, ) -> List[DocumentResult] ``` ```python theme={null} async def retrieve_docs( query: str, filters: Optional[Dict[str, Any]] = None, k: int = 4, min_score: float = 0.0, use_colpali: bool = True, use_reranking: Optional[bool] = None, folder_name: Optional[Union[str, List[str]]] = None, folder_depth: Optional[int] = None, ) -> List[DocumentResult] ``` ## Parameters * `query` (str): Search query text * `filters` (Dict\[str, Any], optional): Optional metadata filters * `k` (int, optional): Number of results. Defaults to 4. * `min_score` (float, optional): Minimum similarity threshold. Defaults to 0.0. * `use_colpali` (bool, optional): Whether to use ColPali-style embedding model to retrieve the documents (only works for documents ingested with `use_colpali=True`). Defaults to True. * `use_reranking` (bool, optional): Override workspace reranking configuration for this request. * `folder_name` (str | List\[str], optional): Optional folder scope. Accepts canonical paths (e.g., `/projects/alpha/specs`) or a list of paths/names. * `folder_depth` (int, optional): Folder scope depth. `None`/`0` = exact match, `-1` = include all descendants, `n > 0` = include descendants up to `n` levels deep. ## Metadata Filters Filters share a common JSON DSL. Review the [Metadata Filtering guide](/concepts/metadata-filtering) for supported operators and typed comparisons. Example: ```python theme={null} filters = { "$and": [ {"department": {"$eq": "research"}}, {"priority": {"$gte": 40}}, {"start_date": {"$lte": "2024-06-01"}} ] } docs = db.retrieve_docs("budget summary", filters=filters, k=5) ``` ## Returns * `List[DocumentResult]`: List of document results ## Examples ```python theme={null} from morphik import Morphik db = Morphik() docs = db.retrieve_docs( "machine learning", k=5, min_score=0.5 ) nested_docs = db.retrieve_docs( "design notes", folder_name="/projects/alpha", folder_depth=-1, ) for doc in docs: print(f"Score: {doc.score}") print(f"Document ID: {doc.document_id}") print(f"Metadata: {doc.metadata}") print(f"Content: {doc.content}") print("---") ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: docs = await db.retrieve_docs( "machine learning", k=5, min_score=0.5 ) nested_docs = await db.retrieve_docs( "design notes", folder_name="/projects/alpha", folder_depth=-1, ) for doc in docs: print(f"Score: {doc.score}") print(f"Document ID: {doc.document_id}") print(f"Metadata: {doc.metadata}") print(f"Content: {doc.content}") print("---") ``` ## DocumentResult Properties The `DocumentResult` objects returned by this method have the following properties: * `score` (float): Relevance score * `document_id` (str): Document ID * `metadata` (Dict\[str, Any]): Document metadata * `content` (DocumentContent): Document content or URL # rotate_app_token Source: https://morphik.ai/docs/python-sdk/rotate_app_token Rotate an app token by ID or name ```python theme={null} def rotate_app_token( app_id: Optional[str] = None, app_name: Optional[str] = None, expiry_days: Optional[int] = None, ) -> Dict[str, Any] ``` ```python theme={null} async def rotate_app_token( app_id: Optional[str] = None, app_name: Optional[str] = None, expiry_days: Optional[int] = None, ) -> Dict[str, Any] ``` ## Parameters * `app_id` (str, optional): App ID to rotate * `app_name` (str, optional): App name to rotate * `expiry_days` (int, optional): New token expiry in days ## Returns * `Dict[str, Any]`: API response containing the rotated token and metadata ## Examples ```python theme={null} db.rotate_app_token(app_name="demo", expiry_days=30) ``` # search_documents Source: https://morphik.ai/docs/python-sdk/search_documents Search for documents by name or filename ```python theme={null} def search_documents( query: str, limit: int = 10, filters: Optional[Dict[str, Any]] = None, folder_name: Optional[Union[str, List[str]]] = None, folder_depth: Optional[int] = None, end_user_id: Optional[str] = None, ) -> List[Document] ``` ```python theme={null} async def search_documents( query: str, limit: int = 10, filters: Optional[Dict[str, Any]] = None, folder_name: Optional[Union[str, List[str]]] = None, folder_depth: Optional[int] = None, end_user_id: Optional[str] = None, ) -> List[Document] ``` ## Parameters * `query` (str): Search query for document names/filenames * `limit` (int, optional): Maximum number of documents to return. Defaults to 10. * `filters` (Dict\[str, Any], optional): Optional metadata filters * `folder_name` (str | List\[str], optional): Optional folder scope (canonical path or list of paths/names) * `folder_depth` (int, optional): Folder scope depth. `None`/`0` = exact match, `-1` = include all descendants, `n > 0` = include descendants up to `n` levels deep. * `end_user_id` (str, optional): Optional end-user scope ## Returns * `List[Document]`: List of matching documents ## Metadata Filters Filters follow the same JSON syntax across the API. See the [Metadata Filtering guide](/concepts/metadata-filtering) for supported operators and typed comparisons. Example: ```python theme={null} filters = { "$and": [ {"department": {"$eq": "research"}}, {"priority": {"$gte": 40}}, ] } docs = db.search_documents("report", filters=filters) ``` ## Examples ```python theme={null} from morphik import Morphik db = Morphik() # Basic document name search docs = db.search_documents("quarterly report") for doc in docs: print(f"{doc.external_id}: {doc.filename}") # Search with limit and filters docs = db.search_documents( query="invoice", limit=20, filters={"department": "finance"}, ) # Search within specific folders docs = db.search_documents( query="contract", folder_name="/projects/legal", folder_depth=-1, ) # Search scoped to an end user docs = db.search_documents( query="notes", end_user_id="user_456", ) ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: # Basic document name search docs = await db.search_documents("quarterly report") for doc in docs: print(f"{doc.external_id}: {doc.filename}") # Search with limit and filters docs = await db.search_documents( query="invoice", limit=20, filters={"department": "finance"}, ) # Search within specific folders docs = await db.search_documents( query="contract", folder_name="/projects/legal", folder_depth=-1, ) # Search scoped to an end user docs = await db.search_documents( query="notes", end_user_id="user_456", ) ``` ## Notes * This method searches document names and filenames, not document content. For content-based search, use [`retrieve_chunks`](./retrieve_chunks) or [`retrieve_docs`](./retrieve_docs). * The `folder_name` parameter accepts a canonical path (leading slash optional) or a list of paths/names; combine with `folder_depth` to include descendants. * Results are returned sorted by relevance to the search query. # signin Source: https://morphik.ai/docs/python-sdk/signin Create a user scope for end-user isolation ```python theme={null} def signin( end_user_id: str, ) -> UserScope ``` ```python theme={null} async def signin( end_user_id: str, ) -> AsyncUserScope ``` ## Parameters * `end_user_id` (str): End-user identifier to scope all operations ## Returns * `UserScope` / `AsyncUserScope`: Scoped client that automatically includes `end_user_id` ## Examples ```python theme={null} from morphik import Morphik db = Morphik() user = db.signin("user_123") docs = user.list_documents() ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: user = db.signin("user_123") docs = await user.list_documents() ``` ## Notes * You can also scope a folder to a user: `folder.signin("user_123")`. # update_document_by_filename_metadata Source: https://morphik.ai/docs/python-sdk/update_document_by_filename_metadata Update a document's metadata using filename to identify the document ```python theme={null} def update_document_by_filename_metadata( filename: str, metadata: Dict[str, Any], new_filename: Optional[str] = None, ) -> Document ``` ```python theme={null} async def update_document_by_filename_metadata( filename: str, metadata: Dict[str, Any], new_filename: Optional[str] = None, ) -> Document ``` ## Parameters * `filename` (str): Filename of the document to update * `metadata` (Dict\[str, Any]): Metadata to update * `new_filename` (str, optional): Optional new filename to assign to the document ## Returns * `Document`: Updated document metadata ## Examples ```python theme={null} from morphik import Morphik db = Morphik() # Update just the metadata of a document identified by filename updated_doc = db.update_document_by_filename_metadata( filename="report.pdf", metadata={"status": "reviewed", "reviewer": "Jane Smith"}, new_filename="reviewed_report.pdf" # Optional: rename the file ) print(f"Updated metadata: {updated_doc.metadata}") ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: # Update just the metadata of a document identified by filename updated_doc = await db.update_document_by_filename_metadata( filename="report.pdf", metadata={"status": "reviewed", "reviewer": "Jane Smith"}, new_filename="reviewed_report.pdf" # Optional: rename the file ) print(f"Updated metadata: {updated_doc.metadata}") ``` # update_document_by_filename_with_file Source: https://morphik.ai/docs/python-sdk/update_document_by_filename_with_file Update a document identified by filename with content from a file ```python theme={null} def update_document_by_filename_with_file( filename: str, file: Union[str, bytes, BinaryIO, Path], new_filename: Optional[str] = None, metadata: Optional[Dict[str, Any]] = None, update_strategy: str = "add", use_colpali: Optional[bool] = None, ) -> Document ``` ```python theme={null} async def update_document_by_filename_with_file( filename: str, file: Union[str, bytes, BinaryIO, Path], new_filename: Optional[str] = None, metadata: Optional[Dict[str, Any]] = None, update_strategy: str = "add", use_colpali: Optional[bool] = None, ) -> Document ``` ## Parameters * `filename` (str): Filename of the document to update * `file` (Union\[str, bytes, BinaryIO, Path]): File to add (path string, bytes, file object, or Path) * `new_filename` (str, optional): Optional new filename for the document (defaults to the filename of the file) * `metadata` (Dict\[str, Any], optional): Additional metadata to update * `update_strategy` (str, optional): Strategy for updating the document (currently only 'add' is supported). Defaults to 'add'. * `use_colpali` (bool, optional): Whether to use multi-vector embedding. If not specified, defaults to True. ## Returns * `Document`: Updated document metadata ## Examples ```python theme={null} from morphik import Morphik db = Morphik() # Add content from a file to an existing document identified by filename updated_doc = db.update_document_by_filename_with_file( filename="report.pdf", file="path/to/update.pdf", metadata={"status": "updated"}, update_strategy="add" ) print(f"Document version: {updated_doc.system_metadata.get('version')}") ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: # Add content from a file to an existing document identified by filename updated_doc = await db.update_document_by_filename_with_file( filename="report.pdf", file="path/to/update.pdf", metadata={"status": "updated"}, update_strategy="add" ) print(f"Document version: {updated_doc.system_metadata.get('version')}") ``` # update_document_by_filename_with_text Source: https://morphik.ai/docs/python-sdk/update_document_by_filename_with_text Update a document identified by filename with new text content ```python theme={null} def update_document_by_filename_with_text( filename: str, content: str, new_filename: Optional[str] = None, metadata: Optional[Dict[str, Any]] = None, update_strategy: str = "add", use_colpali: Optional[bool] = None, ) -> Document ``` ```python theme={null} async def update_document_by_filename_with_text( filename: str, content: str, new_filename: Optional[str] = None, metadata: Optional[Dict[str, Any]] = None, update_strategy: str = "add", use_colpali: Optional[bool] = None, ) -> Document ``` ## Parameters * `filename` (str): Filename of the document to update * `content` (str): The new content to add * `new_filename` (str, optional): Optional new filename for the document * `metadata` (Dict\[str, Any], optional): Additional metadata to update * `update_strategy` (str, optional): Strategy for updating the document (currently only 'add' is supported). Defaults to 'add'. * `use_colpali` (bool, optional): Whether to use multi-vector embedding. If not specified, defaults to True. ## Returns * `Document`: Updated document metadata ## Examples ```python theme={null} from morphik import Morphik db = Morphik() # Add new content to an existing document identified by filename updated_doc = db.update_document_by_filename_with_text( filename="report.pdf", content="This is additional content that will be appended to the document.", new_filename="updated_report.pdf", metadata={"category": "updated"}, update_strategy="add" ) print(f"Document version: {updated_doc.system_metadata.get('version')}") ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: # Add new content to an existing document identified by filename updated_doc = await db.update_document_by_filename_with_text( filename="report.pdf", content="This is additional content that will be appended to the document.", new_filename="updated_report.pdf", metadata={"category": "updated"}, update_strategy="add" ) print(f"Document version: {updated_doc.system_metadata.get('version')}") ``` # update_document_metadata Source: https://morphik.ai/docs/python-sdk/update_document_metadata Update a document's metadata only ```python theme={null} def update_document_metadata( document_id: str, metadata: Dict[str, Any], ) -> Document ``` ```python theme={null} async def update_document_metadata( document_id: str, metadata: Dict[str, Any], ) -> Document ``` ## Parameters * `document_id` (str): ID of the document to update * `metadata` (Dict\[str, Any]): Metadata to update ### Typed Metadata You can supply Python `datetime`, `date`, `Decimal`, and numeric types in `metadata`. The SDK serializes them with the correct `metadata_types`, so subsequent filters can leverage the operators from the [Metadata Filtering guide](/concepts/metadata-filtering). ## Returns * `Document`: Updated document metadata ## Examples ```python theme={null} from morphik import Morphik db = Morphik() # Update just the metadata of a document updated_doc = db.update_document_metadata( document_id="doc_123", metadata={"status": "reviewed", "reviewer": "Jane Smith"} ) print(f"Updated metadata: {updated_doc.metadata}") ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: # Update just the metadata of a document updated_doc = await db.update_document_metadata( document_id="doc_123", metadata={"status": "reviewed", "reviewer": "Jane Smith"} ) print(f"Updated metadata: {updated_doc.metadata}") ``` # update_document_with_file Source: https://morphik.ai/docs/python-sdk/update_document_with_file Update a document with content from a file ```python theme={null} def update_document_with_file( document_id: str, file: Union[str, bytes, BinaryIO, Path], filename: Optional[str] = None, metadata: Optional[Dict[str, Any]] = None, update_strategy: str = "add", use_colpali: Optional[bool] = None, ) -> Document ``` ```python theme={null} async def update_document_with_file( document_id: str, file: Union[str, bytes, BinaryIO, Path], filename: Optional[str] = None, metadata: Optional[Dict[str, Any]] = None, update_strategy: str = "add", use_colpali: Optional[bool] = None, ) -> Document ``` ## Parameters * `document_id` (str): ID of the document to update * `file` (Union\[str, bytes, BinaryIO, Path]): File to add (path string, bytes, file object, or Path) * `filename` (str, optional): Name of the file * `metadata` (Dict\[str, Any], optional): Additional metadata to update * `update_strategy` (str, optional): Strategy for updating the document (currently only 'add' is supported). Defaults to 'add'. * `use_colpali` (bool, optional): Whether to use multi-vector embedding. If not specified, defaults to True. ## Returns * `Document`: Updated document metadata ## Examples ```python theme={null} from morphik import Morphik db = Morphik() # Add content from a file to an existing document updated_doc = db.update_document_with_file( document_id="doc_123", file="path/to/update.pdf", metadata={"status": "updated"}, update_strategy="add" ) print(f"Document version: {updated_doc.system_metadata.get('version')}") ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: # Add content from a file to an existing document updated_doc = await db.update_document_with_file( document_id="doc_123", file="path/to/update.pdf", metadata={"status": "updated"}, update_strategy="add" ) print(f"Document version: {updated_doc.system_metadata.get('version')}") ``` # update_document_with_text Source: https://morphik.ai/docs/python-sdk/update_document_with_text Update a document with new text content ```python theme={null} def update_document_with_text( document_id: str, content: str, filename: Optional[str] = None, metadata: Optional[Dict[str, Any]] = None, update_strategy: str = "add", use_colpali: Optional[bool] = None, ) -> Document ``` ```python theme={null} async def update_document_with_text( document_id: str, content: str, filename: Optional[str] = None, metadata: Optional[Dict[str, Any]] = None, update_strategy: str = "add", use_colpali: Optional[bool] = None, ) -> Document ``` ## Parameters * `document_id` (str): ID of the document to update * `content` (str): The new content to add * `filename` (str, optional): Optional new filename for the document * `metadata` (Dict\[str, Any], optional): Additional metadata to update * `update_strategy` (str, optional): Strategy for updating the document (currently only 'add' is supported). Defaults to 'add'. * `use_colpali` (bool, optional): Whether to use multi-vector embedding. If not specified, defaults to True. ## Returns * `Document`: Updated document metadata ## Examples ```python theme={null} from morphik import Morphik db = Morphik() # Add new content to an existing document updated_doc = db.update_document_with_text( document_id="doc_123", content="This is additional content that will be appended to the document.", filename="updated_document.txt", metadata={"category": "updated"}, update_strategy="add" ) print(f"Document version: {updated_doc.system_metadata.get('version')}") ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: # Add new content to an existing document updated_doc = await db.update_document_with_text( document_id="doc_123", content="This is additional content that will be appended to the document.", filename="updated_document.txt", metadata={"category": "updated"}, update_strategy="add" ) print(f"Document version: {updated_doc.system_metadata.get('version')}") ``` # upsert_document_summary Source: https://morphik.ai/docs/python-sdk/upsert_document_summary Create or update a document summary ```python theme={null} def upsert_document_summary( document_id: str, content: str, versioning: bool = True, overwrite_latest: bool = False, ) -> Summary ``` ```python theme={null} async def upsert_document_summary( document_id: str, content: str, versioning: bool = True, overwrite_latest: bool = False, ) -> Summary ``` ## Parameters * `document_id` (str): ID of the document * `content` (str): Summary content (markdown or plain text) * `versioning` (bool, optional): Create a new version instead of overwriting. Defaults to True. * `overwrite_latest` (bool, optional): Overwrite the latest summary when versioning is enabled. Defaults to False. ## Returns * `Summary`: Updated summary payload ## Examples ```python theme={null} from morphik import Morphik db = Morphik() summary = db.upsert_document_summary( document_id="doc_123", content="This report summarizes Q2 performance.", ) print(summary.version) ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: summary = await db.upsert_document_summary( document_id="doc_123", content="This report summarizes Q2 performance.", ) print(summary.version) ``` # upsert_folder_summary Source: https://morphik.ai/docs/python-sdk/upsert_folder_summary Create or update a folder summary ```python theme={null} def upsert_folder_summary( folder_id_or_path: str, content: str, versioning: bool = True, overwrite_latest: bool = False, ) -> Summary ``` ```python theme={null} async def upsert_folder_summary( folder_id_or_path: str, content: str, versioning: bool = True, overwrite_latest: bool = False, ) -> Summary ``` ## Parameters * `folder_id_or_path` (str): Folder identifier (UUID, name, or canonical path) * `content` (str): Summary content (markdown or plain text) * `versioning` (bool, optional): Create a new version instead of overwriting. Defaults to True. * `overwrite_latest` (bool, optional): Overwrite the latest summary when versioning is enabled. Defaults to False. ## Returns * `Summary`: Updated summary payload ## Examples ```python theme={null} from morphik import Morphik db = Morphik() summary = db.upsert_folder_summary( folder_id_or_path="/projects/alpha", content="Summary of project alpha documents.", ) print(summary.version) ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: summary = await db.upsert_folder_summary( folder_id_or_path="/projects/alpha", content="Summary of project alpha documents.", ) print(summary.version) ``` # upsert_summary Source: https://morphik.ai/docs/python-sdk/upsert_summary Create or update a folder summary This method is available on `Folder` objects. ```python theme={null} def upsert_summary( content: str, versioning: bool = True, overwrite_latest: bool = False, ) -> Summary ``` ```python theme={null} async def upsert_summary( content: str, versioning: bool = True, overwrite_latest: bool = False, ) -> Summary ``` ## Parameters * `content` (str): Summary content (markdown or plain text) * `versioning` (bool, optional): Create a new version instead of overwriting. Defaults to True. * `overwrite_latest` (bool, optional): Overwrite the latest summary when versioning is enabled. Defaults to False. ## Returns * `Summary`: Updated summary payload ## Examples ```python theme={null} from morphik import Morphik db = Morphik() folder = db.get_folder("/projects/alpha") summary = folder.upsert_summary("Summary of project alpha.") print(summary.version) ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: folder = await db.get_folder("/projects/alpha") summary = await folder.upsert_summary("Summary of project alpha.") print(summary.version) ``` # User Management Source: https://morphik.ai/docs/python-sdk/users Organize and isolate data by end user in Morphik ## Overview User scoping in Morphik allows multi-tenant applications to isolate data on a per-user basis. This ensures that in applications serving multiple users, each user can only access their own documents and data. User scoping is particularly valuable for building customer-facing applications where data privacy and separation are essential. ## Creating User Scopes ```python theme={null} from morphik import Morphik db = Morphik() # Create a user scope for a specific end user user_scope = db.signin("customer_12345") # All operations from this scope are now isolated to this user ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: # Create a user scope for a specific end user user_scope = db.signin("customer_12345") # All operations from this scope are now isolated to this user ``` ## Operations Within a User Scope Once you have a user scope, all operations performed on it are automatically scoped to that specific user. Documents created or retrieved are only accessible to that user. ```python theme={null} # Get a user scope user_scope = db.signin("customer_12345") # Ingest a document for this user doc = user_scope.ingest_text("My private notes", filename="notes.txt", metadata={"type": "personal"}) # List only this user's documents user_docs = user_scope.list_documents() # Search only within this user's documents results = user_scope.retrieve_chunks("notes") # Query using only this user's documents as context response = user_scope.query("What are my notes about?") ``` ```python theme={null} # Get a user scope user_scope = db.signin("customer_12345") # Ingest a document for this user doc = await user_scope.ingest_text("My private notes", filename="notes.txt", metadata={"type": "personal"}) # List only this user's documents user_docs = await user_scope.list_documents() # Search only within this user's documents results = await user_scope.retrieve_chunks("notes") # Query using only this user's documents as context response = await user_scope.query("What are my notes about?") ``` ## User Scope Methods The UserScope class provides the same document operations as the main Morphik client, but automatically scoped to the specific user: * `ingest_text` - Ingest text content for this user * `ingest_file` - Ingest a file for this user * `ingest_files` - Ingest multiple files for this user * `ingest_directory` - Ingest all files from a directory for this user * `retrieve_chunks` - Retrieve chunks matching a query from this user's documents (supports [reverse image search](/python-sdk/retrieve_chunks#reverse-image-search)) * `retrieve_docs` - Retrieve documents matching a query from this user's documents * `query` - Generate a completion using context from this user's documents (supports `llm_config` parameter for custom LLM configuration) * `list_documents` - List all documents owned by this user * `batch_get_documents` - Get multiple documents by their IDs for this user * `batch_get_chunks` - Get specific chunks by source for this user * `delete_document_by_filename` - Delete a document by filename for this user ## Scoping to Multiple Folders User scopes can include additional folder filters using `additional_folders` on retrieval/list/query helpers: ```python theme={null} user_scope = db.signin("user123") # Search within specific folders owned by this user results = user_scope.retrieve_chunks( "contract terms", additional_folders=["/legal/contracts", "/legal/archives"], ) ``` ## Using Custom LLM Configuration with User Scopes You can pass a custom LLM configuration when querying within a user scope: ```python theme={null} from morphik import Morphik db = Morphik() user_scope = db.signin("user123") # Use a specific model for this user's query response = user_scope.query( "What documents have I uploaded recently?", llm_config={ "model": "gpt-4-turbo", "temperature": 0.3, "max_tokens": 1000 } ) print(response.completion) ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: user_scope = db.signin("user123") # Use a specific model for this user's query response = await user_scope.query( "What documents have I uploaded recently?", llm_config={ "model": "gpt-4-turbo", "temperature": 0.3, "max_tokens": 1000 } ) print(response.completion) ``` ## Example: Customer Support Application A common use case for user scoping is in customer support applications where each customer has their own set of documents and needs personalized assistance: ```python theme={null} from morphik import Morphik db = Morphik("morphik://owner_id:token@api.morphik.ai") def process_customer_request(customer_id, query): # Sign in as the specific customer customer_scope = db.signin(customer_id) # List the customer's documents documents = customer_scope.list_documents() print(f"Customer {customer_id} has {len(documents)} documents") # Get personalized answer based only on this customer's documents response = customer_scope.query( query, filters={"type": "support_case"}, temperature=0.3 ) return response.completion # Handle customer requests answer = process_customer_request( "customer_12345", "What's the status of my recent support ticket?" ) print(answer) # Only includes information from this customer's documents ``` ```python theme={null} from morphik import AsyncMorphik async def process_customer_request(customer_id, query): async with AsyncMorphik("morphik://owner_id:token@api.morphik.ai") as db: # Sign in as the specific customer customer_scope = db.signin(customer_id) # List the customer's documents documents = await customer_scope.list_documents() print(f"Customer {customer_id} has {len(documents)} documents") # Get personalized answer based only on this customer's documents response = await customer_scope.query( query, filters={"type": "support_case"}, temperature=0.3 ) return response.completion # Handle customer requests import asyncio async def main(): answer = await process_customer_request( "customer_12345", "What's the status of my recent support ticket?" ) print(answer) # Only includes information from this customer's documents # Run the async function asyncio.run(main()) ``` ## Combining User Scope with Folder Scope For advanced data organization, you can combine user scoping with folder scoping. This allows you to organize by both user and functional area: ```python theme={null} # First, create a folder support_folder = db.create_folder("support_cases") # Then, scope to a specific user within that folder user_support = support_folder.signin("customer_12345") # This document is accessible only to customer_12345 # and only within the support_cases folder doc = user_support.ingest_file("path/to/case_details.pdf") # Query is scoped to just this user's documents # within the support_cases folder response = user_support.query("What's the status of my case?") ``` ```python theme={null} # First, create a folder support_folder = db.create_folder("support_cases") # Then, scope to a specific user within that folder user_support = support_folder.signin("customer_12345") # This document is accessible only to customer_12345 # and only within the support_cases folder doc = await user_support.ingest_file("path/to/case_details.pdf") # Query is scoped to just this user's documents # within the support_cases folder response = await user_support.query("What's the status of my case?") ``` ## Example: SaaS Application with Multi-Project Support Here's a more complex example showing how to build a SaaS application that supports multiple customers (tenants), each with their own multiple projects: ```python theme={null} from morphik import Morphik db = Morphik() def handle_customer_request(customer_id, project_name, query): # First, get the project folder project_folder = db.get_folder(project_name) # Then, scope to the specific customer within that project customer_project = project_folder.signin(customer_id) # Generate a response using only this customer's documents # within this specific project response = customer_project.query( query, filters={"status": "active"}, max_tokens=500 ) return { "customer_id": customer_id, "project": project_name, "query": query, "response": response.completion } # Handle request for a specific customer and project result = handle_customer_request( "acme_corp", "website_redesign", "What's our current timeline for the homepage redesign?" ) print(f"Response for {result['customer_id']} on {result['project']}:") print(result['response']) ``` ```python theme={null} from morphik import AsyncMorphik async def handle_customer_request(customer_id, project_name, query): async with AsyncMorphik() as db: # First, get the project folder project_folder = db.get_folder(project_name) # Then, scope to the specific customer within that project customer_project = project_folder.signin(customer_id) # Generate a response using only this customer's documents # within this specific project response = await customer_project.query( query, filters={"status": "active"}, max_tokens=500 ) return { "customer_id": customer_id, "project": project_name, "query": query, "response": response.completion } # Handle request for a specific customer and project import asyncio async def main(): result = await handle_customer_request( "acme_corp", "website_redesign", "What's our current timeline for the homepage redesign?" ) print(f"Response for {result['customer_id']} on {result['project']}:") print(result['response']) # Run the async function asyncio.run(main()) ``` ## Bulk Document Processing for Multiple Users When developing applications that serve multiple users, you might need to process documents in bulk. Here's an example showing how to ingest documents for multiple users: ```python theme={null} from morphik import Morphik from pathlib import Path db = Morphik() # Process documents for multiple customers def ingest_customer_documents(customer_data): """ customer_data is a list of dictionaries with: - customer_id: string identifier - document_path: path to document - metadata: optional metadata dictionary """ results = [] for item in customer_data: # Get user scope for this customer customer = db.signin(item['customer_id']) # Ingest the document for this customer try: doc = customer.ingest_file( item['document_path'], metadata=item.get('metadata', {}) ) results.append({ "customer_id": item['customer_id'], "document_id": doc.external_id, "status": "success" }) except Exception as e: results.append({ "customer_id": item['customer_id'], "document_path": str(item['document_path']), "status": "error", "error": str(e) }) return results # Example usage customer_docs = [ { "customer_id": "customer_1", "document_path": Path("/path/to/customer1_doc.pdf"), "metadata": {"type": "invoice", "amount": 1250} }, { "customer_id": "customer_2", "document_path": Path("/path/to/customer2_doc.pdf"), "metadata": {"type": "contract", "expires": "2024-05-15"} } ] results = ingest_customer_documents(customer_docs) print(f"Processed {len(results)} documents") ``` ```python theme={null} from morphik import AsyncMorphik from pathlib import Path import asyncio async def ingest_customer_documents(customer_data): """ customer_data is a list of dictionaries with: - customer_id: string identifier - document_path: path to document - metadata: optional metadata dictionary """ async with AsyncMorphik() as db: results = [] for item in customer_data: # Get user scope for this customer customer = db.signin(item['customer_id']) # Ingest the document for this customer try: doc = await customer.ingest_file( item['document_path'], metadata=item.get('metadata', {}) ) results.append({ "customer_id": item['customer_id'], "document_id": doc.external_id, "status": "success" }) except Exception as e: results.append({ "customer_id": item['customer_id'], "document_path": str(item['document_path']), "status": "error", "error": str(e) }) return results # Example usage async def main(): customer_docs = [ { "customer_id": "customer_1", "document_path": Path("/path/to/customer1_doc.pdf"), "metadata": {"type": "invoice", "amount": 1250} }, { "customer_id": "customer_2", "document_path": Path("/path/to/customer2_doc.pdf"), "metadata": {"type": "contract", "expires": "2024-05-15"} } ] results = await ingest_customer_documents(customer_docs) print(f"Processed {len(results)} documents") # Run the async function asyncio.run(main()) ``` See [Folder Management](/python-sdk/folders) for more details on working with folder scopes. # wait_for_document_completion Source: https://morphik.ai/docs/python-sdk/wait_for_document_completion Block until a document finishes processing ```python theme={null} def wait_for_document_completion( document_id: str, timeout_seconds: int = 300, check_interval_seconds: int = 2, progress_callback: Optional[Callable[[int, int, str, float], None]] = None, ) -> Document ``` ```python theme={null} async def wait_for_document_completion( document_id: str, timeout_seconds: int = 300, check_interval_seconds: int = 2, progress_callback: Optional[Callable[[int, int, str, float], None]] = None, ) -> Document ``` ## Parameters * `document_id` (str): ID of the document to wait for * `timeout_seconds` (int, optional): Maximum time to wait for completion. Defaults to 300. * `check_interval_seconds` (int, optional): Delay between status checks. Defaults to 2. * `progress_callback` (callable, optional): Receives progress updates as `(current_step, total_steps, step_name, percentage)` ## Returns * `Document`: Updated document metadata once processing completes ## Examples ```python theme={null} from morphik import Morphik db = Morphik() doc = db.ingest_text("Sample content") ready = db.wait_for_document_completion(doc.external_id) print(ready.status) ``` ```python theme={null} from morphik import AsyncMorphik async with AsyncMorphik() as db: doc = await db.ingest_text("Sample content") ready = await db.wait_for_document_completion(doc.external_id) print(ready.status) ``` # Installation Source: https://morphik.ai/docs/self-hosting Install Morphik on your own infrastructure For users who need to run Morphik on their own infrastructure, we provide two installation options: Direct Installation and Docker. ## Direct Installation Please ensure that you have Python 3.12 installed on your machine. Guides for installing Python can be found on the [Python website](https://www.python.org/downloads/release/python-3129/). Morphik requires the Rust toolchain for optimized performance operations (binary quantization, base64 encoding, text processing). Install Rust using rustup: ```bash theme={null} curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh ``` After installation, restart your terminal or run: ```bash theme={null} source $HOME/.cargo/env ``` Download and run the installer from [rustup.rs](https://rustup.rs/), or use winget: ```powershell theme={null} winget install Rustlang.Rustup ``` Verify the installation: ```bash theme={null} rustc --version cargo --version ``` Morphik requires PostgreSQL with the pgvector extension for vector storage and similarity search capabilities. Follow the installation instructions for your operating system: On macOS, you can use Homebrew to install PostgreSQL and pgvector: ```bash theme={null} brew install postgresql@14 brew install pgvector ``` Start the PostgreSQL service: ```bash theme={null} brew services start postgresql@14 ``` Create a database and user for Morphik: ```bash theme={null} createdb morphik createuser -s postgres ``` These commands create a database named "morphik" and a superuser named "postgres" that the application will use to connect. Install PostgreSQL from the official repositories. **We recommend version 14.** Other versions may work, but haven't been extensively tested! ```bash theme={null} sudo apt update sudo apt install postgresql postgresql-contrib ``` Install pgvector: ```bash theme={null} sudo apt install postgresql-14-pgvector ``` Start and enable the PostgreSQL service: ```bash theme={null} sudo systemctl start postgresql sudo systemctl enable postgresql ``` Create a database and user for Morphik: ```bash theme={null} sudo -u postgres createdb morphik sudo -u postgres createuser -s postgres ``` 1. Download and install PostgreSQL from the [official website](https://www.postgresql.org/download/windows/). 2. During installation, make note of the password you set for the postgres user. 3. Install pgvector: * Open pgAdmin (installed with PostgreSQL) * Connect to your PostgreSQL server * Right-click on "Extensions" and select "Create > Extension" * Select "pgvector" from the dropdown and click "Save" 4. Create a database for Morphik: * Right-click on "Databases" and select "Create > Database" * Name it "morphik" and click "Save" After installation, verify that PostgreSQL is running correctly: ```bash theme={null} psql -U postgres -c "SELECT version();" ``` You should see the following output: ``` version ------------------------------------------------------------------------------------------------------------------------------ ``` Some system-level dependencies might be required for processing various document types: ```bash theme={null} # Install via Homebrew brew install poppler libmagic ``` ```bash theme={null} # Install via apt sudo apt-get update sudo apt-get install -y poppler-utils libmagic-dev ``` For Windows, you may need to install these dependencies manually: 1. **Poppler**: Download from [poppler for Windows](https://github.com/oschwartz10612/poppler-windows/releases/) 2. **libmagic**: This is included in the python-magic-bin package which will be installed with pip If you encounter database initialization issues within Docker, you may need to manually initialize the schema: ```bash theme={null} psql -U postgres -d morphik -a -f init.sql ``` Docker is used to spin up auxiliary services automatically (e.g., a local Redis container for Morphik's task queue). Install Docker Desktop (macOS/Windows) or the Docker engine (Linux) and make sure the daemon is running. Morphik supports fully local inference for both embeddings and completions through two powerful engines: * **Lemonade SDK** - Windows only, optimized for AMD GPUs/NPUs * **Ollama** - Cross-platform (Windows, macOS, Linux) Both are pre-configured in Morphik. For detailed setup instructions, see our [Local Inference Guide](/local-inference). To get started with Morphik, we need to first setup the server. This involves cloning the [repository](https://github.com/morphik-org/morphik-core/), installing the dependencies, and the running the server. You are just a few steps away from accurate, agentic RAG over your multi-modal data! First, let's clone the repository from GitHub. ```bash theme={null} git clone https://github.com/morphik-org/morphik-core.git ``` After cloning the repository, navigate into the `morphik-core` folder. ```bash theme={null} cd morphik-core ``` Choose the installer for your OS: From the project root, run: ```bash theme={null} # Make the script executable chmod +x install_and_start.sh # Run the installer ./install_and_start.sh ``` From the project root, run in PowerShell: ```powershell theme={null} Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass ./install_and_start.ps1 ``` This script installs dependencies with uv, attempts a prebuilt `llama-cpp-python` wheel, and starts the server. The installers will: 1. Create & activate a `.venv` with **uv** 2. Ask about GPU availability for multimodal embeddings (macOS/Linux) 3. Install `colpali-engine` (for multimodal document understanding) 4. Install or build `llama-cpp-python` (Metal on Apple Silicon, wheel on Windows) 5. Launch the server via `uv run start_server.py` You only need to run `./install_and_start.sh` the first time to set up the environment. For future sessions, activate your project directory and simply start the server with: ```bash theme={null} uv run start_server.py ``` At this point, you may want to customize the server - such as use a different model, enable or disable certain features, etc. - you can do so by editing the `morphik.toml` file. Morphik uses a registered models approach, which allows you to define hundreds of different models in one place and reference them throughout your configuration. This makes it easy to mix and match models based on your needs (e.g., smaller models for simpler tasks). You can find more details about configuration [here](/configuration). The installer copies `.env.example` to `.env` automatically if it's missing. After the script finishes, open `.env` to add any API keys (e.g. `OPENAI_API_KEY`) or secrets you need. You can tweak `morphik.toml` anytime to switch completion/embedding models, adjust chunking, or enable advanced features. You are now ready to launch the Morphik server! Just run the following command to start the server. ```bash theme={null} uv run start_server.py ``` You should see the following output: ``` INFO: Started server process [15169] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://localhost:8000 (Press CTRL+C to quit) ``` This means that the server is running on [http://localhost:8000](http://localhost:8000). You can now interact with the server using the API or the Python SDK. ## Using Docker Morphik provides a streamlined Docker-based setup that includes all necessary components: the core API, PostgreSQL with pgvector, and Redis for task queuing. If you are using an Apple Silicon (M-series) Mac, we highly recommend using the Direct Installation method instead of Docker. GPU passthrough is not supported via Docker on Apple Silicon, which can significantly impact performance. **Multimodal Embeddings and GPU Recommendations** Morphik achieves ultra-accurate document understanding through advanced multimodal embeddings that excel at processing images, PDFs, and complex layouts. While Morphik will work without a GPU, for best results we recommend using a GPU-enabled machine. The installer will ask if you want multimodal embeddings, and can be turned on and off later. To manually adjust multimodal embeddings, edit `morphik.toml`: ```toml theme={null} [morphik] enable_colpali = false # Set to true when GPU is available ``` ### Prerequisites * Docker and Docker Compose V2 installed on your system * At least 10GB of free disk space (for models and data) * 8GB+ RAM recommended ### Quick Start **Option A — Using Pre-built Image (Recommended)** This is the easiest way to get started with Morphik. Run the command for your OS: ```bash theme={null} curl -sSL https://raw.githubusercontent.com/morphik-org/morphik-core/main/install_docker.sh | bash ``` ```powershell theme={null} Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass iwr -useb https://raw.githubusercontent.com/morphik-org/morphik-core/main/install_docker.ps1 -OutFile install_docker.ps1; .\install_docker.ps1 ``` Run the installer from the folder where you want Morphik to live (for example `~/morphik-core`). The script writes `docker-compose.run.yml`, `.env`, `morphik.toml`, and the helper scripts (`start-morphik.*` / `stop-morphik.*`) into the directory where you run the command. *This video demonstrates the installation flow where we select "No" for GPU support and "Yes" for the Admin UI. The initial startup took approximately 3-4 minutes without GPU. Note that with multimodal embeddings enabled, the initial startup may take longer.* The script handles all setup tasks including: * Checking prerequisites (Docker, Docker Compose V2) * Downloading configurations * Prompting for your OpenAI API key (optional - press Enter to skip) * **Setting up authentication** (optional but recommended for production) * **Offering to install the Admin UI** (optional but recommended) * Copying the Admin UI bundle directly from the Morphik image (with a GitHub fallback) when you opt in, so the UI launches even if you haven't cloned the repo * Generating `start-morphik.*` and `stop-morphik.*` helpers so you can restart or clean up services safely later * Starting all services During installation, you'll be prompted: 1. To enter your OpenAI API key (optional - press Enter to skip and configure later) 2. To set a LOCAL\_URI\_PASSWORD for authentication: * **Skip for local-only access**: Press Enter to enable auth bypass (`bypass_auth_mode=true`) * **Set for production/external access**: Enter a secure password to keep authentication enabled (`bypass_auth_mode=false`) 3. Whether you'd like to install the Admin UI (recommended for easier management) Morphik supports 100s of models including OpenAI, Anthropic (Claude), Google Gemini, local models, and even custom models! You can configure your preferred provider in `morphik.toml` after installation. After installation: ```bash theme={null} # To restart Morphik later (auto-detects UI if installed) ./start-morphik.sh # To stop all services and clean up containers/networks/volumes ./stop-morphik.sh # (Runs docker compose down --volumes --remove-orphans with any active profiles) ``` ```powershell theme={null} # To restart Morphik later (auto-detects UI if installed) ./start-morphik.ps1 # To stop all services and clean up containers/networks/volumes ./stop-morphik.ps1 # (Runs docker compose down --volumes --remove-orphans with any active profiles) ``` Use the generated `./stop-morphik.sh` (or `./stop-morphik.ps1` on Windows) whenever you need to shut Morphik down. It automatically includes the UI profile if installed and runs `docker compose down --volumes --remove-orphans`, so no containers or networks linger. After stopping, update `morphik.toml` as needed and restart with `./start-morphik.sh`. The services will be available at: * **API**: [http://localhost:8000](http://localhost:8000) (or your configured port) * **API Documentation**: [http://localhost:8000/docs](http://localhost:8000/docs) * **Admin UI** (if installed): [http://localhost:3003](http://localhost:3003) ### Authentication and Connection URIs Morphik provides flexible authentication options: **Local Development (bypass\_auth\_mode = true)** * When `bypass_auth_mode = true`, authentication is disabled for local API access * Useful for local development and testing * **Not secure for production or external access** **Authenticated Mode (bypass\_auth\_mode = false)** * Authentication required for all API requests * Set `JWT_SECRET_KEY` and `LOCAL_URI_PASSWORD` in your environment * To generate authorized connection URIs: Navigate to [http://localhost:8000/docs](http://localhost:8000/docs) in your browser Generate Local URI endpoint in Swagger UI Scroll down to find the `/local/generate_uri` endpoint in the API docs 1. Click on the endpoint to expand it 2. Click "Try it out" 3. Fill in the request parameters: * **name**: A descriptive name for this connection (e.g., "admin") * **expiry\_days**: How long the token should remain valid (default: 30) * **password\_token**: Your LOCAL\_URI\_PASSWORD from your environment * **server\_mode**: * Set to `true` if you want to access Morphik from outside the server * Set to `false` for local access only (uses localhost) 4. Click "Execute" The response will contain a secure connection URI that includes authentication tokens for your client applications. The generated URI contains authentication tokens and should be kept secure. Anyone with this URI can access your Morphik instance with the permissions embedded in the token. ### Using Your Connection URI Once you have generated a connection URI, you can use it to connect to Morphik through: **Option 1: Admin UI** If you have the Admin UI installed, you can paste your connection URI directly in the UI: Paste connection URI in Admin UI The URI field is highlighted in red, showing where to paste your `morphik://` connection string. **Option 2: Python SDK** ```python theme={null} from morphik import Morphik # Initialize Morphik with your connection URI morphik = Morphik("morphik://admin:eyJhbGc...") # Use your generated URI # Ingest a file doc = morphik.ingest_file(file_path="document.pdf") doc.wait_for_completion() # Query the ingested content response = morphik.query("What is Morphik?") print(response) # "Morphik is the most accurate end-to-end RAG system" ``` **Option 3: TypeScript/JavaScript SDK** ```typescript theme={null} import Morphik from 'morphik'; import * as fs from 'fs'; // Extract the token from your URI // URI format: morphik://name:token@host const uri = "morphik://admin:eyJhbGc..."; // Your generated URI const token = uri.split(':')[1].split('@')[0]; // Initialize Morphik client const morphik = new Morphik({ apiKey: token, baseURL: 'http://localhost:8000' // Adjust based on your server_mode }); // Ingest a file const file = fs.createReadStream('document.pdf'); const doc = await morphik.ingest.ingestFile({ file }); // Wait for processing await new Promise(resolve => setTimeout(resolve, 5000)); // Query the content const response = await morphik.query.generateCompletion({ query: 'What is Morphik?' }); console.log(response.completion); // "Morphik is the most accurate end-to-end RAG system" ``` **What is Morphik?** Morphik is the most accurate end-to-end RAG (Retrieval-Augmented Generation) system, designed to understand and process multi-modal documents with exceptional precision. *** **Option B — Local Development** For developing and testing changes to Morphik: ```bash theme={null} # Clone and enter the repo git clone https://github.com/morphik-org/morphik-core.git cd morphik-core # Start development environment ./start-dev.sh ``` This builds the Docker image locally and starts all services. To include local inference models, see our [Local Inference Guide](/local-inference) for setting up Ollama or Lemonade SDK. *** ### Changing the Port To run Morphik on a different port, edit the `morphik.toml` file: ```toml theme={null} [api] port = 9000 # Your custom port ``` Then restart using the appropriate start script (`./start-morphik.sh` for production or `./start-dev.sh` for development). ### Advanced Configuration • **Change models** – Edit `morphik.toml` to switch between different models. See the `[registered_models]` section for available options. • **Environment variables** – Customize settings in `.env` file (API keys, database connection, etc.) • **Persist data** – Docker volumes automatically persist PostgreSQL data and uploaded files across restarts. *** ### Configuration The default configuration works out of the box and includes: * PostgreSQL with pgvector for document storage * Redis for task queuing * Local file storage * Basic authentication * Optional: Local inference with [Ollama or Lemonade](/local-inference) You can customize your setup by creating a `.env` file: ```bash theme={null} JWT_SECRET_KEY=your-secure-key-here # Important: Change in production OPENAI_API_KEY=sk-... # Only if using OpenAI HOST=0.0.0.0 # Leave as is for Docker PORT=8000 # Change if needed ``` ### Accessing Services * Morphik API: [http://localhost:8000](http://localhost:8000) * API Documentation: [http://localhost:8000/docs](http://localhost:8000/docs) * Health Check: [http://localhost:8000/health](http://localhost:8000/health) ### Troubleshooting 1. **Service Won't Start** ```bash theme={null} # View all logs docker compose logs # View specific service logs docker compose logs morphik docker compose logs postgres docker compose logs redis ``` 2. **Database Issues** * Check PostgreSQL is healthy: `docker compose ps` * Verify database connection: `docker compose exec postgres psql -U morphik -d morphik` 3. **Local Model Issues** * For local inference setup and troubleshooting, see our [Local Inference Guide](/local-inference) * Verify that your Redis configuration in `morphik.toml` matches your deployment: * For Redis in Docker, use `host = "redis"` (not "localhost") 4. **Memory Issues with Local Models** * If using local models and encountering memory issues: * Increase Docker memory allocation in Docker Desktop (Settings > Resources) * Use smaller quantized models * See our [Local Inference Guide](/local-inference) for model recommendations * Alternatively, switch to cloud providers (OpenAI, Anthropic, etc.) in `morphik.toml` 5. **Performance Issues** * Monitor resources: `docker stats` * Ensure sufficient RAM (8GB+ recommended) * Check disk space: `df -h` # Special Thanks Source: https://morphik.ai/docs/special-thanks Thanking our amazing users for pointing out bugs, requesting features, and being early adopters ❤️ **Special thanks to rex777 from the Morphik Discord community for help with troubleshooting and improving these setup instructions!**\ \ [**Special thanks to Permafacture from Morphik's GitHub Issues for pointing out versioning and initialization issues with our PostgreSQL setup instructions!**](https://github.com/Permafacture) **Special thanks to blackmaria from the Morphik Discord community for help with the PGVector setup instructions for Linux!** **Special thanks to The Dr1ver from the Morphik Discord community for help with the Node installation instructions for Linux!** [**Special thanks to 7hunderbird from GitHub for helping us update our documentation!**](https://github.com/7hunderbird) **Special thanks to Rohan from Cleon (**[**https://www.getcleon.ai/**](https://www.getcleon.ai/)**) for catching some docker setup bugs.** **Special thanks to Joey from the LLM Data Company (**[**https://thellmdatacompany.com/**](https://thellmdatacompany.com/)**) for running evals on Morphik.** # Code Source: https://morphik.ai/docs/using-morphik/code Learn how to use the Morphik Python SDK or REST API Use the Morphik Python SDK to interact with the server. Use the Morphik REST API to interact with the server. # Model Context Protocol (MCP) Source: https://morphik.ai/docs/using-morphik/mcp Enable Claude and other AI assistants to access your Morphik knowledge base