api reference for screenpipe

below is the detailed api reference for screenpipe's core functionality.

search api

endpoint: /search
method: get
description: searches captured data (ocr, audio transcriptions, etc.) stored in screenpipe's local database.

query parameters:

q (string, optional): search term (a SINGLE word)
content_type (enum): type of content to search:
- ocr: optical character recognition text
- audio: audio transcriptions
- ui: user interface elements
limit (int): max results per page (default: 20)
offset (int): pagination offset
start_time (timestamp, optional): filter by start timestamp
end_time (timestamp, optional): filter by end timestamp
app_name (string, optional): filter by application name
window_name (string, optional): filter by window name
include_frames (bool, optional): include base64 encoded frames
min_length (int, optional): minimum content length
max_length (int, optional): maximum content length
speaker_ids (int[], optional): filter by specific speaker ids

sample requests:

# Basic search
curl "http://localhost:3030/search?q=meeting&content_type=ocr&limit=10"
 
# Audio search with speaker filter
curl "http://localhost:3030/search?content_type=audio&speaker_ids=1,2"
 
# UI elements search
curl "http://localhost:3030/search?content_type=ui&app_name=chrome"

sample response:

{
  "data": [
    {
      "type": "OCR",
      "content": {
        "frame_id": 123,
        "text": "meeting notes",
        "timestamp": "2024-03-10T12:00:00Z",
        "file_path": "/frames/frame123.png",
        "offset_index": 0,
        "app_name": "chrome",
        "window_name": "meeting",
        "tags": ["meeting"],
        "frame": "base64_encoded_frame_data" 
      }
    }
  ],
  "pagination": {
    "limit": 5,
    "offset": 0,
    "total": 100
  }
}

audio devices api

endpoint: /audio/list
method: get
description: lists available audio input/output devices

sample response:

[
  {
    "name": "built-in microphone",
    "is_default": true
  }
]

monitors api

endpoint: /vision/list
method: post
description: lists available monitors/displays

sample response:

[
  {
    "id": 1,
    "name": "built-in display",
    "width": 2560,
    "height": 1600,
    "is_default": true
  }
]

tags api

endpoint: /tags/:content_type/:id
methods: post (add), delete (remove)
description: manage tags for content items
content_type: vision or audio

add tags request:

{
  "tags": ["important", "meeting"]
}

sample response:

{
  "success": true
}

pipes api

list pipes

endpoint: /pipes/list
method: get

download pipe

endpoint: /pipes/download
method: post

{
  "url": "https://github.com/user/repo/pipe-example"
}

enable pipe

endpoint: /pipes/enable
method: post

{
  "pipe_id": "pipe-example"
}

disable pipe

endpoint: /pipes/disable
method: post

{
  "pipe_id": "pipe-example" 
}

update pipe config

endpoint: /pipes/update
method: post

{
  "pipe_id": "pipe-example",
  "config": {
    "key": "value"
  }
}

speakers api

list unnamed speakers

endpoint: /speakers/unnamed
method: get
description: get list of speakers without names assigned

query parameters:

limit (int): max results
offset (int): pagination offset
speaker_ids (int[], optional): filter specific speaker ids

sample request:

curl "http://localhost:3030/speakers/unnamed?limit=10&offset=0"

search speakers

endpoint: /speakers/search
method: get
description: search speakers by name

query parameters:

name (string, optional): name prefix to search for

sample request:

curl "http://localhost:3030/speakers/search?name=john"

update speaker

endpoint: /speakers/update
method: post
description: update speaker name or metadata

request body:

{
  "id": 123,
  "name": "john doe",
  "metadata": "{\"role\": \"engineer\"}"
}

delete speaker

endpoint: /speakers/delete
method: post
description: delete a speaker and associated audio chunks

request body:

{
  "id": 123
}

get similar speakers

endpoint: /speakers/similar
method: get
description: find speakers with similar voice patterns

query parameters:

speaker_id (int): reference speaker id
limit (int): max results

sample request:

curl "http://localhost:3030/speakers/similar?speaker_id=123&limit=5"

merge speakers

endpoint: /speakers/merge
method: post
description: merge two speakers into one

request body:

{
  "speaker_to_keep_id": 123,
  "speaker_to_merge_id": 456
}

mark as hallucination

endpoint: /speakers/hallucination
method: post
description: mark a speaker as incorrectly identified

request body:

{
  "speaker_id": 123
}

health api

endpoint: /health
method: get
description: system health status

sample response:

{
  "status": "healthy",
  "last_frame_timestamp": "2024-03-10T12:00:00Z", 
  "last_audio_timestamp": "2024-03-10T12:00:00Z",
  "last_ui_timestamp": "2024-03-10T12:00:00Z",
  "frame_status": "ok",
  "audio_status": "ok",
  "ui_status": "ok",
  "message": "all systems functioning normally"
}

stream frames api

endpoint: /stream/frames
method: get
description: stream frames as server-sent events (sse)

query parameters:

start_time (timestamp): start time for frame stream
end_time (timestamp): end time for frame stream

sample request:

curl "http://localhost:3030/stream/frames?start_time=2024-03-10T12:00:00Z&end_time=2024-03-10T13:00:00Z"

sample event data:

{
  "timestamp": "2024-03-10T12:00:00Z",
  "devices": [
    {
      "device_id": "screen-1",
      "frame": "base64_encoded_frame_data"
    }
  ]
}

architecture overview contributing