FutureSmart Developer Docs
Open Document Intelligence

End-to-end examples

Complete programs you can copy and run — ingest a folder and export the extracted table, and a backend endpoint that answers with sources.

Each example is a whole program. Set DOCUMENT_INTELLIGENCE_API_KEY and DOCUMENT_INTELLIGENCE_COLLECTION_ID (see the Quickstart) and run it as is.

Ingest a folder, then export a spreadsheet

Upload every supported file in a folder, wait until each is read, and write the extracted fields of the whole collection to documents.csv.

Python
"""ingest_folder.py: python ingest_folder.py ./invoices"""
import csv
import os
import sys
import time
from pathlib import Path

import requests

API_ROOT = "https://futuresmart.ai/document-intelligence/api"
COLLECTION_ID = os.environ["DOCUMENT_INTELLIGENCE_COLLECTION_ID"]
HEADERS = {"x-api-key": os.environ["DOCUMENT_INTELLIGENCE_API_KEY"]}
TYPES = {".pdf": "application/pdf", ".docx": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
         ".txt": "text/plain", ".md": "text/markdown"}
DONE = {"completed", "failed", "invalid"}

folder = Path(sys.argv[1] if len(sys.argv) > 1 else ".")
pending = {}
for path in sorted(folder.iterdir()):
    if path.suffix.lower() not in TYPES:
        continue
    if path.stat().st_size > 5 * 1024 * 1024:
        print(f"skip {path.name}: over 5 MB, use an upload URL instead")
        continue
    with path.open("rb") as f:
        r = requests.post(
            f"{API_ROOT}/v1/{COLLECTION_ID}/documents",
            headers=HEADERS,
            files={"file": (path.name, f, TYPES[path.suffix.lower()])},
            timeout=120,
        )
    if r.ok:
        pending[r.json()["object_id"]] = path.name
        print(f"added {path.name}")
    else:
        print(f"refused {path.name}: {r.status_code} {r.json().get('detail')}")

deadline = time.time() + 900
while pending and time.time() < deadline:
    time.sleep(5)
    for object_id, name in list(pending.items()):
        status = requests.get(
            f"{API_ROOT}/v1/{COLLECTION_ID}/documents/{object_id}/status", headers=HEADERS, timeout=60
        ).json()
        if status["status"] in DONE:
            print(f"{name}: {status['status']} {status.get('error_detail') or ''}")
            del pending[object_id]

rows, columns, page = [], None, 1
while True:
    table = requests.post(
        f"{API_ROOT}/v2/{COLLECTION_ID}/documents/metadata/all",
        headers=HEADERS,
        json={"page": page, "page_size": 100},
        timeout=60,
    ).json()
    columns = columns or [c["name"] for c in table["columns"]]
    rows += table["rows"]
    if page >= table["total_pages"]:
        break
    page += 1

with open("documents.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=columns, extrasaction="ignore")
    writer.writeheader()
    writer.writerows(rows)
print(f"documents.csv: {len(rows)} rows, {len(columns)} columns")

A backend endpoint that answers with sources

Your frontend sends a question; the server adds the key, asks the collection, and returns the answer with the documents it came from. Node.js 18 or later, no dependencies.

JavaScript
// server.mjs: POST {"question": "...", "session_id": "..."} to http://localhost:3000/ask
import { createServer } from "node:http";

const API_ROOT = "https://futuresmart.ai/document-intelligence/api";
const COLLECTION_ID = process.env.DOCUMENT_INTELLIGENCE_COLLECTION_ID;
const API_KEY = process.env.DOCUMENT_INTELLIGENCE_API_KEY;

async function readJson(req) {
  let body = "";
  for await (const chunk of req) body += chunk;
  return JSON.parse(body || "{}");
}

createServer(async (req, res) => {
  if (req.method !== "POST" || req.url !== "/ask") {
    res.writeHead(404).end();
    return;
  }
  try {
    const { question, session_id } = await readJson(req);
    if (!question) {
      res.writeHead(400, { "Content-Type": "application/json" }).end(JSON.stringify({ error: "question is required" }));
      return;
    }
    const upstream = await fetch(`${API_ROOT}/v1/${COLLECTION_ID}/chat`, {
      method: "POST",
      headers: { "x-api-key": API_KEY, "Content-Type": "application/json" },
      body: JSON.stringify({ query: question, session_id, stream: false }),
    });
    const data = await upstream.json();
    const body = upstream.ok
      ? {
          answer: data.response,
          session_id: data.session_id,
          sources: (data.data?.knowledge?.relevant_sources?.data ?? []).map((s) => ({ id: s.id, title: s.title })),
        }
      : { error: typeof data.detail === "string" ? data.detail : "The collection could not answer" };
    res.writeHead(upstream.ok ? 200 : upstream.status, { "Content-Type": "application/json" }).end(JSON.stringify(body));
  } catch (err) {
    res.writeHead(500, { "Content-Type": "application/json" }).end(JSON.stringify({ error: String(err) }));
  }
}).listen(3000, () => console.log("Listening on http://localhost:3000/ask"));

Start it with node server.mjs, then:

Shell
curl -sS -X POST http://localhost:3000/ask \
  -H "Content-Type: application/json" \
  -d '{"question": "Which invoices are due in October?"}'

Put your own sign-in in front of /ask before deploying it: every question counts toward the collection owner's limits.

Search results as context for your own model

If you run your own language model, use search to fetch the passages and put them in your prompt. This builds the context block; send it to whichever model you use.

Python
import os

import requests

API_ROOT = "https://futuresmart.ai/document-intelligence/api"
COLLECTION_ID = os.environ["DOCUMENT_INTELLIGENCE_COLLECTION_ID"]

question = "What happens if an invoice is paid late?"
chunks = requests.post(
    f"{API_ROOT}/v1/{COLLECTION_ID}/search",
    headers={"x-api-key": os.environ["DOCUMENT_INTELLIGENCE_API_KEY"]},
    json={"query": question, "num_chunks": 5},
    timeout=60,
).json()["chunks"]

context = "\n\n".join(f"[{i + 1}] ({c['file_name']})\n{c['chunk_text']}" for i, c in enumerate(chunks))
prompt = f"Answer from the numbered sources and cite them.\n\n{context}\n\nQuestion: {question}"
print(prompt[:600])