Feature/merge dev and main (#968)
* Feature/merge graphrag group mgmt (#876) * add group ids to document abstraction, first steps * extend group permissions * up * add tests for new group features * up * fixup auth * onboard extensive regression tests * adding regression tests * finish tests * rm selenium * test observability * uncomment tests * checkin first set of group tests * modify search, passing vector tests * checkin work * full delete logic * update search to use new filters * check in * Clean up * Check in * add search * tests/test_end_to_end.py::test_ingest_txt_document passing * cleanup logging * make schemas explicit * move to run logger abstraction * cleanup some test workflows * revive tests * tweak to pass tests * tweak rrf * finish hybrid search cleanup * fixup on regr tests, regen payloads * refresh payloads * refactor api model * Feature/refactor api model (#868) * cleanup imports * flake and cleanup * coherent global import / export structure * add ingestion response models * add management response models * cleanups * checkin work on routes * remove request models * last fixes * merge * add user / group gating * working test groups * updating client --------- Co-authored-by: NolanTrem <34580718+NolanTrem@users.noreply.github.com> * Clean up API (#878) * Get running * fixes in sdk * Add in more fixes * Feature/merge dev owen changes (#880) * add group ids to document abstraction, first steps * extend group permissions * up * add tests for new group features * up * fixup auth * onboard extensive regression tests * adding regression tests * finish tests * rm selenium * test observability * uncomment tests * checkin first set of group tests * modify search, passing vector tests * checkin work * full delete logic * update search to use new filters * check in * Clean up * Check in * add search * tests/test_end_to_end.py::test_ingest_txt_document passing * cleanup logging * make schemas explicit * move to run logger abstraction * cleanup some test workflows * revive tests * tweak to pass tests * tweak rrf * finish hybrid search cleanup * fixup on regr tests, regen payloads * refresh payloads * refactor api model * Feature/refactor api model (#868) * cleanup imports * flake and cleanup * coherent global import / export structure * add ingestion response models * add management response models * cleanups * checkin work on routes * remove request models * last fixes * merge * add user / group gating * working test groups * updating client * rename service to restructure * add get documents for group endpoint * fix client bugs * return delete format * merge cleanups * merge * finalize --------- Co-authored-by: NolanTrem <34580718+NolanTrem@users.noreply.github.com> * Shreyas/graphrag test (#881) * add group ids to document abstraction, first steps * extend group permissions * up * add tests for new group features * up * fixup auth * onboard extensive regression tests * adding regression tests * finish tests * rm selenium * test observability * uncomment tests * checkin first set of group tests * modify search, passing vector tests * checkin work * full delete logic * update search to use new filters * check in * Clean up * Check in * add search * tests/test_end_to_end.py::test_ingest_txt_document passing * cleanup logging * make schemas explicit * move to run logger abstraction * cleanup some test workflows * revive tests * tweak to pass tests * tweak rrf * finish hybrid search cleanup * fixup on regr tests, regen payloads * refresh payloads * refactor api model * Feature/refactor api model (#868) * cleanup imports * flake and cleanup * coherent global import / export structure * add ingestion response models * add management response models * cleanups * checkin work on routes * remove request models * last fixes * merge * add user / group gating * sync * enrich * up * fix global search * rag * remove client.py * rm configs * rm configs --------- Co-authored-by: emrgnt-cmplxty <owen@algofi.org> Co-authored-by: NolanTrem <34580718+NolanTrem@users.noreply.github.com> Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com> * Feature/fix embedding pipe (#882) * up * fixup concurrency * fix ollama embeddings * fix batching with ollama * checkin all cleanups * rm kg cruft (#884) * rm kg cruft * tweaks * tweak 2 (#885) * Feature/fix retrieval endpoint cruft (#887) * tweak 2 * fix retrieval endpoint descriptions * Python SDK (#886) Clean up Python SDK and routes * Separate out SDK, add js and go sdk to monorepo (#888) * Add r2r-js sdk * Add go sdk * Pull out python sdk * remove venv * Update packages * Check in fixes * Remove alembic dependencies * Feature/merge w nolan (#894) * cleanup hybrid search * cleanups in * Fix structure * Make graspologic optional * fix rag stream (#895) * add py r2r (#896) * Clean up (#897) * fix agent (#898) * define `RAGAgentResponse` (#899) * Shreyas/unstructured (#900) * api + oss lib * rm pdb * rm poetry lock * update version * fixes * Feature/cleanup client obj logic (#901) * define `RAGAgentResponse` * cleanup client logic * Shreyas/tests (#889) * init * tests * rename service * api model * add * merge * rm restructure router * print descriptions * Refactor CLI (#903) * Rm files readded by git (#904) * Remove Execution Wrapper (#905) * Rm files readded by git * Fix merge botch * Feature/fix auth revive tests rebased (#906) * adding the client touch ups * fix auth, revive tests * add back tests * uncomment run auth workflow * decruft * refresh test kg * fixup toml (#908) * Feature/fix ingestion update (#909) * fixup toml * fix update * Fix CLI Tests (#912) Fix CLI tests * Shreyas/kg runtime cfg (#913) add kg runtime config * rename kgenrichmentresponse (#914) * Feature/add nltk hybrid expansion rebased (#917) * expand hybrid search with nltk * cleanups * cleanup hybrid search * format * add setup.py * update * add script (#918) * Fix bug in document chunks (#921) * Fix bug in update files (#923) * Shreyas/unstructured (#922) * fix dockerfiles * adding config * fix paths * mv unstructured dep to docker * clean * Update docker_utils.py * Update unstructured_parsing.py * Update r2r_chunking.py * Update app_entry.py * Feature/repair logging (#925) * fixing logs * fix * rm double logging (#929) * Configs (#926) * Fix config logic * Update config * Clean up cli entry point * Disable SSL when installing nltk wordnet (#930) * Fix analytics endpoint * Update OpenAI sdk calls (#933) * Feature/revive advanced rag (#932) * rm double logging * revive advanced rag examples * merge (#934) * sync model (#935) * Feature/remove version from ingestion end pt (#936) * sync model * remove ability to set version * tweak versions impl * fix version bug * Move docker (#938) * Move docker * remove from root * Clean up sdk/restructure.py * Fix js tests, completion scoring (#939) * Shreyas/unstructured docker image (#940) unstructured docker image * Update JS (#941) * Update models (#942) * Feature/complete group logic (#945) * fix group logic * up * Fix Dockerbuild, Symlink Readme (#944) * Add back tast prompt override and include title if availible * Fix docker, sym link readme * Fix compose file path * Shreyas/KG Search Result model (#937) * return type to kg_search_result * add model * local and global results * modify config * refresh should not be gated by auth (#946) * Linting sync (#947) * Remove email from refresh (#948) * Fix link to image * Feature/rm print cruft rebase (#953) * refresh should not be gated by auth * rm print cruft * black and sort * merge * rm * update api return type * Update Actions (#954) * Update Github Actions (#956) * Update Actions * Update actions * Shreyas/kgsearchresult model (#957) * return type to kg_search_result * add model * local and global results * modify config * add models * up * fix config path * fix models * Login and refresh token bug (#959) * Update Actions * Fix bug in login with refresh token * Point pytest to linux (#960) * collection docs (#955) * Feature/merge dev to main (#962) * merge dev and main * git rm * add back collection fix * fix docker builds (#963) * Running unstructured docker + code cleanups (#964) * Small bugfixes on prompts, return types (#965) * Fix failing CLI tests * NPM publish action * remove tarball * Feature/fix dev tests (#966) * update auth tests * fix tests * back and sort * decruft * revert back to gpt-4o --------- Co-authored-by: NolanTrem <34580718+NolanTrem@users.noreply.github.com> Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>
This commit is contained in:
emrgnt-cmplxty
@@ -34,8 +34,8 @@ jobs:
|
||||
- name: Docker Auth
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.DOCKER_USERNAME }}
|
||||
password: ${{ secrets.DOCKER_TOKEN }}
|
||||
username: ${{ secrets.RAGTORICHES_DOCKER_UNAME }}
|
||||
password: ${{ secrets.RAGTORICHES_DOCKER_TOKEN }}
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
@@ -57,7 +57,7 @@ jobs:
|
||||
uses: docker/metadata-action@v5
|
||||
with:
|
||||
images: |
|
||||
emrgntcmplxty/r2r
|
||||
ragtoriches/prod
|
||||
us-east1-docker.pkg.dev/alert-rush-397022/sciphi-r2r/r2r
|
||||
tags: |
|
||||
type=raw,value=${{ steps.version.outputs.RELEASE_VERSION }}
|
||||
@@ -67,17 +67,18 @@ jobs:
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: .
|
||||
file: ./Dockerfile
|
||||
file: ./py/Dockerfile
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
platforms: linux/amd64,linux/arm64
|
||||
|
||||
- name: Trigger Dependent Repositories
|
||||
if: success()
|
||||
run: |
|
||||
curl -X POST \
|
||||
-H "Accept: application/vnd.github.v3+json" \
|
||||
-H "Authorization: token ${{ secrets.PERSONAL_ACCESS_TOKEN }}" \
|
||||
https://api.github.com/repos/SciPhi-AI/R2R-basic-rag-template/dispatches \
|
||||
-d '{"event_type": "rebuild", "client_payload": {"r2r_tag": "${{ steps.version.outputs.RELEASE_VERSION }}"}}'
|
||||
- name: Build and Push Docker Image
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: .
|
||||
file: ./py/Dockerfile.unstructured
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}-unstructured
|
||||
labels: ${{ steps.meta.outputs.labels }}-unstructured
|
||||
platforms: linux/amd64,linux/arm64
|
||||
|
||||
@@ -1,89 +0,0 @@
|
||||
name: Build and Publish ragtoriches/prod
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
version:
|
||||
description: 'Version to publish (leave empty to use default versioning)'
|
||||
required: false
|
||||
type: string
|
||||
|
||||
jobs:
|
||||
build-and-publish:
|
||||
runs-on: ubuntu-latest
|
||||
permissions:
|
||||
packages: write
|
||||
contents: read
|
||||
actions: write
|
||||
steps:
|
||||
- name: Checkout Repository
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: Docker Auth
|
||||
uses: docker/login-action@v3
|
||||
with:
|
||||
username: ${{ secrets.RAGTORICHES_DOCKER_UNAME }}
|
||||
password: ${{ secrets.RAGTORICHES_DOCKER_TOKEN }}
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v3
|
||||
|
||||
- name: Set up QEMU
|
||||
uses: docker/setup-qemu-action@v3
|
||||
|
||||
- name: Determine version to use
|
||||
id: version
|
||||
run: |
|
||||
if [ -n "${{ github.event.inputs.version }}" ]; then
|
||||
echo "RELEASE_VERSION=${{ github.event.inputs.version }}" >> $GITHUB_OUTPUT
|
||||
else
|
||||
echo "RELEASE_VERSION=main" >> $GITHUB_OUTPUT
|
||||
fi
|
||||
|
||||
- name: Generate Cache Buster
|
||||
id: cache-buster
|
||||
run: echo "CACHE_BUSTER=$(date +%s)" >> $GITHUB_OUTPUT
|
||||
|
||||
- name: Extract metadata (tags, labels) for Docker
|
||||
id: meta
|
||||
uses: docker/metadata-action@v5
|
||||
with:
|
||||
images: ragtoriches/prod
|
||||
tags: |
|
||||
type=raw,value=${{ steps.version.outputs.RELEASE_VERSION }}
|
||||
type=raw,value=latest
|
||||
|
||||
- name: Build and Push Docker Image (Standard)
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: .
|
||||
file: ./Dockerfile
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
platforms: linux/amd64,linux/arm64
|
||||
build-args: |
|
||||
CACHE_BUST=${{ steps.cache-buster.outputs.CACHE_BUSTER }}
|
||||
|
||||
- name: Build and Push Docker Image (Unstructured)
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: .
|
||||
file: ./Dockerfile.unstructured
|
||||
push: true
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
platforms: linux/amd64,linux/arm64
|
||||
build-args: |
|
||||
CACHE_BUST=${{ steps.cache-buster.outputs.CACHE_BUSTER }}
|
||||
|
||||
- name: Modify and Push Unstructured Tags
|
||||
run: |
|
||||
for tag in ${{ steps.meta.outputs.tags }}; do
|
||||
IFS=':' read -ra PARTS <<< "$tag"
|
||||
REPO="${PARTS[0]}"
|
||||
VERSION="${PARTS[1]}"
|
||||
NEW_TAG="${REPO}:${VERSION}-unstructured"
|
||||
docker tag $tag $NEW_TAG
|
||||
docker push $NEW_TAG
|
||||
done
|
||||
@@ -1,43 +0,0 @@
|
||||
name: Links
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
pull_request:
|
||||
branches:
|
||||
- main
|
||||
schedule:
|
||||
- cron: "00 18 * * *"
|
||||
|
||||
jobs:
|
||||
linkChecker:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v2
|
||||
|
||||
- name: Link Checker
|
||||
id: lychee
|
||||
uses: lycheeverse/lychee-action@v1
|
||||
continue-on-error: true
|
||||
with:
|
||||
args: "--output ./docs/lychee/out.md --config ./docs/.lycheerc"
|
||||
|
||||
- name: Get Lychee Exit Code
|
||||
id: get-exit-code
|
||||
run: echo "lychee_exit_code=$?" >> $GITHUB_ENV
|
||||
|
||||
- name: Create Custom Report
|
||||
if: failure()
|
||||
run: |
|
||||
echo "Lychee doesn't play nice with relative imports for .mdx files. Add a link to the .lycheeignore if needed!" > ./docs/lychee/custom_report.md
|
||||
echo "" >> ./docs/lychee/custom_report.md
|
||||
cat ./docs/lychee/out.md >> ./docs/lychee/custom_report.md
|
||||
|
||||
- name: Create Issue From File
|
||||
if: failure()
|
||||
uses: peter-evans/create-issue-from-file@v4
|
||||
with:
|
||||
title: Link Checker Report
|
||||
content-filepath: ./docs/lychee/custom_report.md
|
||||
labels: report, automated issue
|
||||
@@ -52,7 +52,7 @@ jobs:
|
||||
uses: docker/build-push-action@v5
|
||||
with:
|
||||
context: .
|
||||
file: ./Dockerfile
|
||||
file: ./py/Dockerfile
|
||||
push: true
|
||||
tags: ragtoriches/dev:latest
|
||||
platforms: linux/amd64
|
||||
@@ -60,6 +60,7 @@ jobs:
|
||||
pull: true
|
||||
|
||||
- name: Run cloud LLM integration tests in Docker
|
||||
working-directory: ./py
|
||||
run: |
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
@@ -90,13 +91,13 @@ jobs:
|
||||
r2r search --query="What was Uber'\''s profit in 2020?"
|
||||
|
||||
echo "Hybrid Search"
|
||||
r2r search --query="What is a fierce nerd?" --do-hybrid-search
|
||||
r2r search --query="What is a fierce nerd?" --use-hybrid-search
|
||||
|
||||
echo "Basic RAG"
|
||||
r2r rag --query="What was Uber'\''s profit in 2020?"
|
||||
|
||||
echo "RAG with Hybrid Search"
|
||||
r2r rag --query="Who is John Snow?" --do-hybrid-search
|
||||
r2r rag --query="Who is John Snow?" --use-hybrid-search
|
||||
|
||||
echo "Streaming RAG"
|
||||
r2r rag --query="What was Lyft'\''s profit in 2020?" --stream
|
||||
|
||||
@@ -1,122 +0,0 @@
|
||||
name: Windows R2R Docker Build and Integration Test (Windows Server 2022 Datacenter x64)
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
build-and-test:
|
||||
runs-on: [self-hosted, Windows]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
|
||||
- name: Set up Python virtual environment
|
||||
run: |
|
||||
python -m venv %USERPROFILE%\r2r_venv
|
||||
echo %USERPROFILE%\r2r_venv\Scripts >> %GITHUB_PATH%
|
||||
shell: cmd
|
||||
|
||||
- name: Activate virtual environment and install R2R
|
||||
run: |
|
||||
call %USERPROFILE%\r2r_venv\Scripts\activate.bat
|
||||
pip install --upgrade pip
|
||||
pip install --upgrade r2r
|
||||
shell: cmd
|
||||
|
||||
- name: Build R2R Docker image
|
||||
run: |
|
||||
docker build -t emrgntcmplxty/r2r:${{ github.sha }} .
|
||||
shell: cmd
|
||||
|
||||
- name: Build R2R Docker image with args
|
||||
run: |
|
||||
docker build ^
|
||||
--build-arg GITHUB_SHA=${{ github.sha }} ^
|
||||
--build-arg GITHUB_REF=${{ github.ref }} ^
|
||||
-t emrgntcmplxty/r2r:${{ github.sha }} .
|
||||
shell: cmd
|
||||
|
||||
- name: Pull pgvector image
|
||||
run: docker pull pgvector/pgvector:pg16
|
||||
shell: cmd
|
||||
|
||||
- name: Create Docker network
|
||||
run: docker network create r2r-network
|
||||
shell: cmd
|
||||
|
||||
- name: Start Postgres
|
||||
run: |
|
||||
docker run -d --name postgres --network r2r-network ^
|
||||
-e POSTGRES_USER=postgres ^
|
||||
-e POSTGRES_PASSWORD=postgres ^
|
||||
-e POSTGRES_DB=postgres ^
|
||||
pgvector/pgvector:pg16
|
||||
shell: cmd
|
||||
|
||||
- name: Start R2R
|
||||
run: |
|
||||
docker run -d --name r2r --network r2r-network ^
|
||||
-e PYTHONUNBUFFERED=1 ^
|
||||
-e OPENAI_API_KEY=${{ secrets.OPENAI_API_KEY }} ^
|
||||
-e POSTGRES_USER=postgres ^
|
||||
-e POSTGRES_PASSWORD=postgres ^
|
||||
-e POSTGRES_HOST=postgres ^
|
||||
-e POSTGRES_PORT=5432 ^
|
||||
-e POSTGRES_DBNAME=postgres ^
|
||||
-e POSTGRES_VECS_COLLECTION=vecs ^
|
||||
-p 8000:8000 ^
|
||||
emrgntcmplxty/r2r:${{ github.sha }}
|
||||
|
||||
timeout /t 10
|
||||
shell: cmd
|
||||
|
||||
- name: Run integration tests
|
||||
run: |
|
||||
echo R2R Version
|
||||
r2r version
|
||||
|
||||
echo Ingest Data
|
||||
r2r ingest-sample-files
|
||||
|
||||
echo Get Documents Overview
|
||||
r2r documents-overview
|
||||
|
||||
echo Get Document Chunks
|
||||
r2r document-chunks 93123a68-d668-51de-8291-92162730dc87
|
||||
|
||||
echo Delete Documents
|
||||
r2r delete --keys=document_id --values=b736292c-11e6-5453-9686-055da3edb866
|
||||
|
||||
echo Vector Search
|
||||
r2r search --query="What was Uber's profit in 2020?"
|
||||
|
||||
echo Hybrid Search
|
||||
r2r search --query="What is a fierce nerd?" --do-hybrid-search
|
||||
|
||||
echo Basic RAG
|
||||
r2r rag --query="What was Uber's profit in 2020?"
|
||||
|
||||
echo RAG with Hybrid Search
|
||||
r2r rag --query="Who is John Snow?" --do-hybrid-search
|
||||
|
||||
echo Streaming RAG
|
||||
r2r rag --query="What was Lyft's profit in 2020?" --stream
|
||||
|
||||
echo Users Overview
|
||||
r2r users-overview
|
||||
|
||||
echo Logging
|
||||
r2r logs
|
||||
|
||||
echo Analytics
|
||||
r2r analytics --filters "{\"search_latencies\": \"search_latency\"}" --analysis-types "{\"search_latencies\": [\"basic_statistics\", \"search_latency\"]}"
|
||||
shell: cmd
|
||||
|
||||
- name: Clean up
|
||||
if: always()
|
||||
run: |
|
||||
docker stop r2r postgres
|
||||
docker rm r2r postgres
|
||||
docker network rm r2r-network
|
||||
docker system prune -af --volumes
|
||||
shell: cmd
|
||||
@@ -1,39 +0,0 @@
|
||||
name: Links
|
||||
|
||||
on:
|
||||
repository_dispatch:
|
||||
workflow_dispatch:
|
||||
schedule:
|
||||
- cron: "00 18 * * *"
|
||||
|
||||
jobs:
|
||||
linkChecker:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v2
|
||||
|
||||
- name: Link Checker
|
||||
id: lychee
|
||||
uses: lycheeverse/lychee-action@v1
|
||||
continue-on-error: true
|
||||
with:
|
||||
args: "--output ./docs/lychee/out.md --config ./docs/.lycheerc"
|
||||
|
||||
- name: Get Lychee Exit Code
|
||||
id: get-exit-code
|
||||
run: echo "lychee_exit_code=$?" >> $GITHUB_ENV
|
||||
|
||||
- name: Create Custom Report
|
||||
if: failure()
|
||||
run: |
|
||||
echo "Lychee doesn't play nice with relative imports for .mdx files. Add a link to the .lycheeignore if needed!" > ./docs/lychee/custom_report.md
|
||||
echo "" >> ./docs/lychee/custom_report.md
|
||||
cat ./docs/lychee/out.md >> ./docs/lychee/custom_report.md
|
||||
|
||||
- name: Create Issue From File
|
||||
if: failure()
|
||||
uses: peter-evans/create-issue-from-file@v4
|
||||
with:
|
||||
title: Link Checker Report
|
||||
content-filepath: ./docs/lychee/custom_report.md
|
||||
labels: report, automated issue
|
||||
@@ -0,0 +1,35 @@
|
||||
name: Publish NPM Package
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
publish:
|
||||
runs-on: ubuntu-latest
|
||||
defaults:
|
||||
run:
|
||||
working-directory: js/sdk
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
|
||||
- name: Set up Node.js
|
||||
uses: actions/setup-node@v3
|
||||
with:
|
||||
node-version: '16'
|
||||
registry-url: 'https://registry.npmjs.org'
|
||||
|
||||
- name: Install pnpm
|
||||
uses: pnpm/action-setup@v2
|
||||
with:
|
||||
version: 6.0.2
|
||||
|
||||
- name: Install dependencies
|
||||
run: pnpm install
|
||||
|
||||
- name: Build
|
||||
run: pnpm run build
|
||||
|
||||
- name: Publish to npm
|
||||
run: pnpm publish --no-git-checks
|
||||
env:
|
||||
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
|
||||
@@ -4,7 +4,7 @@ on:
|
||||
push:
|
||||
tags:
|
||||
- "*"
|
||||
workflow_dispatch: # This line adds manual trigger support
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
publish:
|
||||
@@ -19,9 +19,11 @@ jobs:
|
||||
python-version: 3.9
|
||||
|
||||
- name: Install poetry
|
||||
working-directory: ./py
|
||||
run: pip install poetry
|
||||
|
||||
- name: Build and publish
|
||||
working-directory: ./py
|
||||
run: |
|
||||
poetry build
|
||||
poetry publish --username __token__ --password ${{ secrets.PYPI_API_TOKEN }}
|
||||
|
||||
@@ -2,15 +2,15 @@ name: CI
|
||||
|
||||
on:
|
||||
push:
|
||||
branches:
|
||||
- main
|
||||
- '**/feature/*'
|
||||
paths:
|
||||
- 'py/**'
|
||||
pull_request:
|
||||
branches:
|
||||
- main
|
||||
paths:
|
||||
- 'py/**'
|
||||
|
||||
jobs:
|
||||
pre-commit:
|
||||
continue-on-error: true
|
||||
runs-on: [ self-hosted, Linux ]
|
||||
|
||||
steps:
|
||||
@@ -18,6 +18,7 @@ jobs:
|
||||
uses: actions/checkout@v3
|
||||
|
||||
- name: Run pre-commit
|
||||
working-directory: ./py
|
||||
run: |
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
@@ -39,6 +40,7 @@ jobs:
|
||||
swap-storage: true
|
||||
|
||||
pytest:
|
||||
continue-on-error: true
|
||||
runs-on: [ self-hosted, Linux ]
|
||||
timeout-minutes: 15
|
||||
|
||||
@@ -62,12 +64,12 @@ jobs:
|
||||
curl -i http://localhost:11434
|
||||
|
||||
- name: Run pytest
|
||||
working-directory: ./py
|
||||
run: |
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
poetry install
|
||||
pip install pytest
|
||||
pip install pytest
|
||||
poetry run pytest tests/ -k "not redis and not sentence_transformer"
|
||||
deactivate
|
||||
rm -rf venv
|
||||
@@ -0,0 +1,38 @@
|
||||
name: R2R JS SDK Integration CI
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- 'js/sdk/**'
|
||||
pull_request:
|
||||
branches: [main]
|
||||
paths:
|
||||
- 'js/sdk/**'
|
||||
|
||||
jobs:
|
||||
build-and-test:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
defaults:
|
||||
run:
|
||||
working-directory: ./js/sdk
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v2
|
||||
|
||||
- name: Use Node.js
|
||||
uses: actions/setup-node@v2
|
||||
with:
|
||||
node-version: "18"
|
||||
|
||||
- name: Install pnpm
|
||||
uses: pnpm/action-setup@v2
|
||||
with:
|
||||
version: 8
|
||||
|
||||
- name: Install dependencies
|
||||
run: pnpm install
|
||||
|
||||
- name: Build
|
||||
run: pnpm run build
|
||||
@@ -0,0 +1,68 @@
|
||||
name: R2R JS SDK Integration Tests
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- 'js/sdk/**'
|
||||
pull_request:
|
||||
branches: [main]
|
||||
paths:
|
||||
- 'js/sdk/**'
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
defaults:
|
||||
run:
|
||||
working-directory: ./js/sdk
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v2
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v2
|
||||
with:
|
||||
python-version: "3.x"
|
||||
|
||||
- name: Install R2R
|
||||
run: |
|
||||
python -m pip install --upgrade pip
|
||||
pip install r2r
|
||||
|
||||
- name: Start R2R server
|
||||
env:
|
||||
POSTGRES_HOST: ${{ secrets.POSTGRES_HOST }}
|
||||
POSTGRES_USER: ${{ secrets.POSTGRES_USER }}
|
||||
POSTGRES_PASSWORD: ${{ secrets.POSTGRES_PASSWORD }}
|
||||
POSTGRES_PORT: ${{ secrets.POSTGRES_PORT }}
|
||||
POSTGRES_DBNAME: ${{ secrets.POSTGRES_DBNAME }}
|
||||
POSTGRES_VECS_COLLECTION: ${{ secrets.POSTGRES_VECS_COLLECTION }}
|
||||
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
|
||||
run: |
|
||||
r2r serve --port=8000 > r2r_server.log 2>&1 &
|
||||
sleep 15
|
||||
|
||||
- name: Use Node.js
|
||||
uses: actions/setup-node@v2
|
||||
with:
|
||||
node-version: "20.x"
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
|
||||
- name: Check if R2R server is running
|
||||
run: |
|
||||
curl http://localhost:8000/v2/health || echo "Server not responding"
|
||||
|
||||
- name: Display R2R server logs if server not responding
|
||||
if: failure()
|
||||
run: cat r2r_server.log
|
||||
|
||||
- name: Run integration tests
|
||||
run: npm run test
|
||||
|
||||
- name: Display R2R server logs if tests fail
|
||||
if: failure()
|
||||
run: cat r2r_server.log
|
||||
+13
@@ -1,3 +1,4 @@
|
||||
**/.DS_Store
|
||||
.env
|
||||
.DS_Store
|
||||
*.gguf
|
||||
@@ -15,3 +16,15 @@ coverage.xml
|
||||
|
||||
**/*.sqlite*
|
||||
**/*.sqlite3*
|
||||
|
||||
node_modules/
|
||||
dist/
|
||||
|
||||
*.exe
|
||||
*.exe~
|
||||
*.dll
|
||||
*.so
|
||||
*.dylib
|
||||
*.test
|
||||
go.work
|
||||
go.work.sum
|
||||
|
||||
@@ -12,14 +12,14 @@ repos:
|
||||
hooks:
|
||||
- id: isort
|
||||
name: isort
|
||||
entry: poetry run isort
|
||||
entry: bash -c 'cd py && poetry run isort $(git diff --cached --name-only --diff-filter=ACM | grep ".py$" | xargs -I {} echo ../{}) || true'
|
||||
language: system
|
||||
types: [python]
|
||||
args: ["--check", "--diff"]
|
||||
pass_filenames: false
|
||||
|
||||
- id: black
|
||||
name: black
|
||||
entry: poetry run black
|
||||
entry: bash -c 'cd py && poetry run black .'
|
||||
language: system
|
||||
types: [python]
|
||||
args: ["--check", "--diff"]
|
||||
pass_filenames: false
|
||||
|
||||
@@ -1,98 +0,0 @@
|
||||
<p align="left">
|
||||
<a href="https://r2r-docs.sciphi.ai"><img src="https://img.shields.io/badge/docs.sciphi.ai-3F16E4" alt="Docs"></a>
|
||||
<a href="https://discord.gg/p6KqD2kjtB"><img src="https://img.shields.io/discord/1120774652915105934?style=social&logo=discord" alt="Discord"></a>
|
||||
<a href="https://github.com/SciPhi-AI"><img src="https://img.shields.io/github/stars/SciPhi-AI/R2R" alt="Github Stars"></a>
|
||||
<a href="https://github.com/SciPhi-AI/R2R/pulse"><img src="https://img.shields.io/github/commit-activity/w/SciPhi-AI/R2R" alt="Commits-per-week"></a>
|
||||
<a href="https://opensource.org/licenses/MIT"><img src="https://img.shields.io/badge/License-MIT-purple.svg" alt="License: MIT"></a>
|
||||
</p>
|
||||
|
||||
<img src="./assets/r2r.png" alt="R2R Answer Engine">
|
||||
<h3 align="center">
|
||||
Build, scale, and manage user-facing Retrieval-Augmented Generation applications in production.
|
||||
</h3>
|
||||
|
||||
# About
|
||||
R2R was designed to bridge the gap between local LLM experimentation and scalable, production-ready Retrieval-Augmented Generation (RAG) applications. R2R provides a the latest techniques in RAG and is built around a RESTful API for ease of use.
|
||||
|
||||
For a more complete view of R2R, check out the [full documentation](https://r2r-docs.sciphi.ai/).
|
||||
|
||||
## Key Features
|
||||
- **📁 Multimodal Support**: Ingest files ranging from `.txt`, `.pdf`, `.json` to `.png`, `.mp3`, and more.
|
||||
- **🔍 Hybrid Search**: Combine semantic and keyword search with reciprocal rank fusion for enhanced relevancy.
|
||||
- **🔗 Graph RAG**: Automatically extract relationships and build knowledge graphs.
|
||||
- **🗂️ App Management**: Efficiently manage documents and users with full authentication.
|
||||
- **🔭 Observability**: Observe and analyze your RAG engine performance.
|
||||
- **🧩 Configurable**: Provision your application using intuitive configuration files.
|
||||
- **🔌 Extensibility**: Develop your application further with easy builder + factory pattern.
|
||||
- **🖥️ Dashboard**: Use the [R2R Dashboard](https://github.com/SciPhi-AI/R2R-Dashboard), an open-source React+Next.js app with optional authentication, to interact with R2R via GUI.
|
||||
|
||||
|
||||
## Install with pip
|
||||
The recommended way to get started with R2R is by using our CLI.
|
||||
|
||||
```bash
|
||||
pip install r2r
|
||||
```
|
||||
|
||||
Then, after installing R2R, it is recommended to launch with Docker, if possible:
|
||||
|
||||
```bash
|
||||
# export OPENAI_API_KEY=sk-...
|
||||
r2r serve --docker
|
||||
```
|
||||
|
||||
Alternatively, you may run R2R directly from the python package, but additional dependencies like Postgres+pgvector must be configured:
|
||||
|
||||
```bash
|
||||
# export OPENAI_API_KEY=sk-...
|
||||
# export POSTGRES...
|
||||
r2r --config-name=default serve
|
||||
```
|
||||
|
||||
|
||||
## Quickstart
|
||||
After [installing](https://r2r-docs.sciphi.ai/installation), the [R2R Quickstart](https://r2r-docs.sciphi.ai/quickstart) is your go to for a step-by-step guide to get up and running with R2R in minutes. The guide demonstrates R2R's Retrieval-Augmented Generation (RAG) system by ingesting sample documents and then showcasing features for search, RAG, logging, analytics, and document management.
|
||||
## Getting Started
|
||||
|
||||
To get started with R2R, we recommend starting with the quickstart and then moving on to specific cookbooks.
|
||||
|
||||
- [Installation](https://r2r-docs.sciphi.ai/installation): Quick setup using Docker or `pip`
|
||||
- [R2R Quickstart](https://r2r-docs.sciphi.ai/quickstart): A quickstart guide designed to get you familiarized with R2R.
|
||||
|
||||
|
||||
### API & SDKs
|
||||
- [Python SDK](https://r2r-docs.sciphi.ai/python-sdk)
|
||||
- [JS SDK](https://r2r-docs.sciphi.ai/js-sdk)
|
||||
- [API Reference](https://r2r-docs.sciphi.ai/api-reference/introduction)
|
||||
|
||||
|
||||
### Auth & Admin Features
|
||||
- [User Auth](https://r2r-docs.sciphi.ai/cookbooks/user-auth): A cookbook showing how to authenticate users using R2R.
|
||||
- [Analytics & Observability](https://r2r-docs.sciphi.ai/cookbooks/observability): A cookbook showing R2Rs end to end logging and analytics.
|
||||
- [Dashboard](https://r2r-docs.sciphi.ai/cookbooks/dashboard): A how-to guide on connecting with the R2R Admin/User Dashboard.
|
||||
|
||||
### RAG Cookbooks
|
||||
|
||||
- [Multiple LLMs](https://r2r-docs.sciphi.ai/cookbooks/multiple-llms): A simple cookbook showing how R2R supports multiple LLMs.
|
||||
- [Hybrid Search](https://r2r-docs.sciphi.ai/cookbooks/hybrid-search): A brief introduction to running hybrid search with R2R.
|
||||
- [Multimodal RAG](https://r2r-docs.sciphi.ai/cookbooks/multimodal): A cookbook on multimodal RAG with R2R.
|
||||
- [Knowledge Graphs](https://r2r-docs.sciphi.ai/cookbooks/knowledge-graph): A walkthrough of automatic knowledge graph generation with R2R.
|
||||
- [Advanced Graphs](https://r2r-docs.sciphi.ai/cookbooks/advanced-rag): A walkthrough of R2Rs advanced RAG features.
|
||||
- [Local RAG](https://r2r-docs.sciphi.ai/cookbooks/local-rag): A quick cookbook demonstration of how to run R2R with local LLMs.
|
||||
- [Reranking](https://r2r-docs.sciphi.ai/cookbooks/rerank-search): A short guide on how to apply reranking to R2R results.
|
||||
|
||||
## Community
|
||||
|
||||
[Join our Discord server](https://discord.gg/p6KqD2kjtB) to get support and connect with both the R2R team and other developers in the community. Whether you're encountering issues, looking for advice on best practices, or just want to share your experiences, we're here to help.
|
||||
|
||||
# Contributing
|
||||
|
||||
We welcome contributions of all sizes! Here's how you can help:
|
||||
|
||||
- Open a PR for new features, improvements, or better documentation.
|
||||
- Submit a [feature request](https://github.com/SciPhi-AI/R2R/issues/new?assignees=&labels=&projects=&template=feature_request.md&title=) or [bug report](https://github.com/SciPhi-AI/R2R/issues/new?assignees=&labels=&projects=&template=bug_report.md&title=)
|
||||
|
||||
### Our Contributors
|
||||
<a href="https://github.com/SciPhi-AI/R2R/graphs/contributors">
|
||||
<img src="https://contrib.rocks/image?repo=SciPhi-AI/R2R" />
|
||||
</a>
|
||||
-116
@@ -1,116 +0,0 @@
|
||||
# A generic, single database configuration.
|
||||
|
||||
[alembic]
|
||||
# path to migration scripts
|
||||
# Use forward slashes (/) also on windows to provide an os agnostic path
|
||||
script_location = alembic
|
||||
|
||||
# template used to generate migration file names; The default value is %%(rev)s_%%(slug)s
|
||||
# Uncomment the line below if you want the files to be prepended with date and time
|
||||
# see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file
|
||||
# for all available tokens
|
||||
# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
|
||||
|
||||
# sys.path path, will be prepended to sys.path if present.
|
||||
# defaults to the current working directory.
|
||||
prepend_sys_path = .
|
||||
|
||||
# timezone to use when rendering the date within the migration file
|
||||
# as well as the filename.
|
||||
# If specified, requires the python>=3.9 or backports.zoneinfo library.
|
||||
# Any required deps can installed by adding `alembic[tz]` to the pip requirements
|
||||
# string value is passed to ZoneInfo()
|
||||
# leave blank for localtime
|
||||
# timezone =
|
||||
|
||||
# max length of characters to apply to the "slug" field
|
||||
# truncate_slug_length = 40
|
||||
|
||||
# set to 'true' to run the environment during
|
||||
# the 'revision' command, regardless of autogenerate
|
||||
# revision_environment = false
|
||||
|
||||
# set to 'true' to allow .pyc and .pyo files without
|
||||
# a source .py file to be detected as revisions in the
|
||||
# versions/ directory
|
||||
# sourceless = false
|
||||
|
||||
# version location specification; This defaults
|
||||
# to alembic/versions. When using multiple version
|
||||
# directories, initial revisions must be specified with --version-path.
|
||||
# The path separator used here should be the separator specified by "version_path_separator" below.
|
||||
# version_locations = %(here)s/bar:%(here)s/bat:alembic/versions
|
||||
|
||||
# version path separator; As mentioned above, this is the character used to split
|
||||
# version_locations. The default within new alembic.ini files is "os", which uses os.pathsep.
|
||||
# If this key is omitted entirely, it falls back to the legacy behavior of splitting on spaces and/or commas.
|
||||
# Valid values for version_path_separator are:
|
||||
#
|
||||
# version_path_separator = :
|
||||
# version_path_separator = ;
|
||||
# version_path_separator = space
|
||||
version_path_separator = os # Use os.pathsep. Default configuration used for new projects.
|
||||
|
||||
# set to 'true' to search source files recursively
|
||||
# in each "version_locations" directory
|
||||
# new in Alembic version 1.10
|
||||
# recursive_version_locations = false
|
||||
|
||||
# the output encoding used when revision files
|
||||
# are written from script.py.mako
|
||||
# output_encoding = utf-8
|
||||
|
||||
sqlalchemy.url = sqlite:///local.sqlite
|
||||
|
||||
|
||||
[post_write_hooks]
|
||||
# post_write_hooks defines scripts or Python functions that are run
|
||||
# on newly generated revision scripts. See the documentation for further
|
||||
# detail and examples
|
||||
|
||||
# format using "black" - use the console_scripts runner, against the "black" entrypoint
|
||||
# hooks = black
|
||||
# black.type = console_scripts
|
||||
# black.entrypoint = black
|
||||
# black.options = -l 79 REVISION_SCRIPT_FILENAME
|
||||
|
||||
# lint with attempts to fix using "ruff" - use the exec runner, execute a binary
|
||||
# hooks = ruff
|
||||
# ruff.type = exec
|
||||
# ruff.executable = %(here)s/.venv/bin/ruff
|
||||
# ruff.options = --fix REVISION_SCRIPT_FILENAME
|
||||
|
||||
# Logging configuration
|
||||
[loggers]
|
||||
keys = root,sqlalchemy,alembic
|
||||
|
||||
[handlers]
|
||||
keys = console
|
||||
|
||||
[formatters]
|
||||
keys = generic
|
||||
|
||||
[logger_root]
|
||||
level = WARN
|
||||
handlers = console
|
||||
qualname =
|
||||
|
||||
[logger_sqlalchemy]
|
||||
level = WARN
|
||||
handlers =
|
||||
qualname = sqlalchemy.engine
|
||||
|
||||
[logger_alembic]
|
||||
level = INFO
|
||||
handlers =
|
||||
qualname = alembic
|
||||
|
||||
[handler_console]
|
||||
class = StreamHandler
|
||||
args = (sys.stderr,)
|
||||
level = NOTSET
|
||||
formatter = generic
|
||||
|
||||
[formatter_generic]
|
||||
format = %(levelname)-5.5s [%(name)s] %(message)s
|
||||
datefmt = %H:%M:%S
|
||||
@@ -1,77 +0,0 @@
|
||||
from logging.config import fileConfig
|
||||
|
||||
from sqlalchemy import engine_from_config, pool
|
||||
|
||||
from alembic import context
|
||||
|
||||
# this is the Alembic Config object, which provides
|
||||
# access to the values within the .ini file in use.
|
||||
config = context.config
|
||||
|
||||
# Interpret the config file for Python logging.
|
||||
# This line sets up loggers basically.
|
||||
if config.config_file_name is not None:
|
||||
fileConfig(config.config_file_name)
|
||||
|
||||
# add your model's MetaData object here
|
||||
# for 'autogenerate' support
|
||||
# from myapp import mymodel
|
||||
# target_metadata = mymodel.Base.metadata
|
||||
target_metadata = None
|
||||
|
||||
# other values from the config, defined by the needs of env.py,
|
||||
# can be acquired:
|
||||
# my_important_option = config.get_main_option("my_important_option")
|
||||
# ... etc.
|
||||
|
||||
|
||||
def run_migrations_offline() -> None:
|
||||
"""Run migrations in 'offline' mode.
|
||||
|
||||
This configures the context with just a URL
|
||||
and not an Engine, though an Engine is acceptable
|
||||
here as well. By skipping the Engine creation
|
||||
we don't even need a DBAPI to be available.
|
||||
|
||||
Calls to context.execute() here emit the given string to the
|
||||
script output.
|
||||
|
||||
"""
|
||||
url = config.get_main_option("sqlalchemy.url")
|
||||
context.configure(
|
||||
url=url,
|
||||
target_metadata=target_metadata,
|
||||
literal_binds=True,
|
||||
dialect_opts={"paramstyle": "named"},
|
||||
)
|
||||
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
def run_migrations_online() -> None:
|
||||
"""Run migrations in 'online' mode.
|
||||
|
||||
In this scenario we need to create an Engine
|
||||
and associate a connection with the context.
|
||||
|
||||
"""
|
||||
connectable = engine_from_config(
|
||||
config.get_section(config.config_ini_section, {}),
|
||||
prefix="sqlalchemy.",
|
||||
poolclass=pool.NullPool,
|
||||
)
|
||||
|
||||
with connectable.connect() as connection:
|
||||
context.configure(
|
||||
connection=connection, target_metadata=target_metadata
|
||||
)
|
||||
|
||||
with context.begin_transaction():
|
||||
context.run_migrations()
|
||||
|
||||
|
||||
if context.is_offline_mode():
|
||||
run_migrations_offline()
|
||||
else:
|
||||
run_migrations_online()
|
||||
@@ -1,26 +0,0 @@
|
||||
"""${message}
|
||||
|
||||
Revision ID: ${up_revision}
|
||||
Revises: ${down_revision | comma,n}
|
||||
Create Date: ${create_date}
|
||||
|
||||
"""
|
||||
from typing import Sequence, Union
|
||||
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
${imports if imports else ""}
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = ${repr(up_revision)}
|
||||
down_revision: Union[str, None] = ${repr(down_revision)}
|
||||
branch_labels: Union[str, Sequence[str], None] = ${repr(branch_labels)}
|
||||
depends_on: Union[str, Sequence[str], None] = ${repr(depends_on)}
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
${upgrades if upgrades else "pass"}
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
${downgrades if downgrades else "pass"}
|
||||
@@ -1,47 +0,0 @@
|
||||
"""add_user_id_to_logs
|
||||
|
||||
Revision ID: 7423a09421e9
|
||||
Revises:
|
||||
Create Date: 2024-08-05 10:49:10.714423
|
||||
|
||||
"""
|
||||
|
||||
from typing import Sequence, Union
|
||||
|
||||
import sqlalchemy as sa
|
||||
from sqlalchemy.engine.reflection import Inspector
|
||||
|
||||
from alembic import op
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision: str = "7423a09421e9"
|
||||
down_revision: Union[str, None] = None
|
||||
branch_labels: Union[str, Sequence[str], None] = None
|
||||
depends_on: Union[str, Sequence[str], None] = None
|
||||
|
||||
|
||||
def upgrade():
|
||||
conn = op.get_bind()
|
||||
inspector = Inspector.from_engine(conn)
|
||||
|
||||
# Check and add user_id to logs table if it doesn't exist
|
||||
if "user_id" not in [col["name"] for col in inspector.get_columns("logs")]:
|
||||
op.add_column("logs", sa.Column("user_id", sa.String(), nullable=True))
|
||||
|
||||
# Check and add user_id to log_info table if it doesn't exist
|
||||
if "user_id" not in [
|
||||
col["name"] for col in inspector.get_columns("log_info")
|
||||
]:
|
||||
op.add_column(
|
||||
"log_info", sa.Column("user_id", sa.String(), nullable=True)
|
||||
)
|
||||
|
||||
|
||||
def downgrade():
|
||||
# Remove user_id column from logs table if it exists
|
||||
with op.batch_alter_table("logs") as batch_op:
|
||||
batch_op.drop_column("user_id")
|
||||
|
||||
# Remove user_id column from log_info table if it exists
|
||||
with op.batch_alter_table("log_info") as batch_op:
|
||||
batch_op.drop_column("user_id")
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 4.8 MiB |
Binary file not shown.
|
Before Width: | Height: | Size: 1.2 MiB |
@@ -0,0 +1,15 @@
|
||||
module r2r-cli
|
||||
|
||||
go 1.23.0
|
||||
|
||||
replace github.com/yourusername/r2r_go_sdk => ../sdk
|
||||
|
||||
require (
|
||||
github.com/spf13/cobra v1.8.1
|
||||
github.com/yourusername/r2r_go_sdk v0.0.0-00010101000000-000000000000
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/inconshreveable/mousetrap v1.1.0 // indirect
|
||||
github.com/spf13/pflag v1.0.5 // indirect
|
||||
)
|
||||
@@ -0,0 +1,10 @@
|
||||
github.com/cpuguy83/go-md2man/v2 v2.0.4/go.mod h1:tgQtvFlXSQOSOSIRvRPT7W67SCa46tRHOmNcaadrF8o=
|
||||
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
|
||||
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
|
||||
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
|
||||
github.com/spf13/cobra v1.8.1 h1:e5/vxKd/rZsfSJMUX1agtjeTDf+qv1/JdBF8gg5k9ZM=
|
||||
github.com/spf13/cobra v1.8.1/go.mod h1:wHxEcudfqmLYa8iTfL+OuZPbBZkmvliBWKIezN3kD9Y=
|
||||
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
|
||||
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
@@ -0,0 +1,21 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
var rootCmd = &cobra.Command{
|
||||
Use: "r2r",
|
||||
Short: "R2R CLI for all core operations",
|
||||
Long: `R2R CLI for all core operations using the Go SDK.`,
|
||||
}
|
||||
|
||||
func main() {
|
||||
if err := rootCmd.Execute(); err != nil {
|
||||
fmt.Println(err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,181 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
"github.com/yourusername/r2r_go_sdk/pkg/sdk"
|
||||
)
|
||||
|
||||
func init() {
|
||||
rootCmd.AddCommand(ragCmd)
|
||||
}
|
||||
|
||||
func init() {
|
||||
// rootCmd.AddCommand(inspectKnowledgeGraphCmd)
|
||||
rootCmd.AddCommand(ragCmd)
|
||||
rootCmd.AddCommand(searchCmd)
|
||||
}
|
||||
|
||||
var inspectKnowledgeGraphCmd = &cobra.Command{
|
||||
Use: "inspect-knowledge-graph",
|
||||
Short: "Print relationships from the knowledge graph",
|
||||
Run: withTimer(func(cmd *cobra.Command, args []string) {
|
||||
client := sdk.NewClient("http://localhost:8000/v2", sdk.LogConfig{Verbose: true})
|
||||
limit, _ := cmd.Flags().GetInt("limit")
|
||||
// printDescriptions, _ := cmd.Flags().GetBool("print-descriptions")
|
||||
|
||||
response, err := client.InspectKnowledgeGraph()
|
||||
if err != nil {
|
||||
fmt.Printf("Error inspecting knowledge graph: %v\n", err)
|
||||
return
|
||||
}
|
||||
fmt.Println(response)
|
||||
}),
|
||||
}
|
||||
|
||||
var ragCmd = &cobra.Command{
|
||||
Use: "rag",
|
||||
Short: "Perform a RAG query",
|
||||
Run: withTimer(func(cmd *cobra.Command, args []string) {
|
||||
client := sdk.NewClient("http://localhost:8000/v2", sdk.LogConfig{Verbose: true})
|
||||
query, _ := cmd.Flags().GetString("query")
|
||||
useVectorSearch, _ := cmd.Flags().GetBool("use-vector-search")
|
||||
filtersStr, _ := cmd.Flags().GetString("filters")
|
||||
var filters map[string]interface{}
|
||||
if filtersStr != "" {
|
||||
json.Unmarshal([]byte(filtersStr), &filters)
|
||||
}
|
||||
searchLimit, _ := cmd.Flags().GetInt("search-limit")
|
||||
doHybridSearch, _ := cmd.Flags().GetBool("use-hybrid-search")
|
||||
useKgSearch, _ := cmd.Flags().GetBool("use-kg-search")
|
||||
kgSearchModel, _ := cmd.Flags().GetString("kg-search-model")
|
||||
stream, _ := cmd.Flags().GetBool("stream")
|
||||
ragModel, _ := cmd.Flags().GetString("rag-model")
|
||||
kgSearchType, _ := cmd.Flags().GetString("kg-search-type")
|
||||
kgSearchLevel, _ := cmd.Flags().GetString("kg-search-level")
|
||||
|
||||
generationConfig := sdk.GenerationConfig{
|
||||
Stream: stream,
|
||||
}
|
||||
if ragModel != "" {
|
||||
generationConfig.Model = ragModel
|
||||
}
|
||||
|
||||
vectorSearchSettings := &sdk.VectorSearchSettings{
|
||||
UseVectorSearch: useVectorSearch,
|
||||
Filters: filters,
|
||||
SearchLimit: searchLimit,
|
||||
DoHybridSearch: doHybridSearch,
|
||||
}
|
||||
|
||||
kgSearchSettings := &sdk.KGSearchSettings{
|
||||
UseKGSearch: useKgSearch,
|
||||
Model: kgSearchModel,
|
||||
SearchType: kgSearchType,
|
||||
SearchLevel: kgSearchLevel,
|
||||
}
|
||||
|
||||
response, err := client.RAG(query, generationConfig, vectorSearchSettings, kgSearchSettings)
|
||||
if err != nil {
|
||||
fmt.Printf("Error performing RAG query: %v\n", err)
|
||||
return
|
||||
}
|
||||
|
||||
if stream {
|
||||
for chunk := range response.StreamingChannel {
|
||||
fmt.Print(chunk)
|
||||
}
|
||||
fmt.Println()
|
||||
} else {
|
||||
fmt.Printf("Search Results:\n%s\n", response.SearchResults)
|
||||
fmt.Printf("Completion:\n%s\n", response.Completion)
|
||||
}
|
||||
}),
|
||||
}
|
||||
|
||||
var searchCmd = &cobra.Command{
|
||||
Use: "search",
|
||||
Short: "Perform a search query",
|
||||
Run: withTimer(func(cmd *cobra.Command, args []string) {
|
||||
client := sdk.NewClient("http://localhost:8000/v2", sdk.LogConfig{Verbose: true})
|
||||
query, _ := cmd.Flags().GetString("query")
|
||||
useVectorSearch, _ := cmd.Flags().GetBool("use-vector-search")
|
||||
filtersStr, _ := cmd.Flags().GetString("filters")
|
||||
var filters map[string]interface{}
|
||||
if filtersStr != "" {
|
||||
json.Unmarshal([]byte(filtersStr), &filters)
|
||||
}
|
||||
searchLimit, _ := cmd.Flags().GetInt("search-limit")
|
||||
doHybridSearch, _ := cmd.Flags().GetBool("use-hybrid-search")
|
||||
useKgSearch, _ := cmd.Flags().GetBool("use-kg-search")
|
||||
kgSearchModel, _ := cmd.Flags().GetString("kg-search-model")
|
||||
kgSearchType, _ := cmd.Flags().GetString("kg-search-type")
|
||||
kgSearchLevel, _ := cmd.Flags().GetString("kg-search-level")
|
||||
|
||||
vectorSearchSettings := &sdk.VectorSearchSettings{
|
||||
UseVectorSearch: useVectorSearch,
|
||||
Filters: filters,
|
||||
SearchLimit: searchLimit,
|
||||
DoHybridSearch: doHybridSearch,
|
||||
}
|
||||
|
||||
kgSearchSettings := &sdk.KGSearchSettings{
|
||||
UseKGSearch: useKgSearch,
|
||||
Model: kgSearchModel,
|
||||
SearchType: kgSearchType,
|
||||
SearchLevel: kgSearchLevel,
|
||||
}
|
||||
|
||||
results, err := client.Search(query, vectorSearchSettings, kgSearchSettings)
|
||||
if err != nil {
|
||||
fmt.Printf("Error performing search: %v\n", err)
|
||||
return
|
||||
}
|
||||
|
||||
fmt.Printf("KG_Search Enabled: %v\n", useKgSearch)
|
||||
fmt.Printf("KG_Search Type: %s\n", kgSearchType)
|
||||
|
||||
if len(results.VectorSearchResults) > 0 {
|
||||
fmt.Println("Vector search results:")
|
||||
for _, result := range results.VectorSearchResults {
|
||||
fmt.Println(result)
|
||||
}
|
||||
}
|
||||
|
||||
if len(results.KGSearchResults) > 0 {
|
||||
fmt.Println("KG search results:")
|
||||
for _, result := range results.KGSearchResults {
|
||||
fmt.Println(result)
|
||||
}
|
||||
}
|
||||
}),
|
||||
}
|
||||
|
||||
func init() {
|
||||
inspectKnowledgeGraphCmd.Flags().Int("limit", 100, "Limit the number of relationships returned")
|
||||
inspectKnowledgeGraphCmd.Flags().Bool("print-descriptions", false, "Print descriptions of entities and relationships")
|
||||
|
||||
ragCmd.Flags().String("query", "", "The query for RAG")
|
||||
ragCmd.Flags().Bool("use-vector-search", true, "Use vector search")
|
||||
ragCmd.Flags().String("filters", "", "Search filters as JSON")
|
||||
ragCmd.Flags().Int("search-limit", 10, "Number of search results to return")
|
||||
ragCmd.Flags().Bool("use-hybrid-search", false, "Perform hybrid search")
|
||||
ragCmd.Flags().Bool("use-kg-search", false, "Use knowledge graph search")
|
||||
ragCmd.Flags().String("kg-search-model", "", "Model for KG agent")
|
||||
ragCmd.Flags().Bool("stream", false, "Stream the RAG response")
|
||||
ragCmd.Flags().String("rag-model", "", "Model for RAG")
|
||||
ragCmd.Flags().String("kg-search-type", "global", "Local or Global")
|
||||
ragCmd.Flags().String("kg-search-level", "", "Level of cluster to use for Global KG search")
|
||||
|
||||
searchCmd.Flags().String("query", "", "The search query")
|
||||
searchCmd.Flags().Bool("use-vector-search", true, "Use vector search")
|
||||
searchCmd.Flags().String("filters", "", "Search filters as JSON")
|
||||
searchCmd.Flags().Int("search-limit", 10, "Number of search results to return")
|
||||
searchCmd.Flags().Bool("use-hybrid-search", false, "Perform hybrid search")
|
||||
searchCmd.Flags().Bool("use-kg-search", false, "Use knowledge graph search")
|
||||
searchCmd.Flags().String("kg-search-model", "", "Model for KG agent")
|
||||
searchCmd.Flags().String("kg-search-type", "global", "Local or Global")
|
||||
searchCmd.Flags().String("kg-search-level", "", "Level of KG search")
|
||||
}
|
||||
@@ -0,0 +1,111 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os/exec"
|
||||
"runtime"
|
||||
"runtime/debug"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
"github.com/yourusername/r2r_go_sdk/pkg/sdk"
|
||||
)
|
||||
|
||||
func init() {
|
||||
rootCmd.AddCommand(dockerDownCmd)
|
||||
rootCmd.AddCommand(healthCmd)
|
||||
rootCmd.AddCommand(generateReportCmd)
|
||||
|
||||
dockerDownCmd.Flags().Bool("volumes", false, "Remove named volumes")
|
||||
dockerDownCmd.Flags().Bool("remove-orphans", false, "Remove containers for services not defined in the Compose file")
|
||||
dockerDownCmd.Flags().String("project-name", "r2r", "Project name for Docker")
|
||||
}
|
||||
|
||||
// TODO: Implement the actual Docker Compose down logic
|
||||
var dockerDownCmd = &cobra.Command{
|
||||
Use: "docker-down",
|
||||
Short: "Bring down the Docker Compose setup",
|
||||
Run: func(cmd *cobra.Command, args []string) {
|
||||
volumes, _ := cmd.Flags().GetBool("volumes")
|
||||
removeOrphans, _ := cmd.Flags().GetBool("remove-orphans")
|
||||
projectName, _ := cmd.Flags().GetString("project-name")
|
||||
|
||||
fmt.Printf("Bringing down Docker Compose setup (Project: %s, Volumes: %v, Remove Orphans: %v)\n", projectName, volumes, removeOrphans)
|
||||
},
|
||||
}
|
||||
|
||||
// TODO: Implement the actual system report generation logic
|
||||
var generateReportCmd = &cobra.Command{
|
||||
Use: "generate-report",
|
||||
Short: "Generate a system report",
|
||||
Run: func(cmd *cobra.Command, args []string) {
|
||||
report := make(map[string]interface{})
|
||||
|
||||
// Get R2R version
|
||||
// TODO: Implement the actual version retrieval logic
|
||||
report["r2r_version"] = "1.0.0"
|
||||
|
||||
// Get Docker info
|
||||
dockerVerison, _ := exec.Command("docker", "--version").Output()
|
||||
report["docker_version"] = string(dockerVerison)
|
||||
|
||||
dockerPs, _ := exec.Command("docker", "ps", "--format", "{{.ID}}\t{{.Names}}\t{{.Status}}").Output()
|
||||
report["docker_info"] = string(dockerPs)
|
||||
|
||||
dockerNetworks, _ := exec.Command("docker", "network", "ls", "--format", "{{.ID}}\t{{.Name}}\t{{.Driver}}").Output()
|
||||
report["docker_networks"] = string(dockerNetworks)
|
||||
|
||||
dockerSubnets, _ := exec.Command("docker", "network", "inspect", "bridge", "--format", "{{.IPAM.Config}}").Output()
|
||||
report["docker_subnets"] = string(dockerSubnets)
|
||||
|
||||
// Get OS information
|
||||
report["os_info"] = map[string]string{
|
||||
"system": runtime.GOOS,
|
||||
"version": runtime.Version(),
|
||||
"machine": runtime.GOARCH,
|
||||
"processor": runtime.GOARCH,
|
||||
}
|
||||
|
||||
jsonReport, _ := json.MarshalIndent(report, "", " ")
|
||||
fmt.Println(string(jsonReport))
|
||||
},
|
||||
}
|
||||
|
||||
var healthCmd = &cobra.Command{
|
||||
Use: "health",
|
||||
Short: "Check the health of the server",
|
||||
Run: withTimer(func(cmd *cobra.Command, args []string) {
|
||||
client := sdk.NewClient("http://localhost:8000/v2", sdk.LogConfig{Verbose: true})
|
||||
response, err := client.Health()
|
||||
if err != nil {
|
||||
fmt.Printf("Error checking health: %v\n", err)
|
||||
return
|
||||
}
|
||||
fmt.Println(response)
|
||||
}),
|
||||
}
|
||||
|
||||
// TODO: r2r serve command
|
||||
|
||||
// TODO: update
|
||||
|
||||
var versionCmd = &cobra.Command{
|
||||
Use: "version",
|
||||
Short: "Print the version of R2R",
|
||||
Run: func(cmd *cobra.Command, args []string) {
|
||||
info, ok := debug.ReadBuildInfo()
|
||||
if !ok {
|
||||
fmt.Println("No version information available")
|
||||
return
|
||||
}
|
||||
|
||||
for _, dep := range info.Deps {
|
||||
if dep.Path == "github.com/yourusername/r2r_go_sdk" {
|
||||
fmt.Printf("R2R version: %s\n", dep.Version)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
fmt.Println("R2R version information not found")
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func withTimer(f func(*cobra.Command, []string)) func(*cobra.Command, []string) {
|
||||
return func(cmd *cobra.Command, args []string) {
|
||||
start := time.Now()
|
||||
f(cmd, args)
|
||||
duration := time.Since(start)
|
||||
fmt.Printf("Time taken: %.2f seconds\n", duration.Seconds())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
module github.com/yourusername/r2r_go_sdk
|
||||
|
||||
go 1.23.0
|
||||
@@ -0,0 +1,55 @@
|
||||
package integration
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"github.com/yourusername/r2r_go_sdk/pkg/sdk"
|
||||
)
|
||||
|
||||
func TestServerStatsIntegration(t *testing.T) {
|
||||
logConfig := sdk.LogConfig{
|
||||
Verbose: true,
|
||||
LogWriter: os.Stderr, // or any io.Writer
|
||||
}
|
||||
|
||||
client := sdk.NewClient("http://localhost:8000/v2", logConfig)
|
||||
|
||||
// Call Health
|
||||
health, err := client.Health()
|
||||
if err != nil {
|
||||
t.Fatalf("Health returned an error: %v", err)
|
||||
}
|
||||
|
||||
// Check if we received a response
|
||||
if health == nil {
|
||||
t.Fatalf("Health returned nil")
|
||||
}
|
||||
|
||||
// Call ServerStats
|
||||
stats, err := client.ServerStats()
|
||||
if err != nil {
|
||||
t.Fatalf("ServerStats returned an error: %v", err)
|
||||
}
|
||||
|
||||
// Check if we received a response
|
||||
if stats == nil {
|
||||
t.Fatalf("ServerStats returned nil")
|
||||
}
|
||||
|
||||
// Print the received stats for debugging
|
||||
t.Logf("Received server stats: %+v", stats)
|
||||
|
||||
// Check app settings
|
||||
settings, err := client.AppSettings()
|
||||
|
||||
if err != nil {
|
||||
t.Fatalf("AppSettings returned an error: %v", err)
|
||||
}
|
||||
|
||||
// Check if we received a response
|
||||
if settings == nil {
|
||||
t.Fatalf("AppSettings returned nil")
|
||||
}
|
||||
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user