Commit Graph

62 Commits

Author SHA1 Message Date
Nolan Tremelling e7db62e6bd Add Scheduler and Postgres Vacuum (#2089)
* Add scheduler and vacuum

* Lint

* Refactor test workflow, add mock scheduler to tests

* 0 3 * * *

* Missing quote in toml

* Add maintenance service mock
2025-03-24 16:06:18 -07:00
emrgnt-cmplxty f7f977e69c flake, rm clerk dep 2025-03-20 00:58:28 -07:00
Karthik Srinivasan 1914fc2bc5 Feat: Add support clerk auth provider 2025-03-18 18:59:46 -07:00
emrgnt-cmplxty a0ad6bf86a fix tests 2025-03-14 15:19:38 -07:00
NolanTrem 2cf9bab8fc Hotfix: pre-commit 2025-03-14 09:14:41 -07:00
Loger 10cda399aa Email provider mailersend (#2042)
* Add MailerSend email provider implementation

- Implement MailerSend email provider with full email sending capabilities
- Add support for verification, password reset, and password changed emails
- Update provider factory and configuration to include MailerSend
- Create test script for MailerSend email functionality
- Extend email configuration to support MailerSend API key and templates

* Enhance MailerSend email provider with improved logging and template handling

* Add MailerSend email provider to dependencies and update provider references

- Include MailerSend in the project dependencies within pyproject.toml.
- Update R2RProviders to include MailerSendEmailProvider.
- Clean up MailerSendEmailProvider by removing unnecessary logging statements.

* Remove test_mailersend.py file,
2025-03-14 09:12:07 -07:00
Nolan Tremelling fa59b93e67 Remove websocket logs in favor of Victorialogs (#2032) 2025-03-06 13:23:40 -08:00
Nolan Tremelling fb6353eebc Ruff Formatting and Tests Fixes (#1988)
* Check in

* Add code quality checks to CI/CD

* Fix workflow
2025-02-19 15:18:12 -08:00
Nolan Tremelling eadf196de0 Fix bad imports (#1985) 2025-02-18 10:20:26 -08:00
Nolan Tremelling df93e6e139 Move from black/isort to ruff (#1983) 2025-02-17 16:27:43 -08:00
Nolan Tremelling 01b66276e0 Add routes tests (#1971)
* Add routes tests

* Recomment rate limit on users.create

* Add response models for completions and embeddings
2025-02-12 11:31:44 -08:00
Nolan Tremelling b83348c245 Fix NaN comparison (#1958) 2025-02-10 19:53:43 -08:00
emrgnt-cmplxty e5b54733f9 up 2025-02-05 19:21:25 -06:00
Nolan Tremelling 5f6843e538 Separate config settings for retrieval vs ingestion embeddings (#1934) 2025-02-03 14:24:19 -08:00
Nolan Tremelling 9bf4da5ed2 Make sync client actually sync (#1908)
* Make sync client actually sync

* Add missing methods to sync

* Use self.client

* Ellipsis was wrong
2025-01-30 13:55:45 -08:00
emrgnt-cmplxty d07e72a063 finish r2r provider 2025-01-29 17:24:27 -08:00
emrgnt-cmplxty b404b3a4af finish r2r provider 2025-01-29 17:24:08 -08:00
NolanTrem 9d707cda72 First pass at Anthropic 2025-01-27 14:21:50 -08:00
emrgnt-cmplxty 88cb5fad69 merge 2025-01-20 18:50:21 -08:00
emrgnt-cmplxty df09b6d150 Merge pull request #1833 from mstickel/jwt-auth
Feat-1816 add JWT auth provider
2025-01-20 18:10:28 -08:00
emrgnt-cmplxty 6e550bc9ba first pass commit 2025-01-20 18:08:07 -08:00
emrgnt-cmplxty 0edafee419 fix agent default behavior 2025-01-16 11:47:33 -08:00
Mark Stickel 1ff24c47cd Feat-1816 add JWT auth provider 2025-01-13 15:36:30 -08:00
Nolan Tremelling dff76c5a24 Revise graph workflow (#1803) 2025-01-10 16:21:17 -08:00
Nolan Tremelling 3a2405c684 Further cleanup, remove old deduplication logic (#1746)
* Remove old deduplication pipeline

* Remove deprecated env vars

* Remove deprecated env vars

* Fix type errors on SDK

* Further clean up

* Fix string with union operator error

* String enum--don't need .value
2024-12-31 21:56:26 -06:00
emrgnt-cmplxty 0b9abee685 adding ability to set api-key 2024-12-27 18:59:24 -05:00
Nolan Tremelling 76ce750e4f Fix service level type errors (#1732) 2024-12-26 16:43:57 -06:00
Nolan Tremelling 0ddfd13552 Fix Crypto Providers in Factory (#1727)
* Fix crypto provider factory implementation

* Fix initialization in bcrypt
2024-12-24 01:26:04 -06:00
emrgnt-cmplxty 43cbe47801 up 2024-12-18 20:22:32 -08:00
emrgnt-cmplxty 563771862c a 2024-12-17 18:13:38 -08:00
emrgnt-cmplxty 0d7290ec0e up 2024-12-15 22:08:51 -08:00
emrgnt-cmplxty cf5e3aac9a checkin progress 2024-12-13 15:13:04 -08:00
Nolan Tremelling 87222ec3d8 Nolan/collections summary (#1685)
* Add document summary to extraction process

* Collection summarization and inclusion in community generation

* Add test, fix mypy

* Clean up

* Fix test with new required body, remove print statements and add check
2024-12-11 11:12:04 -08:00
emrgnt-cmplxty 2efb6e9f95 up (#1679)
* up

* up

* up

* up
2024-12-10 14:38:38 -08:00
emrgnt-cmplxty 1d5f1eb782 Dev minor (#1665)
* Feature/add web search (#1662)

* add web search to rag agent

* add web search to rag agent

* Fix agent bug, remove alias (#1661)

* Log Streaming (#1663)

* Fix agent bug, remove alias

* Allow ws on logs

* up (#1664)

* up

* add ingestion settings too

* up

* up

* up

* up

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
2024-12-05 21:31:26 -08:00
emrgnt-cmplxty d693f690a6 Feature/v3 api (#1656)
* improve ci/cd runtime

* update prompt tests

* improve ci/cd runtime (#1535)

* improve ci/cd runtime

* update prompt tests

* Support Python ^3.10 (#1534)

* add azure

* up

* up

* spec out v3 api

* checkin document router

* adding chunk abstr

* add list chunks

* add chunk search

* up

* add users routes

* up

* checkin progress

* add collections annotations

* add indices

* add user

* checkin work

* up

* complete conversations CRUD

* fix type errors

* add graph router

* add graphs

* Update JS (#1563)

* Feature/add graph to v3 (#1565)

* complete simple tests, cleanup routers

* up

* Harmonize Pagination across endpoints (#1564)

* Pagination

* Add fixmes

* Fix nested deletion filter bug (#1567)

* Remove Mintlify docs (#1569)

* Nolan/list collection (#1568)

* Check in

* More

* Fix nested transactions issue in sqlite logger

* Fix update collection return type

* JS V3 (#1571)

* Sync collections JS

* More documents

* Clean up messy code

* list not List

* Users first pass

* User tests and fixmes

* More

* typo

* More prompts

* Pre-commit improvements

* Remove prints

* Cleanups on conversations

* Branches response

* Chunks

* More work on the return types

* Jest config

* Fix branch creation time

* Fix lock

* Nolan/v3 tests (#1578)

* Add deprecated command back

* Add warning

* Fix GraphRAG tests (#1579)

* More cleanup (#1580)

* More cleanup

* More

* Fix test

* More cleanups

* More cleanups

* More

* Merge main

* Python SDK V3 (#1585)

* Python SDK V3

* Fix

* First pass (#1586)

* More V3 (#1587)

* Validation errors

* Update js test

* more

* Fix sync methods on v2 sdk, add check for download files (#1588)

* More CLI (#1589)

* Print logs on failing tests (#1590)

* Print logs on failing tests

* MOre

* cleanup

* Again

* Again

* More JS testing (#1591)

* More JS testing

* Cleanup

* More refactors for tests (#1592)

* System Routes (#1594)

* Fix type errors, pass collection id (#1595)

* Hotfix: dict

* V3 graph implmentations (#1593)

* complete simple tests, cleanup routers

* up

* up

* checkin

* up

* up

* response models

* checkin

* up

* checkin

* up

* up

* up

* up

* up

* up

* v2

* up

* up

* up

* up

---------

Co-authored-by: emrgnt-cmplxty <owen@algofi.org>

* Allow passing of collection id at document ingestion (#1596)

* KG Response sync (#1597)

* fix

* Fix Prompt Override (#1599)

* Fix Prompt Override

* print

* Caching

* Fix

* Updated Graph Models, Drop SID (#1598)

* New Graph Models

* Fix

* minor tweaks

* fix summary model (#1604)

* incr progress

* Add /users/me (#1605)

* Add /users/me

* oops

* Resolve Merge Conflicts (#1607)

* Fix conflicts

* Clean up

* Nolan/conflicts (#1608)

* expose reset data to admin (#1602)

* up (#1603)

* up

* up

* wtf github is a piece of garbage

---------

Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>

* wrapup walkthrough

* Add delete user method, sync JS to camel case (#1609)

* V3 graph testing (#1606)

* up

* up

* up

* graph crud

* up

* community endpts

* up

* up

* up

* up

* up

* up

* up

* up

* add back routers

* up

* pre-commit

* Fix Broken V2 Graphs, Better Response Models (#1612)

* Increase test coverage

* Fix v2 graphs, better response models

* Remaining types

* Add types to Python SDK

* Typo

* update tests

* revert test change

* up

* Add types to package export (#1613)

* Graph refactor (#1611)

* up

* up

* add back routers

* up

* pre-commit

* update tests

* revert test change

* up

* simplify

* up

* add the add/remove endpoints

* up

* include routers back

* Create branch update (#1617)

* Graph refactor (#1616)

* up

* up

* add back routers

* up

* pre-commit

* update tests

* revert test change

* up

* simplify

* up

* add the add/remove endpoints

* up

* include routers back

* List collections (#1619)

* up

* up

* up

* up

* Graph refactor (#1620)

* up

* up

* add back routers

* up

* pre-commit

* update tests

* revert test change

* up

* simplify

* up

* add the add/remove endpoints

* up

* include routers back

* up

* up

* up

* up

* Nolan/update graph (#1621)

* List collections

* Update Graph JS SDK

* up

* up

* cleanup

* Graph refactor (#1622)

* up

* up

* add back routers

* up

* pre-commit

* update tests

* revert test change

* up

* simplify

* up

* add the add/remove endpoints

* up

* include routers back

* up

* up

* up

* up

* up

* up

* cleanup

* up

* up

* up

* remove unnecessary functions

* up

* up

* complete document embedding workflow

* working get command on graph

* checkin progress

* up

* add entity and relationship deletions

* no verif

* up

* up

* up

* up (#1636)

* up

* sync graph

* up

* up

* fix relationship distance calc.

* fix issue with faulty collection filter (#1637)

* Patch/alternative fix logics 2 (#1638)

* fix issue with faulty collection filter

* further refinements, like fixing limits

* up

* fix logic around include metadata and scores

* fix double collection assignment

* up

* fix communities

* working clusters

* up

* add collection extraction

* add collection extraction

* up

* prep for merge

* Patch/alternative up with nolan (#1643)

* SDK First pass

* Add feature tracking

* Typo

* Check in

* Rebase

* Add Graph tests

* Fix Agent empty message bug

* Check in JS routes

* More tests, examples

* Sync python

* Expose Entity/Relationship Params in Routes (#1640)

* Expose Entity/Relationship Params

* Descriptions

* Modify create entities

* Create relationships

* set parent_id

* Update entitiy

* Update Relationships

* Check in

* Ellipsis fixes

* More cleanup

* Start CRUD on communities

* Communities DB

* Explicit working path

* Once again

* Fail fast false

* Testing around community creation

* Delete community test

* Update community tests

* Clean up type errors, cleaner code

* More cleanup

* More

* remove chunk_entity

* Delete bad, unused methods

* More

* fixup crud

* rm pull

---------

Co-authored-by: NolanTrem <34580718+NolanTrem@users.noreply.github.com>

* Feature/fix graph permissions (#1645)

* update docs / collections

* up

* Feature/fix auth checks (#1647)

* update docs / collections

* up

* fix super user and more

* up

* up (#1648)

* Feature/rm v2 api (#1649)

* SDK First pass

* Add feature tracking

* Typo

* Check in

* Rebase

* Add Graph tests

* Fix Agent empty message bug

* Check in JS routes

* More tests, examples

* Sync python

* Expose Entity/Relationship Params in Routes (#1640)

* Expose Entity/Relationship Params

* Descriptions

* Modify create entities

* Create relationships

* set parent_id

* Update entitiy

* Update Relationships

* Check in

* Ellipsis fixes

* More cleanup

* Start CRUD on communities

* Communities DB

* Explicit working path

* Once again

* Fail fast false

* Testing around community creation

* Delete community test

* Update community tests

* Clean up type errors, cleaner code

* More cleanup

* More

* remove chunk_entity

* Delete bad, unused methods

* More

* remove v2 api

* rm kg router

* cleanups

* fixup delete by filter

* fixup delete by filter

* fixes

* up

* up

---------

Co-authored-by: NolanTrem <34580718+NolanTrem@users.noreply.github.com>

* Improved Data Structures (#1650)

* Check in

* Most tests fixed

* fix tables

* Once more

* Move to a single community table

* Don't modify existing migration script--keep them atomic

* Migration

* Migration, more clean up

* All but deletion working

* Up

* Feature/tweaks for prod (#1651)

* tweaks for prod

* up

* final tweaks

* Nolan/deletion (#1652)

* Check in

* Most tests fixed

* fix tables

* Once more

* Move to a single community table

* Don't modify existing migration script--keep them atomic

* Migration

* Migration, more clean up

* All but deletion working

* Up

* Fix deletion

* Working migration (#1654)

* Feature/production tweaks (#1653)

* tweaks for prod

* up

* final tweaks

* prod tweaks

* fixed

---------

Co-authored-by: NolanTrem <34580718+NolanTrem@users.noreply.github.com>

* sort

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>
2024-12-03 18:45:23 -08:00
Nolan Tremelling 2dbecbec0e Change email_name to sender_name (#1634)
* +sendgrid email provider

* Update py/tests/core/providers/email/test_email_providers.py

The template_id parameter shown here is an example and is not intended to represent actual data. I included it as a placeholder.

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update py/tests/core/providers/email/test_email_providers.py

The template_id parameter shown here is an example and is not intended to represent actual data. I included it as a placeholder.

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* +Sendgrid email name

* Change email_name to sender_name

* Try not to unpack unnecessary values

* Another spot

---------

Co-authored-by: logerzerox <loger@mirai.mn>
Co-authored-by: logerzerox <70188352+logerzerox@users.noreply.github.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2024-11-25 08:57:37 -06:00
emrgnt-cmplxty 5c99a849ba Sendgrid Email Provider Implementation (#1614) (#1618)
* +sendgrid email provider

* Update py/tests/core/providers/email/test_email_providers.py

The template_id parameter shown here is an example and is not intended to represent actual data. I included it as a placeholder.



* Update py/tests/core/providers/email/test_email_providers.py

The template_id parameter shown here is an example and is not intended to represent actual data. I included it as a placeholder.



---------

Co-authored-by: logerzerox <70188352+logerzerox@users.noreply.github.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2024-11-20 15:47:28 -08:00
emrgnt-cmplxty ddfe870f42 dev-minor (#1509)
* adding bin support and make it default (#1508)

* Feature/tweak actions (#1507)

* up

* tweak actions

* adding bin sup and making it default

* tested and vetted

* up (#1510)

* up

* set verification to default false

* cleanup (#1512)

* cleanup

* cleanup prompt mgmt

* up

* cleanup printout

* cleanup new parser logic, set vlm as default for all providers

* allow user to re-override

* modify exp backoff implementation (#1513)

* Feature/tweak actions (#1507)

* up

* tweak actions

* modify exp backoff impl

---------

Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>

* Patch/touchups (#1515)

* cleanup

* cleanup prompt mgmt

* up

* cleanup printout

* cleanup new parser logic, set vlm as default for all providers

* allow user to re-override

* add touchups

* add extra parsers (#1516)

* add extra parsers (#1518)

* minor fixes (#1514)

* Feature/add back ollama provider (#1522)

* add extra parsers

* add back ollama

* rvert auth workflow

* Feature/add prompt tests and cleanup (#1523)

* add extra parsers

* add prompt tests, cleanup

* add prompt tests, cleanup

* merge

* set mock console as default

* set mock console as default

* fix config

* Update community model (#1524)

* Feature/tweak actions (#1507)

* up

* tweak actions

* Sync JS SDK, Harmonize Python SDK KG Methods (#1511)

* Feature/move logging (#1492)

* move logging provider out

* move logging provider to own directory, remove singleton

* cleanup

* fix refactoring tweak (#1496)

* Fix JSON serialization and Prompt ID Bugs for Prompts (#1491)

* Bug in get prompts

* Add tests

* Prevent verbose logging on standup

* Remove kg as required key in config, await get_all_prompts

* Remove reference to fragment id

* comment out ingestion

* complete logging port (#1499)

* Feature/dev rebased (#1500)

* Feature/move logging (#1493)

* move logging provider out

* move logging provider to own directory, remove singleton

* cleanup

* Update js package (#1498)

* fix refactoring tweak (#1496)

* Fix JSON serialization and Prompt ID Bugs for Prompts (#1491)

* Bug in get prompts

* Add tests

* Prevent verbose logging on standup

* Remove kg as required key in config, await get_all_prompts

* Remove reference to fragment id

* comment out ingestion

* complete logging port (#1499)

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>

* Fix handling for R2R exceptions (#1501)

* fix doc test (#1502)

* Harmonize python SDK KG methods for optional params, add missing JS methods

---------

Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>

* Clean up pagination and offset around KG (#1519)

* Move to R2R light for integration testing (#1521)

* Update community model

---------

Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>

* Patch/fix import bleed (#1526)

* Feature/tweak actions (#1507)

* up

* tweak actions

* Sync JS SDK, Harmonize Python SDK KG Methods (#1511)

* Feature/move logging (#1492)

* move logging provider out

* move logging provider to own directory, remove singleton

* cleanup

* fix refactoring tweak (#1496)

* Fix JSON serialization and Prompt ID Bugs for Prompts (#1491)

* Bug in get prompts

* Add tests

* Prevent verbose logging on standup

* Remove kg as required key in config, await get_all_prompts

* Remove reference to fragment id

* comment out ingestion

* complete logging port (#1499)

* Feature/dev rebased (#1500)

* Feature/move logging (#1493)

* move logging provider out

* move logging provider to own directory, remove singleton

* cleanup

* Update js package (#1498)

* fix refactoring tweak (#1496)

* Fix JSON serialization and Prompt ID Bugs for Prompts (#1491)

* Bug in get prompts

* Add tests

* Prevent verbose logging on standup

* Remove kg as required key in config, await get_all_prompts

* Remove reference to fragment id

* comment out ingestion

* complete logging port (#1499)

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>

* Fix handling for R2R exceptions (#1501)

* fix doc test (#1502)

* Harmonize python SDK KG methods for optional params, add missing JS methods

---------

Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>

* Clean up pagination and offset around KG (#1519)

* Move to R2R light for integration testing (#1521)

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>

* Patch/fix import bleed (#1527)

* Feature/tweak actions (#1507)

* up

* tweak actions

* Sync JS SDK, Harmonize Python SDK KG Methods (#1511)

* Feature/move logging (#1492)

* move logging provider out

* move logging provider to own directory, remove singleton

* cleanup

* fix refactoring tweak (#1496)

* Fix JSON serialization and Prompt ID Bugs for Prompts (#1491)

* Bug in get prompts

* Add tests

* Prevent verbose logging on standup

* Remove kg as required key in config, await get_all_prompts

* Remove reference to fragment id

* comment out ingestion

* complete logging port (#1499)

* Feature/dev rebased (#1500)

* Feature/move logging (#1493)

* move logging provider out

* move logging provider to own directory, remove singleton

* cleanup

* Update js package (#1498)

* fix refactoring tweak (#1496)

* Fix JSON serialization and Prompt ID Bugs for Prompts (#1491)

* Bug in get prompts

* Add tests

* Prevent verbose logging on standup

* Remove kg as required key in config, await get_all_prompts

* Remove reference to fragment id

* comment out ingestion

* complete logging port (#1499)

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>

* Fix handling for R2R exceptions (#1501)

* fix doc test (#1502)

* Harmonize python SDK KG methods for optional params, add missing JS methods

---------

Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>

* Clean up pagination and offset around KG (#1519)

* Move to R2R light for integration testing (#1521)

* fix ollama pdf parser

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>

* email auth false and js bump

* fix actions (#1528)

* Feature/add poppler check and fallback (#1529)

* fix actions

* fallback

* Patch/import shutil (#1530)

* fix actions

* fallback

* import shutil

* Feature/include basic pdf parsing everywhere (#1531)

* fix actions

* fallback

* import shutil

* add basic pdf as extra parser in all configs

* Remove non existent user login?

* attempt login

* Change password back

* add explicit setting, trigger rebuild

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>
Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
2024-10-30 08:24:40 -07:00
emrgnt-cmplxty 52b7b1afaf Dev rebased (#1503)
* fix refactoring tweak (#1496)

* Fix JSON serialization and Prompt ID Bugs for Prompts (#1491)

* Bug in get prompts

* Add tests

* Prevent verbose logging on standup

* Remove kg as required key in config, await get_all_prompts

* Remove reference to fragment id

* comment out ingestion

* complete logging port (#1499)

* Fix handling for R2R exceptions (#1501)

* fix doc test (#1502)

* up

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
2024-10-25 17:29:29 -07:00
emrgnt-cmplxty 6181c73d77 Feature/move logging (#1493)
* move logging provider out

* move logging provider to own directory, remove singleton

* cleanup
2024-10-25 11:21:54 -07:00
emrgnt-cmplxty ff38e1c33b up (#1488) 2024-10-24 19:10:23 -07:00
emrgnt-cmplxty 1ab36930fc Feature/dev minor merge main again (#1487)
* Feature/port kg to handler rebased (#1478)

* up

* up

* add back database

* merge

* merge

* removing prompt provider (#1479)

* removing prompt provider

* migrate prompt provider

* update lock file

* rm file provider

* refactor imports (#1481)

* making types more explicit across codebase (#1482)

* cleanup prompt handler

* cleanup prompt handler

* add kg create tables

* Feature/dev minor rebased (#1485)

* Update pyproject.toml (#1471)

* add alembic migrations to pypi release (#1472)

* add alembic migrations to pypi release

* modify path

* run on main (#1474)

* minor kg-search bug (#1476)

* minor kg-search bug

* fix

* Update pyproject.toml

* Move to psycopg binary (#1477)

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>
Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>

* Chunk CRUD (#1483)

* check in

* Finish ingest chunks, delete, and testing

* Docs and JS tests

* Ensure that we retrieve KG Status for documents in collection (#1484)

* Update pyproject.toml (#1471)

* add alembic migrations to pypi release (#1472)

* add alembic migrations to pypi release

* modify path

* run on main (#1474)

* minor kg-search bug (#1476)

* minor kg-search bug

* fix

* Update pyproject.toml

* Move to psycopg binary (#1477)

* Ensure that we retrieve KG Status for documents in collection

* github sucks

---------

Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>
Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
2024-10-24 17:49:29 -07:00
emrgnt-cmplxty bc53dc9c4f Dev minor (#1436)
* fix-actions (#1426)

* up

* modify

* add to github path

* Contextual Chunk Enrichment (#1433)

* add semantic chunking

* working

* precommit

* pre-commits

* Entity Deduplication (#1431)

* Modify graphrag prompt (#1421)

* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq (#1412)

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* Prod fixes + enhancements (#1407)

* change default settings back to fp32

* add logging and cache triples

* up

* up

* pre-commit and cleanups

* making community summary prompt async

* up

* up

* revert prompt changes

* up

* up

* modify default

* bump test timeout due to stricter concurrency limits

* bump sleep

* rm ubuntu from windows/mac workflows

* up

* add tests

* Feature/include vectors option document chunks (#1419)

* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* add option to include vectors in document chunks

* checkin

* update vector

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>

* Allow env var to set the default R2R deployment for the dashboard (#1417)

* modify community_summary_prompt function and corresponding prompt

* add tests

* up

* Feature/various documentation tweaks (#1422)

* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* add option to include vectors in document chunks

* checkin

* update vector

* some various documentation tweaks

* some various documentation tweaks

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>

* Graphrag tests (#1418)

* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq (#1412)

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* Prod fixes + enhancements (#1407)

* change default settings back to fp32

* add logging and cache triples

* up

* up

* pre-commit and cleanups

* making community summary prompt async

* up

* up

* revert prompt changes

* up

* up

* modify default

* bump test timeout due to stricter concurrency limits

* bump sleep

* rm ubuntu from windows/mac workflows

* up

* add tests

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>

* Modify graphrag tests timeouts (#1416)

* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq (#1412)

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* Prod fixes + enhancements (#1407)

* change default settings back to fp32

* add logging and cache triples

* up

* up

* pre-commit and cleanups

* making community summary prompt async

* up

* up

* revert prompt changes

* up

* up

* modify default

* bump test timeout due to stricter concurrency limits

* bump sleep

* rm ubuntu from windows/mac workflows

* modify timeouts

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>

* feat: Make prompt provider methods asynchronous (comments below) (#1415)

* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq (#1412)

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* Prod fixes + enhancements (#1407)

* change default settings back to fp32

* add logging and cache triples

* up

* up

* pre-commit and cleanups

* making community summary prompt async

* up

* up

* revert prompt changes

* up

* up

* modify default

* bump test timeout due to stricter concurrency limits

* bump sleep

* rm ubuntu from windows/mac workflows

* feat: Make prompt provider methods asynchronous

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>

* up

* up

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>

* Add routes, service level methods around conversations (#1420)

* Add routes, service level methods around conversations

* Slight refactor to match project conventions, add JS methods

* Updated JS methods

* JS docs

* Add python

* Update JS user tests

* add deduplication pipe, workflow, api, sdk, cli

* add summary workflow

* bug fixes

* pre-commit

* working

* search working

* adding dedup test files

* modify the update query

* precommit

* more testing

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>

* Refactor Python SDK for Intellisense, Thread Safety (#1430)

* Refactor Python SDK

* Fix CLI after SDK changes

* Add convo to agent

* Update conversation error handling, JS

* Remove unused, bad import

* Feature/cleanup refactor vector collection (#1432)

* Feature/include vectors option document chunks (#1419)

* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* add option to include vectors in document chunks

* checkin

* update vector

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>

* Allow env var to set the default R2R deployment for the dashboard (#1417)

* Feature/various documentation tweaks (#1422)

* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* add option to include vectors in document chunks

* checkin

* update vector

* some various documentation tweaks

* some various documentation tweaks

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>

* Graphrag tests (#1418)

* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq (#1412)

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* Prod fixes + enhancements (#1407)

* change default settings back to fp32

* add logging and cache triples

* up

* up

* pre-commit and cleanups

* making community summary prompt async

* up

* up

* revert prompt changes

* up

* up

* modify default

* bump test timeout due to stricter concurrency limits

* bump sleep

* rm ubuntu from windows/mac workflows

* up

* add tests

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>

* Modify graphrag tests timeouts (#1416)

* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq (#1412)

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* Prod fixes + enhancements (#1407)

* change default settings back to fp32

* add logging and cache triples

* up

* up

* pre-commit and cleanups

* making community summary prompt async

* up

* up

* revert prompt changes

* up

* up

* modify default

* bump test timeout due to stricter concurrency limits

* bump sleep

* rm ubuntu from windows/mac workflows

* modify timeouts

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>

* feat: Make prompt provider methods asynchronous (comments below) (#1415)

* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq (#1412)

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* Prod fixes + enhancements (#1407)

* change default settings back to fp32

* add logging and cache triples

* up

* up

* pre-commit and cleanups

* making community summary prompt async

* up

* up

* revert prompt changes

* up

* up

* modify default

* bump test timeout due to stricter concurrency limits

* bump sleep

* rm ubuntu from windows/mac workflows

* feat: Make prompt provider methods asynchronous

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>

* bump pyproject version

* first commit

* towards slimmer vector implementation logic

* up

* iterate

* up

* checkin

* up

* work doc chunks

* working vector search

* working full text search

* remove asyncpg

* passing vector tests

* up

* merge

* rm pytest

* up

* up

* fix delete

* up

* up

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>

* Add tests + Cleanup (#1437)

* up

* add tests

* test rename to sdk

* up

* fix tests

* typo

* modify chunk enrichment prompt (#1438)

* modify prompt

* up

* Fix type error on port argument of CLI (#1439)

* finish (#1440)

* finish

* up

* fix

* fix

* up

* fix

* final cleanups

* fix naming convention

* fix schema error

* increase timeout

* split graphrag actions

* fix collection exists error

* up (#1442)

* Add error message (#1443)

* up

* sdk fix

* locally testing build

* up docs (#1445)

* checkin work (#1444)

* checkin work

* finish index functionality extension

* fix concurrency

* add alembic (#1446)

* Prompt Tuning (#1447)

* Check in

* Fix after merging dev-minor in

* Ensure to not cause int overflow with hatchet (#1454)

* Bump JS (#1456)

* Ensure to not cause int overflow with hatchet

* bump js

* improve migration implementation (#1452)

* improve migration implementation

* refine migrations to include kg

* add alembic cli

* extend documentation

* extend docs and all that

* Revert change of default behaviour of entities endpoint, docs, tests (#1455)

* change def behavior of entities + delete endpoint

* pre-commit

* add deduplication tests

* Delete graph (#1450)

* up docs

* up

* up

* rename to raw_chunks

* up

* add tests

* up

* up

* change default

* change cli

* separate out deduplication tests

* change run type in the test

* up

* up

* add test concurrency

* up

* rm concurrency groups

* rm dedup tests

* remove json

* tests

* up

* fix lock

* Update postgres.py

* Feature/merge dev minor main (#1457)

* add run without orchestration (#1448)

* add run without orchestration

* bump versions

* bump versions

* bump versions

* fix

* up

* add end points

* add run without orchestration (#1448) (#1458)

* add run without orchestration

* bump versions

* bump versions

* bump versions

* fix

* up

* sync migration changes

* Ensure that we await ingest files in ingest_files method (#1460)

* Nolan/await update files (#1461)

* Ensure that we await ingest files in ingest_files method

* Await update files as well

* Docs changes (#1462)

* up

* up

* up

* up

* fix failed find and replace (#1463)

* fix failed find and replace

* fix

* Fix JS Client for Ingest Chunks (#1464)

* Ensure that we await ingest files in ingest_files method

* Await update files as well

* Fix js client

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>
Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
2024-10-22 21:59:10 -07:00
emrgnt-cmplxty 21686b35a4 Dev minor (#1410)
* Fix async JSON parsing (#1408)

* Fix async JSON parsing

* Remove score completion from js

* clean up js

* lockfile

* Feature/build custom logger (#1409)

* building a custom logger for r2r

* fix log

* maintain bkwd compat

* Feature/add kg description prompt (#1411)

* add kg desc prompt

* add kg desc prompt

* add kg desc prompt

* fix prompt name

* separate test run freq (#1412)

* separate test run freq

* task_id check fix

* add ingestion docs

* updatet

* add

* rm old prompts

* rm old prompots

* rm old prompts

* rm old prompts

* Prod fixes + enhancements (#1407)

* change default settings back to fp32

* add logging and cache triples

* up

* up

* pre-commit and cleanups

* making community summary prompt async

* up

* up

* revert prompt changes

* up

* up

* modify default

* bump test timeout due to stricter concurrency limits

* bump sleep

* rm ubuntu from windows/mac workflows

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>
2024-10-16 08:37:41 -07:00
emrgnt-cmplxty eed926235d D3v (#1398) (#1399)
* D3v (#1398)

* Feature/add zerox parser (#1396)

* Add KG tests (#1351)

* cli tests

* add sdk tests

* typo fix

* change workflow ordering

* add collection integration tests (#1352)

* bump pkg

* remove workflows

* fix sdk test port

* fix delete collection return check

* Fix document info serialization (#1353)

* Update integration-test-workflow-debian.yml

* pre-commit

* slightly modify

* up

* up

* smaller file

* up

* typo, change order

* up

* up

* change order

---------

Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>
Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>

* add graphrag docs (#1362)

* add documentation

* up

* Update js/sdk/src/models.tsx

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* pre-commit

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Concurrent index creation, allow -1 for paginated entries (#1363)

* update webdev-template for current next.js and r2r-js sdk (#1218)

Co-authored-by: Simeon <simeon@theobald.nz>

* Feature/extend integration tests rebased (#1361)

* cleanups

* add back overzealous edits

* extend workflows

* fix full setup

* simplify cli

* add ymls

* rename to light

* try again

* start light

* add cli tests

* fix

* fix

* testing..

* trying complete matrix testflow

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* up

* up

* up

* All actions

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* try offic pgvec formula

* sudo make

* sudo make

* push and pray

* push and pray

* add new actions

* add new actions

* docker push & pray

* inspect manifests during launch

* inspect manifests during launch

* inspect manifests during launch

* inspect manifests during launch

* setup docker

* setup docker

* fix default

* fix default

* Feature/rebase to r2r vars (#1364)

* cleanups

* add back overzealous edits

* extend workflows

* fix full setup

* simplify cli

* add ymls

* rename to light

* try again

* start light

* add cli tests

* fix

* fix

* testing..

* trying complete matrix testflow

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* up

* up

* up

* All actions

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* try offic pgvec formula

* sudo make

* sudo make

* push and pray

* push and pray

* add new actions

* add new actions

* docker push & pray

* inspect manifests during launch

* inspect manifests during launch

* inspect manifests during launch

* inspect manifests during launch

* setup docker

* setup docker

* fix default

* fix default

* make changes

* update the windows workflow

* update the windows workflow

* remove extra workflows for now

* bump pkg

* push and pray

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive tests

* revive tests

* revive tests

* revive tests

* update tests

* fix typos (#1366)

* update tests

* up

* up

* up

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* Add ingestion concurrency limit (#1367)

* up

* up

* up

---------

Co-authored-by: --global=Shreyas Pimpalgaonkar <--global=shreyas.gp.7@gmail.com>

* tweaks and fixes

* Fix Ollama Tool Calling (#1372)

* Update graphrag.mdx

* Fix Ollama tool calling

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>

* Clean up Docker Compose (#1368)

* Fix hatchet, dockerfile

* Update compose

* point to correct docker image

* Fix bug in deletion, better validation error handling (#1374)

* Update graphrag.mdx

* Fix bug in deletion, better validation error handling

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>

* vec index creation endpoint (#1373)

* Update graphrag.mdx

* upload files

* create vector index endpoint

* add to fastapi background task

* pre-commit

* move logging

* add api spec, support for all vecs

* pre-commit

* add workflow

* Modify KG Endpoints and update API spec (#1369)

* Update graphrag.mdx

* modify API endpoints and update documentation

* Update ingestion_router.py

* try different docker setup (#1371)

* try different docker setup

* action

* add login

* add full

* update action

* cleanup upload script

* cleanup upload script

* tweak action

* tweak action

* tweak action

* tweak action

* tweak action

* tweak action

* Nolan/ingest chunks js (#1375)

* Update graphrag.mdx

* Clean up ingest chunks, add to JS SDK

* Update JS docs

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>

* up (#1376)

* Bump JS package (#1378)

* add conversation

* checkin progress

* checkin progress

* Fix Create Graph (#1379)

* up

* up

* modify assertion

* up

* up

* increase entity limit

* changing aristotle back to v2

* pre-commit

* typos

* add test_ingest_sample_file_2_sdk

* Update server.py

* checkin progress

* up

* update

* Graphrag docs (#1382)

* add docs and refine code

* add python SDK documentation

* up

* update

* checkin

* up

* cleanup

* working sync logging

* test conversation history

* fix runner tests, rename `CHUNKS` to `chunks`

* adding zerox parser

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>
Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: FutureProofTechOps <operations@theobald.nz>
Co-authored-by: Simeon <simeon@theobald.nz>
Co-authored-by: --global=Shreyas Pimpalgaonkar <--global=shreyas.gp.7@gmail.com>

* Nolan/update hatchet (#1397)

* Move Hatchet to latest

* Update js package-lock

---------

Co-authored-by: NolanTrem <34580718+NolanTrem@users.noreply.github.com>

* Add hatchet logging (#1391)

* Add KG tests (#1351)

* cli tests

* add sdk tests

* typo fix

* change workflow ordering

* add collection integration tests (#1352)

* bump pkg

* remove workflows

* fix sdk test port

* fix delete collection return check

* Fix document info serialization (#1353)

* Update integration-test-workflow-debian.yml

* pre-commit

* slightly modify

* up

* up

* smaller file

* up

* typo, change order

* up

* up

* change order

---------

Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>
Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>

* add graphrag docs (#1362)

* add documentation

* up

* Update js/sdk/src/models.tsx

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* pre-commit

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Concurrent index creation, allow -1 for paginated entries (#1363)

* update webdev-template for current next.js and r2r-js sdk (#1218)

Co-authored-by: Simeon <simeon@theobald.nz>

* Feature/extend integration tests rebased (#1361)

* cleanups

* add back overzealous edits

* extend workflows

* fix full setup

* simplify cli

* add ymls

* rename to light

* try again

* start light

* add cli tests

* fix

* fix

* testing..

* trying complete matrix testflow

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* up

* up

* up

* All actions

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* try offic pgvec formula

* sudo make

* sudo make

* push and pray

* push and pray

* add new actions

* add new actions

* docker push & pray

* inspect manifests during launch

* inspect manifests during launch

* inspect manifests during launch

* inspect manifests during launch

* setup docker

* setup docker

* fix default

* fix default

* Feature/rebase to r2r vars (#1364)

* cleanups

* add back overzealous edits

* extend workflows

* fix full setup

* simplify cli

* add ymls

* rename to light

* try again

* start light

* add cli tests

* fix

* fix

* testing..

* trying complete matrix testflow

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* up

* up

* up

* All actions

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* try offic pgvec formula

* sudo make

* sudo make

* push and pray

* push and pray

* add new actions

* add new actions

* docker push & pray

* inspect manifests during launch

* inspect manifests during launch

* inspect manifests during launch

* inspect manifests during launch

* setup docker

* setup docker

* fix default

* fix default

* make changes

* update the windows workflow

* update the windows workflow

* remove extra workflows for now

* bump pkg

* push and pray

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive tests

* revive tests

* revive tests

* revive tests

* update tests

* fix typos (#1366)

* update tests

* up

* up

* up

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* Add ingestion concurrency limit (#1367)

* up

* up

* up

---------

Co-authored-by: --global=Shreyas Pimpalgaonkar <--global=shreyas.gp.7@gmail.com>

* tweaks and fixes

* Fix Ollama Tool Calling (#1372)

* Update graphrag.mdx

* Fix Ollama tool calling

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>

* Clean up Docker Compose (#1368)

* Fix hatchet, dockerfile

* Update compose

* point to correct docker image

* Fix bug in deletion, better validation error handling (#1374)

* Update graphrag.mdx

* Fix bug in deletion, better validation error handling

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>

* vec index creation endpoint (#1373)

* Update graphrag.mdx

* upload files

* create vector index endpoint

* add to fastapi background task

* pre-commit

* move logging

* add api spec, support for all vecs

* pre-commit

* add workflow

* Modify KG Endpoints and update API spec (#1369)

* Update graphrag.mdx

* modify API endpoints and update documentation

* Update ingestion_router.py

* try different docker setup (#1371)

* try different docker setup

* action

* add login

* add full

* update action

* cleanup upload script

* cleanup upload script

* tweak action

* tweak action

* tweak action

* tweak action

* tweak action

* tweak action

* Nolan/ingest chunks js (#1375)

* Update graphrag.mdx

* Clean up ingest chunks, add to JS SDK

* Update JS docs

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>

* up (#1376)

* Bump JS package (#1378)

* Fix Create Graph (#1379)

* up

* up

* modify assertion

* up

* up

* increase entity limit

* changing aristotle back to v2

* pre-commit

* typos

* add test_ingest_sample_file_2_sdk

* Update server.py

* add docs and refine code

* add python SDK documentation

* up

* add logs

* clean

* rm vq

* rm conflicts

* pre-commit

* up

* add logging

* update logs

* up

* up

* Update kg_service.py

---------

Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>
Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: FutureProofTechOps <operations@theobald.nz>
Co-authored-by: Simeon <simeon@theobald.nz>
Co-authored-by: --global=Shreyas Pimpalgaonkar <--global=shreyas.gp.7@gmail.com>

* feat: Add delete_node_via_document_id method to KGProvider (#1387)

Co-authored-by: shou.hsu <shou.hsu@utonia.com>

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>
Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: FutureProofTechOps <operations@theobald.nz>
Co-authored-by: Simeon <simeon@theobald.nz>
Co-authored-by: --global=Shreyas Pimpalgaonkar <--global=shreyas.gp.7@gmail.com>
Co-authored-by: Shou-Hsu <148736054+Shou-Hsu@users.noreply.github.com>
Co-authored-by: shou.hsu <shou.hsu@utonia.com>

* Vector + Index quantization (#1400)

* Add KG tests (#1351)

* cli tests

* add sdk tests

* typo fix

* change workflow ordering

* add collection integration tests (#1352)

* bump pkg

* remove workflows

* fix sdk test port

* fix delete collection return check

* Fix document info serialization (#1353)

* Update integration-test-workflow-debian.yml

* pre-commit

* slightly modify

* up

* up

* smaller file

* up

* typo, change order

* up

* up

* change order

---------

Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>
Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>

* add graphrag docs (#1362)

* add documentation

* up

* Update js/sdk/src/models.tsx

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* pre-commit

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Concurrent index creation, allow -1 for paginated entries (#1363)

* update webdev-template for current next.js and r2r-js sdk (#1218)

Co-authored-by: Simeon <simeon@theobald.nz>

* Feature/extend integration tests rebased (#1361)

* cleanups

* add back overzealous edits

* extend workflows

* fix full setup

* simplify cli

* add ymls

* rename to light

* try again

* start light

* add cli tests

* fix

* fix

* testing..

* trying complete matrix testflow

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* up

* up

* up

* All actions

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* try offic pgvec formula

* sudo make

* sudo make

* push and pray

* push and pray

* add new actions

* add new actions

* docker push & pray

* inspect manifests during launch

* inspect manifests during launch

* inspect manifests during launch

* inspect manifests during launch

* setup docker

* setup docker

* fix default

* fix default

* Feature/rebase to r2r vars (#1364)

* cleanups

* add back overzealous edits

* extend workflows

* fix full setup

* simplify cli

* add ymls

* rename to light

* try again

* start light

* add cli tests

* fix

* fix

* testing..

* trying complete matrix testflow

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* up

* up

* up

* All actions

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* try offic pgvec formula

* sudo make

* sudo make

* push and pray

* push and pray

* add new actions

* add new actions

* docker push & pray

* inspect manifests during launch

* inspect manifests during launch

* inspect manifests during launch

* inspect manifests during launch

* setup docker

* setup docker

* fix default

* fix default

* make changes

* update the windows workflow

* update the windows workflow

* remove extra workflows for now

* bump pkg

* push and pray

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive tests

* revive tests

* revive tests

* revive tests

* update tests

* fix typos (#1366)

* update tests

* up

* up

* up

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* Add ingestion concurrency limit (#1367)

* up

* up

* up

---------

Co-authored-by: --global=Shreyas Pimpalgaonkar <--global=shreyas.gp.7@gmail.com>

* tweaks and fixes

* Fix Ollama Tool Calling (#1372)

* Update graphrag.mdx

* Fix Ollama tool calling

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>

* Clean up Docker Compose (#1368)

* Fix hatchet, dockerfile

* Update compose

* point to correct docker image

* Fix bug in deletion, better validation error handling (#1374)

* Update graphrag.mdx

* Fix bug in deletion, better validation error handling

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>

* vec index creation endpoint (#1373)

* Update graphrag.mdx

* upload files

* create vector index endpoint

* add to fastapi background task

* pre-commit

* move logging

* add api spec, support for all vecs

* pre-commit

* add workflow

* Modify KG Endpoints and update API spec (#1369)

* Update graphrag.mdx

* modify API endpoints and update documentation

* Update ingestion_router.py

* try different docker setup (#1371)

* try different docker setup

* action

* add login

* add full

* update action

* cleanup upload script

* cleanup upload script

* tweak action

* tweak action

* tweak action

* tweak action

* tweak action

* tweak action

* Nolan/ingest chunks js (#1375)

* Update graphrag.mdx

* Clean up ingest chunks, add to JS SDK

* Update JS docs

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>

* up (#1376)

* Bump JS package (#1378)

* Fix Create Graph (#1379)

* up

* up

* modify assertion

* up

* up

* increase entity limit

* changing aristotle back to v2

* pre-commit

* typos

* add test_ingest_sample_file_2_sdk

* Update server.py

* add docs and refine code

* add python SDK documentation

* up

* add logs

* merge changes

* mc

* more

* add index + vector quantization

* pre-commits

* chnage default back to FP32

* kg vector and index test

* rm duplicate import

* Update r2r.toml

---------

Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>
Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: FutureProofTechOps <operations@theobald.nz>
Co-authored-by: Simeon <simeon@theobald.nz>
Co-authored-by: --global=Shreyas Pimpalgaonkar <--global=shreyas.gp.7@gmail.com>

* Feature/add zerox to unstructured rebased (#1406)

* Feature/add zerox parser (#1396)

* Add KG tests (#1351)

* cli tests

* add sdk tests

* typo fix

* change workflow ordering

* add collection integration tests (#1352)

* bump pkg

* remove workflows

* fix sdk test port

* fix delete collection return check

* Fix document info serialization (#1353)

* Update integration-test-workflow-debian.yml

* pre-commit

* slightly modify

* up

* up

* smaller file

* up

* typo, change order

* up

* up

* change order

---------

Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>
Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>

* add graphrag docs (#1362)

* add documentation

* up

* Update js/sdk/src/models.tsx

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* pre-commit

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Concurrent index creation, allow -1 for paginated entries (#1363)

* update webdev-template for current next.js and r2r-js sdk (#1218)

Co-authored-by: Simeon <simeon@theobald.nz>

* Feature/extend integration tests rebased (#1361)

* cleanups

* add back overzealous edits

* extend workflows

* fix full setup

* simplify cli

* add ymls

* rename to light

* try again

* start light

* add cli tests

* fix

* fix

* testing..

* trying complete matrix testflow

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* up

* up

* up

* All actions

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* try offic pgvec formula

* sudo make

* sudo make

* push and pray

* push and pray

* add new actions

* add new actions

* docker push & pray

* inspect manifests during launch

* inspect manifests during launch

* inspect manifests during launch

* inspect manifests during launch

* setup docker

* setup docker

* fix default

* fix default

* Feature/rebase to r2r vars (#1364)

* cleanups

* add back overzealous edits

* extend workflows

* fix full setup

* simplify cli

* add ymls

* rename to light

* try again

* start light

* add cli tests

* fix

* fix

* testing..

* trying complete matrix testflow

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* up

* up

* up

* All actions

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* try offic pgvec formula

* sudo make

* sudo make

* push and pray

* push and pray

* add new actions

* add new actions

* docker push & pray

* inspect manifests during launch

* inspect manifests during launch

* inspect manifests during launch

* inspect manifests during launch

* setup docker

* setup docker

* fix default

* fix default

* make changes

* update the windows workflow

* update the windows workflow

* remove extra workflows for now

* bump pkg

* push and pray

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive tests

* revive tests

* revive tests

* revive tests

* update tests

* fix typos (#1366)

* update tests

* up

* up

* up

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* Add ingestion concurrency limit (#1367)

* up

* up

* up

---------

Co-authored-by: --global=Shreyas Pimpalgaonkar <--global=shreyas.gp.7@gmail.com>

* tweaks and fixes

* Fix Ollama Tool Calling (#1372)

* Update graphrag.mdx

* Fix Ollama tool calling

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>

* Clean up Docker Compose (#1368)

* Fix hatchet, dockerfile

* Update compose

* point to correct docker image

* Fix bug in deletion, better validation error handling (#1374)

* Update graphrag.mdx

* Fix bug in deletion, better validation error handling

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>

* vec index creation endpoint (#1373)

* Update graphrag.mdx

* upload files

* create vector index endpoint

* add to fastapi background task

* pre-commit

* move logging

* add api spec, support for all vecs

* pre-commit

* add workflow

* Modify KG Endpoints and update API spec (#1369)

* Update graphrag.mdx

* modify API endpoints and update documentation

* Update ingestion_router.py

* try different docker setup (#1371)

* try different docker setup

* action

* add login

* add full

* update action

* cleanup upload script

* cleanup upload script

* tweak action

* tweak action

* tweak action

* tweak action

* tweak action

* tweak action

* Nolan/ingest chunks js (#1375)

* Update graphrag.mdx

* Clean up ingest chunks, add to JS SDK

* Update JS docs

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>

* up (#1376)

* Bump JS package (#1378)

* add conversation

* checkin progress

* checkin progress

* Fix Create Graph (#1379)

* up

* up

* modify assertion

* up

* up

* increase entity limit

* changing aristotle back to v2

* pre-commit

* typos

* add test_ingest_sample_file_2_sdk

* Update server.py

* checkin progress

* up

* update

* Graphrag docs (#1382)

* add docs and refine code

* add python SDK documentation

* up

* update

* checkin

* up

* cleanup

* working sync logging

* test conversation history

* fix runner tests, rename `CHUNKS` to `chunks`

* adding zerox parser

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>
Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: FutureProofTechOps <operations@theobald.nz>
Co-authored-by: Simeon <simeon@theobald.nz>
Co-authored-by: --global=Shreyas Pimpalgaonkar <--global=shreyas.gp.7@gmail.com>

* Nolan/update hatchet (#1397)

* Move Hatchet to latest

* Update js package-lock

---------

Co-authored-by: NolanTrem <34580718+NolanTrem@users.noreply.github.com>

* Add hatchet logging (#1391)

* Add KG tests (#1351)

* cli tests

* add sdk tests

* typo fix

* change workflow ordering

* add collection integration tests (#1352)

* bump pkg

* remove workflows

* fix sdk test port

* fix delete collection return check

* Fix document info serialization (#1353)

* Update integration-test-workflow-debian.yml

* pre-commit

* slightly modify

* up

* up

* smaller file

* up

* typo, change order

* up

* up

* change order

---------

Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>
Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>

* add graphrag docs (#1362)

* add documentation

* up

* Update js/sdk/src/models.tsx

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* pre-commit

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Concurrent index creation, allow -1 for paginated entries (#1363)

* update webdev-template for current next.js and r2r-js sdk (#1218)

Co-authored-by: Simeon <simeon@theobald.nz>

* Feature/extend integration tests rebased (#1361)

* cleanups

* add back overzealous edits

* extend workflows

* fix full setup

* simplify cli

* add ymls

* rename to light

* try again

* start light

* add cli tests

* fix

* fix

* testing..

* trying complete matrix testflow

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* up

* up

* up

* All actions

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* try offic pgvec formula

* sudo make

* sudo make

* push and pray

* push and pray

* add new actions

* add new actions

* docker push & pray

* inspect manifests during launch

* inspect manifests during launch

* inspect manifests during launch

* inspect manifests during launch

* setup docker

* setup docker

* fix default

* fix default

* Feature/rebase to r2r vars (#1364)

* cleanups

* add back overzealous edits

* extend workflows

* fix full setup

* simplify cli

* add ymls

* rename to light

* try again

* start light

* add cli tests

* fix

* fix

* testing..

* trying complete matrix testflow

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* cleanup matrix logic

* up

* up

* up

* All actions

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* rename to runner

* try offic pgvec formula

* sudo make

* sudo make

* push and pray

* push and pray

* add new actions

* add new actions

* docker push & pray

* inspect manifests during launch

* inspect manifests during launch

* inspect manifests during launch

* inspect manifests during launch

* setup docker

* setup docker

* fix default

* fix default

* make changes

* update the windows workflow

* update the windows workflow

* remove extra workflows for now

* bump pkg

* push and pray

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive full workflow

* revive tests

* revive tests

* revive tests

* revive tests

* update tests

* fix typos (#1366)

* update tests

* up

* up

* up

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* bump max connections

* Add ingestion concurrency limit (#1367)

* up

* up

* up

---------

Co-authored-by: --global=Shreyas Pimpalgaonkar <--global=shreyas.gp.7@gmail.com>

* tweaks and fixes

* Fix Ollama Tool Calling (#1372)

* Update graphrag.mdx

* Fix Ollama tool calling

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>

* Clean up Docker Compose (#1368)

* Fix hatchet, dockerfile

* Update compose

* point to correct docker image

* Fix bug in deletion, better validation error handling (#1374)

* Update graphrag.mdx

* Fix bug in deletion, better validation error handling

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>

* vec index creation endpoint (#1373)

* Update graphrag.mdx

* upload files

* create vector index endpoint

* add to fastapi background task

* pre-commit

* move logging

* add api spec, support for all vecs

* pre-commit

* add workflow

* Modify KG Endpoints and update API spec (#1369)

* Update graphrag.mdx

* modify API endpoints and update documentation

* Update ingestion_router.py

* try different docker setup (#1371)

* try different docker setup

* action

* add login

* add full

* update action

* cleanup upload script

* cleanup upload script

* tweak action

* tweak action

* tweak action

* tweak action

* tweak action

* tweak action

* Nolan/ingest chunks js (#1375)

* Update graphrag.mdx

* Clean up ingest chunks, add to JS SDK

* Update JS docs

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>

* up (#1376)

* Bump JS package (#1378)

* Fix Create Graph (#1379)

* up

* up

* modify assertion

* up

* up

* increase entity limit

* changing aristotle back to v2

* pre-commit

* typos

* add test_ingest_sample_file_2_sdk

* Update server.py

* add docs and refine code

* add python SDK documentation

* up

* add logs

* clean

* rm vq

* rm conflicts

* pre-commit

* up

* add logging

* update logs

* up

* up

* Update kg_service.py

---------

Co-authored-by: emrgnt-cmplxty <68796651+emrgnt-cmplxty@users.noreply.github.com>
Co-authored-by: emrgnt-cmplxty <owen@algofi.org>
Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: FutureProofTechOps <operations@theobald.nz>
Co-authored-by: Simeon <simeon@theobald.nz>
Co-authored-by: --global=Shreyas Pimpalgaonkar <--global=shreyas.gp.7@gmail.com>

* feat: Add delete_node_via_document_id method to KGProvider (#1387)

Co-authored-by: shou.hsu <shou.hsu@utonia.com>

* Finishing `zerox` integration

* Finishing `zerox` integration

* fix zerox, add cli docs

* mention runtime config

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>
Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: FutureProofTechOps <operations@theobald.nz>
Co-authored-by: Simeon <simeon@theobald.nz>
Co-authored-by: --global=Shreyas Pimpalgaonkar <--global=shreyas.gp.7@gmail.com>
Co-authored-by: Shou-Hsu <148736054+Shou-Hsu@users.noreply.github.com>
Co-authored-by: shou.hsu <shou.hsu@utonia.com>

* Improve ingestion errors, status bugs (#1405)

* Move Hatchet to latest

* Update js package-lock

* Better handling for errors on ingestion

* Filter out already processed chunks in create graph (#1404)

* add simple filter

* up

* modify query

* add zerox test

* fix conversation history

* attempt fix tests

* adding sleep  before retrieving the list of entities

* bump version

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>
Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: FutureProofTechOps <operations@theobald.nz>
Co-authored-by: Simeon <simeon@theobald.nz>
Co-authored-by: --global=Shreyas Pimpalgaonkar <--global=shreyas.gp.7@gmail.com>
Co-authored-by: Shou-Hsu <148736054+Shou-Hsu@users.noreply.github.com>
Co-authored-by: shou.hsu <shou.hsu@utonia.com>
2024-10-14 20:01:06 -07:00
emrgnt-cmplxty a640fb20b2 Release 3.2.11 (#1365)
• Added Knowledge Graph (KG) tests (#1351)
  - CLI tests
  - SDK tests
  - Workflow ordering changes

• Added collection integration tests (#1352)
  - Fixed SDK test port
  - Fixed delete collection return check

• Fixed document info serialization (#1353)

• Added GraphRAG documentation (#1362)

• Implemented concurrent index creation (#1363)
  - Allow -1 for paginated entries

• Updated webdev-template for current Next.js and r2r-js SDK (#1218)

• Extended integration tests (#1361)
  - Cleaned up and simplified CLI
  - Added new workflows and actions
  - Implemented matrix test flow
  - Docker setup and configuration

• Rebased to r2r vars (#1364)
  - Updated Windows workflow
  - Revived full workflow and tests

• Fixed typos (#1366)

• Added ingestion concurrency limit (#1367)

• Fixed Ollama Tool Calling (#1372)

• Cleaned up Docker Compose (#1368)

• Fixed bug in deletion, improved validation error handling (#1374)

• Added vector index creation endpoint (#1373)

• Modified KG Endpoints and updated API spec (#1369)

• Implemented new Docker setup (#1371)
  - Updated actions and scripts

• Cleaned up ingest chunks, added to JS SDK (#1375)

• Various updates and fixes (#1376, #1378, #1379)
  - Increased entity limit
  - Changed Aristotle back to v2
  - Added test_ingest_sample_file_2_sdk

• Added GraphRAG documentation and refined code (#1382)
  - Added Python SDK documentation

• Added R2R_ prefix to env vars (#1383)
  - Cleaned up docker compose
  - Bumped Python and JS package versions
  - Updated form-data

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>
Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: FutureProofTechOps <operations@theobald.nz>
Co-authored-by: Simeon <simeon@theobald.nz>
Co-authored-by: --global=Shreyas Pimpalgaonkar <--global=shreyas.gp.7@gmail.com>
2024-10-11 15:51:49 -07:00
emrgnt-cmplxty 3843e6e2db Feature/revive integration tests (#1343) (#1345)
* add postgres to integration

* add postgres to integration

* up

* rename

* hardcode

* add back postgres

* add back postgres

* add pgvector

* add pgvector

* add pgvector

* add pgvector

* add pgvector

* add pgvector

* add pgvector

* tweak config docs

* fix integration suite

* fix integration suite

* fix integration suite

* up

* up

* up

* up

* up

* up

* up

* up

* up

* update integration test

* final user tests

* final user tests
2024-10-04 22:33:59 -07:00
emrgnt-cmplxty 3721fcb7ad dev (#1269)
* Feature/encapsulate orchestration (#1265)

* fully encapsulate orchestration

* fully encapsulate orchestration

* complete encapsulation

* revert import cmt

* making default r2r lighter (#1268)

* making default r2r lighter

* fix bug in ingest files

* checkin

* workingupdate

* complete simple orch

* update docs

* up (#1273)

* up

* up

* merge (#1276)

* Postgres configuration settings (#1277)

* Improvements on Auth in JS, CLI (#1267)

* CLI Telemetry (#1266)

* check in

* working

* redundant

* JS auth improvements (#1263)

* Check in JS auth improvements

* Update login with toke

* Fix to allow disabling telemetry

* fix lock

* Try to avoid merge conflicts

* Clean up collection bugs

* remove comments

* Add Postgres configuration settings

* Image

* bad github conflict

* merge (#1278)

* port KG to postgres (#1272)

* create + cluster

* local search

* up

* clean

* format

* basics

* add collection_id and paginate

* rename

* change api

* up

* kg_creation_status

* up

* up

* up

* Feature/cleanup docker (#1279)

* merge

* up

* rm neo4j refs and cleanup docker cmds

* fixup

* Patch/cleanup kg migration (#1281)

* cleanup kg migration

* up

* Kg testing (#1280)

* up

* up

* up

* up

* slay neo4j

---------

Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>

* add back poetry lock

* Default Collections (#1282)

* Default collections

* Naughty naughty need to follow the SRP

* Testing (#1284)

* CICD

* actions

* poetry

* poetry

* Add env vars

* name

* increase timeout

* add user to collection

* Kg testing (#1283)

* up

* up

* cleanup kg migration

* up

* up

* up

* Kg testing (#1280)

* up

* up

* up

* up

* rename

* project name

* up

* add chunk order

* fragments => extractions

* bug squash

* up

* up

* up

* change postgres project name

---------

Co-authored-by: emrgnt-cmplxty <owen@algofi.org>

* Feature/fix logic bugs (#1285)

* fixing minor logic bugs in dev branch

* fixing minor logic bugs in dev branch

* merge

* Application docs

* add image (#1287)

* Add version to CLI telemetry (#1288)

* add image

* Add version to cli telemetry

* KG hatchet orchestration (#1286)

* up

* up

* cleanup kg migration

* up

* up

* up

* Kg testing (#1280)

* up

* up

* up

* up

* rename

* project name

* up

* add chunk order

* fragments => extractions

* bug squash

* up

* up

* up

* change postgres project name

* up

* up

---------

Co-authored-by: emrgnt-cmplxty <owen@algofi.org>

* Feature/update documentation rebased (#1289)

* up

* merge

* rebase

* fix ingestion issues (#1291)

* fix ingestion issues

* fix lock file

* fix embedding

* Fix SDK KG Serialization (#1292)

* add image

* serialization

* cleanup cli (#1294)

* CLI serialization (#1295)

* add image

* Fix more serialization around kg

* Nolan/schemacreation (#1296)

* add image

* Fix more serialization around kg

* add quotes to prevent reserved keywords from failing

* Prevent errors if config name is reserved name in postgres (#1297)

* Prevent reserved words (#1298)

* Move default collection id method to utils (#1299)

* Allow json fallback (#1301)

* hotfix: import

* Fix description error (#1302)

* up (#1303)

* rename to `full` (#1304)

* rename to `full`

* add html parser

* Remove postgres vecs variables (#1306)

* Feature/rename ingest files (#1307)

* rename to `full`

* add html parser

* Vec Removal (#1308)

* Remove postgres vecs variables

* up

* change kg settings parsing (#1309)

* offset + limit (#1305)

* offset + limit

* fix order

* update query

* change entity offset

* leiden seed

---------

Co-authored-by: Nolan Tremelling <34580718+NolanTrem@users.noreply.github.com>
Co-authored-by: Shreyas Pimpalgaonkar <shreyas.gp.7@gmail.com>
2024-10-01 18:43:52 -07:00
Nolan Tremelling 8644a086c3 Improvements on Auth in JS, CLI (#1267)
* CLI Telemetry (#1266)

* check in

* working

* redundant

* JS auth improvements (#1263)

* Check in JS auth improvements

* Update login with toke

* Fix to allow disabling telemetry

* fix lock

* Try to avoid merge conflicts
2024-09-25 17:38:08 -07:00