Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: index update does not process deleted documents #1613

Open
3 tasks done
mmaitre314 opened this issue Jan 10, 2025 · 0 comments
Open
3 tasks done

[Bug]: index update does not process deleted documents #1613

mmaitre314 opened this issue Jan 10, 2025 · 0 comments
Labels
bug Something isn't working triage Default label assignment, indicates new issue needs reviewed by a maintainer

Comments

@mmaitre314
Copy link
Contributor

Do you need to file an issue?

  • I have searched the existing issues and this bug is not already filed.
  • My model is hosted on OpenAI or Azure. If not, please look at the "model providers" issue and don't file a new one here.
  • I believe this is a legitimate bug, not just a question. If this is a question, please use the Discussions area.

Describe the bug

Two issues related to handling deleted documents during index updates:

The code here fails if there are only documents to delete and no documents to add:

# Fail on empty delta dataset
if delta_dataset.new_inputs.empty:
    error_msg = "Incremental Indexing Error: No new documents to process."
    raise ValueError(error_msg)

if delta_dataset.new_inputs.empty:

(related to #1600)

In the same function, delta_dataset.deleted_inputs is not referenced and only delta_dataset.new_inputs is. So it looks like document deletion is not implemented. If that's the case, with some guidance I may be able to provide a PR.

Steps to reproduce

Run an index update with only documents to delete and no documents to add.

Expected Behavior

Updates containing only document deletions succeed and deleted documents are removed from the index.

GraphRAG Config Used

N/A

Logs and screenshots

No response

Additional Information

@mmaitre314 mmaitre314 added bug Something isn't working triage Default label assignment, indicates new issue needs reviewed by a maintainer labels Jan 10, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage Default label assignment, indicates new issue needs reviewed by a maintainer
Projects
None yet
Development

No branches or pull requests

1 participant