Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding csv escape and quote logic #1630

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

adhazel
Copy link

@adhazel adhazel commented Jan 16, 2025

Description

Adding escape and quote characters to the pandas read_csv logic used by the csv file loader.

Related Issues

N/A

Proposed Changes

Added escapechar and quotechar to the line below in the csv.py file

data = pd.read_csv(buffer, encoding=config.encoding or "latin-1", escapechar='\\', quotechar='"')

Checklist

  • [X ] I have tested these changes locally.
  • [X ] I have reviewed the code changes.
  • [X ] I have updated the documentation (if necessary).
  • I have added appropriate unit tests (if applicable).

Additional Notes

Most chunk content will need to have and retain commas and line breaks so this appears to be a quick and easy enablement patch. It does add new logic to existing functionality but I do not see a way this change could negatively impact existing usage.

Added backslash as the escape character and double quotes as the quote character to the pandas read_csv call used by the load_file function.
Added documentation for the backslash escape character and double-quote for the quote character.
@adhazel adhazel requested review from a team as code owners January 16, 2025 15:12
@adhazel
Copy link
Author

adhazel commented Jan 16, 2025

Sorry for the multiple check-ins. This is my first time using some of these libraries or attempting a pull request in this repo. I'm not sure how to resolve the smoke tests or to set them up to run in GitHub actions. Can someone reach out to assist me on this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant